PC Engines APU1D4 with OPNsense

After many years (2015) of operation as a home firewall, my PC Engines APU1D4 croaked (so I thought).  Initially I was thinking it was the mSATA drive, so I started there.  New drive came in, plugged it in and same issue, click, click, click and no POST.  Considering the age of the board, I actually found a replacement which I quickly ordered assuming that was the next best option.  New board arrives, I quickly plug it in and click, click, click.  Moving on to the next logical piece, the power supply.  Ran over to my local electronics parts store and found a suitable power supply with proper voltage/amperage.  Plug everything in and up it goes!  It was an expensive lesson (with ultimately a cheap solution), but now I have spare parts.

After noticing pfSense had not made updates in well over a year, I moved to OPNsense.  The install and setup was flawless, no issues.  One thing I did notice was the near daily crashes (page faults) and reboots.  The APU1D4 boards have Realtek NIC's and OPNsense does not come bundled with the correct driver, so it uses the default kernel driver.  Under Plugins in the OPNsense console, you will notice a "os-realtek-re" option.  Install it and reboot.  So far its been nearly a day now without a reboot and I am assuming this addresses the issue.  

Hopefully this helps someone else in the same frustrating situation.

Update: Crashes continue even after installing the Realtek driver but less frequent.  Several posts have mentioned disabling both the Spectre and Meltdown tunables.  These are "vm.pmap.pti" = 0 and "hw.ibrs_disable" set to 1.  Being a home based firewall, I am not too concerned with either of these.  Plus, there is apparently quite a bit of overhead with both settings which should lead to faster throughput being disabled.  Firewall has been up for nearly two days now without a crash, so we will see.  Next up, updating the coreboot BIOS to v4.17.0.3.  Here is a handy little script to do so.

2021 Mileage Roundup

Not as many miles as I would have hoped, but still not terrible for 2021.

Get NetScaler HA status using PowerShell

I had the need today to pull the HA status from numerous NetScalers before doing some maintenance.  Instead of logging in to each device, this script takes advantage of the NetScaler API to pull and summarize this information.  Hope this helps someone...

if (-not $login){
    $login = @(
        @{
            name = "somedomain"
            credential = Get-Credential -Message somedomain
        },
        @{
            name = "root"
            credential = Get-Credential -Message nsroot -UserName nsroot
        }
    )
}

$clusters = @(
    @{
        nodes = @(
            "nsr1.domain.local"
            "nsr3.domain.local"
        )
        credential = "somedomain"
    },
    @{
        nodes = @(
            "nsr1-fe.domain.local.remote"
            "nsr3-fe.domain.local.remote"
        )
        credential = "root"
    }
)

$result = foreach($cluster in $clusters){

    # Build login object
    $cred = $login | Where-Object {$_.name -eq $cluster.credential}
    $auth = @{ login= @{ username = $cred.credential.UserName; password = $cred.credential.GetNetworkCredential().Password } }

    # Enumerate nodes
    foreach($node in $cluster.nodes){

        Write-Verbose ("Working on node {0} with credential {1}" -f $node, $cred.name) -Verbose

        # Get auth token from NSR
        $token = Invoke-RestMethod -Uri ("https://{0}/nitro/v1/config/login" -f $node) -Method Post -Body ($auth | ConvertTo-Json) -ContentType "application/json"

        # Build auth session cookie
        $session = New-Object Microsoft.PowerShell.Commands.WebRequestSession
        $cookie = New-Object System.Net.Cookie 
        $cookie.Name = "NITRO_AUTH_TOKEN"
        $cookie.Value = $token.sessionid
        $cookie.Domain = $node
        $session.Cookies.Add($cookie)

        # Get NSR HA configuration
        $hainfo = Invoke-RestMethod -Uri ("https://{0}/nitro/v1/config/hanode" -f $node) -WebSession $session -Method Get

        # Log off session
        Invoke-RestMethod -Uri ("https://{0}/nitro/v1/config/logout" -f $node) -WebSession $session -Method Post -Body (@{logout = @{} } | ConvertTo-Json) -ContentType "application/json"

        [PSCustomObject]@{
            NodeName = $node
            NodeIP =  $hainfo.hanode.ipaddress -join ', '
            HAStatus = $hainfo.hanode.hastatus -join ', '
            State = $hainfo.hanode.state -join ', '
        }
    }
}

$result | Format-Table -AutoSize

Azure DevOps Download secure file

Ran in to an interesting issue today in regards to on-prem Azure DevOps and the "Download secure file" task. The step was throwing the message "##[error]Error: self signed certificate in certificate chain". We use self signed certificates internally and the agents are not trusting the server. In order to get around this, set the system environment variable NODE_TLS_REJECT_UNAUTHORIZED to 0, then restart each agent. Should work fine after this.

I am simply posting what worked for me, but others are seeing this as well:

https://developercommunity.visualstudio.com/content/problem/483108/unable-to-verify-the-first-certificate.html

Check a site changes in PowerShell

I am in the process of purchasing a new backpack. As part of this process, I have been working with the MFG on a date for when their new pack is coming out. Instead of hitting the site X times a day, this simple PowerShell script monitors for changes on a specific page. There are several sites out there which do this for you, but they normally charge. Yeah, they are a bit more fancy and allow change regions, but I need to detect a simple change. So, here it is:

$checkUrl    = "https://somesite/products/packs/"
$checkMin    = 30
$origcontent = Invoke-WebRequest -Uri $checkUrl -Method Get

do {
    
    $content = Invoke-WebRequest -Uri $checkUrl -Method Get

    if ($content.Content -eq $origcontent.Content){
        Write-Verbose "Page contents are the same, sleeping.." -Verbose
        Start-Sleep -Seconds ($checkMin*60)
    }else{
        Write-Host "Page difference detected, exiting!" -ForegroundColor Green
        break
    }

} while ($true)

2020 Mileage Roundup

Cleanup Windows AD delegation

Throughout the course of time, we have gone through two iterations of naming standards in regards to Hyper-V cluster node names. Since delegation is not tied to the associated computer account, the delegated server name persists within computer accounts. Meaning, when the computer account is removed, the delegation remains behind. This script will go through each, validate if the computer account actually still exists, then remove it if not. Great little way to keep things clean...

$hvObjects = Get-ADComputer -SearchBase "<ou_path>" -Filter 'Name -like "some*filter*"' -Properties msDS-AllowedToDelegateTo

foreach ($hvComputer in $hvObjects){

    Write-Verbose ('Processing computer account {0}' -f $hvComputer.Name) -Verbose

    $delegList = @()

    foreach ($deleg in ($hvComputer | Select -ExpandProperty msDS-AllowedToDelegateTo | Sort-Object)){

        $compName = (($deleg -split "/")[1] -split "\.")[0]

        if (Get-ADComputer -Filter ('Name -eq "{0}"' -f $compName) -SearchBase "<ou_path>"){
            $delegList += $deleg.ToString()
        }else{
            Write-Warning ('Computer account {0} not found, removing from delegation [{1}]' -f $compName,$deleg)            
        }

    }

    if ($delegList){
        Write-Verbose ('Setting updated delegation on computer account {0}' -f $hvComputer.Name) -Verbose
        $hvComputer| Set-ADComputer -Replace @{'msDS-AllowedToDelegateTo'= $delegList} -Verbose
    }

}

Saving personal Git snippets

I had a need this week to save all my personal Git Snippets.  Obviously, viewing and saving each Snippet would have been extremely time consuming and simply not necessary given Git has an API.  The PowerShell 7 script below will enumerate each of your Snippets and save them to the folder specified.  This will also save a file with the same name as the Snippet, but with a .info extension which includes all pertinent information about the Snippet itself. Quick and dirty, but works :-)

$git_token  = @{ "PRIVATE-TOKEN" = "<your_git_api_token" }
$git_url    = "https://git.some_domain.io/api/v4/snippets"
$snip_count = 100
$export_loc = "E:\Snippets"

# Enumerate each snippet
foreach ($snip in (Invoke-RestMethod -Uri ("{0}?per_page={1}" -f $git_url, $snip_count) -Headers $git_token)) {
    
    Write-Verbose ("Processing snippet: {0} [Id: {1}]" -f $snip.title, $snip.id) -Verbose
    
    # Determine file path based on file_name or title if file_name does not exist
    $out_file = if(!$snip.file_name){
        ("{0}.snip" -f ($snip.title.Split([IO.Path]::GetInvalidFileNameChars()) -join '_' -replace ' ', '_'))
    }else{
        $snip.file_name
    }
    
    Write-Verbose (" Out file: {0}" -f $out_file) -Verbose
    
    # Save snippet code
    Invoke-RestMethod -Uri ("{0}/{1}/raw" -f $git_url, $snip.id) -Headers $git_token -OutFile (Join-Path -Path $export_loc -ChildPath $out_file)

    # Save info file
    $snip | Out-File -FilePath (Join-Path -Path $export_loc -ChildPath ("{0}.info" -f (Split-Path -Path $out_file -LeafBase)))
}

Extracting Prometheus Metric Names from Grafana

Had the need to extract all our Prometheus metric names out of our Grafana dashboards to determine what was in use and what was not. The PowerShell script below will use the Grafana API to pull back all the dashboards within a specific folder and with a partial name, enumerate the dashboards, extract the expressions, then extract the metric name(s):

[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12

$boards = Invoke-WebRequest "https://[url]/api/search?query=hyper&folderIds=2308" -Method Get | Select -ExpandProperty Content | ConvertFrom-Json

foreach($board in $boards){
    
    $boardinfo = Invoke-WebRequest ("https://[url]/api/dashboards/uid/{0}" -f $board.uid) -Method Get | Select -ExpandProperty Content | ConvertFrom-Json

    $graphs  = @()
    $graphs  = ($boardinfo.dashboard.panels | Where-Object type -eq row).panels
    $graphs += ($boardinfo.dashboard.panels | Where-Object type -notmatch "row")

    $result = Select-String "(?<=\()(\w*)(?=\{)" -input $graphs.targets.expr -AllMatches | Foreach-Object {$_.matches} | Sort-Object Value | Select-Object -ExpandProperty Value -Unique

    [PSCustomObject]@{
        Title = $board.title
        Count = $result.count
        Metrics = $result        
    }

}

Hyper-V Storage Jobs with Progress Bar

Simple script to show Hyper-V storage jobs with a progress bar...

This is a little flaky, but does work.  YMMV...

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29

$prevJobs = {}.Invoke() While ($jobs = Get-StorageJob | Sort-Object Name) { $idGen = 0 $jobs | % { $jobObj = [PSCustomObject]@{ Name = $_.Name ProgId = $idGen++ UniqueId = $_.UniqueId Activity = ("Disk Operation: {0} [{1}]" -f $_.Name, $_.JobState) Complete = $false } if (($prevJobs | Where UniqueId -eq $jobObj.UniqueId).Count -eq 0){ $prevJobs.Add($jobObj) } Write-Progress -Id $jobObj.ProgId -Activity $jobObj.Activity -Status ("{0}% Complete:" -f $_.PercentComplete) -PercentComplete $_.PercentComplete } $prevJobs | Where Complete -eq $false | %{ if (-not ($jobs | Where UniqueId -eq $_.UniqueId)) { Write-Progress -Id $_.ProgId -Activity $_.Activity -Completed $_.Complete = $true } } Start-Sleep -Seconds 5 }