VMware Log Insight – Duplicate Hosts Listed

After returning from a VMware Design course and having learned that Log Insight was now free to valid vCenter licence holders, I set straight to work deploying it into our development environment. However, despite having under 25 hosts Log Insight soon started complaining about exceeding the allowed 25 OSI limit. ESXi hosts were being displayed in the statistics with both their defined hostname, but also what appeared to be their FQDN (e.g. vmhosta-01 and vmhosta-01.domain.com).

Checking on the hosts themselves (esxcli system hostname get) there was no domain or FQDN set, but they do have FQDNs in DNS, and vCenter knows them by their FQDN.

To fix the issue, I made things consistent by setting the domain and FQDN on all hosts using PowerCLI:

$vmhosts | Get-VMHostService | Where-Object -FilterScript { $_.key -eq 'TSM-SSH' } | Start-VMHostService
$vmhosts | % {
    $esxcli = Get-EsxCli -VMHost $_ 
    $esxcli.system.hostname.set($null,$($_.name),$null) 
    Start-Sleep -Seconds 3 
    $esxcli.system.hostname.get()
}
$vmhosts | Get-VMHostService | Where-Object -FilterScript { $_.key -eq 'TSM-SSH' } | Stop-VMHostService

You can verify whether the change has helped quite easily, by visiting: https://your-loginsight/admin/hosts – the ‘Last Received Event’ for hosts without FQDNs simply growing greater and greater. After 12 hours, the ‘Average Active OSIs’ (as reported by https://your-loginsight/admin/license) had dropped slightly, but it’s got a way to go before being under 25.

PowerCLI 6.0 and PSModulePath

This article is to look at the problems with PSModulePath when installing PowerCLI 6.0 on Windows 10 (I’m aware that it is not yet certified for Win 10). The problem does not occur with the NetApp Data ONTAP PowerShell Toolkit (as noted at the bottom).

On a clean installation of Windows 10 we can observe the following behaviour with regard to PSModulePath:

[Environment]::GetEnvironmentVariable('PSModulePath') -split ';'
C:\Users\DefaultAdmin\Documents\WindowsPowerShell\Modules
C:\Program Files\WindowsPowerShell\Modules
C:\Windows\system32\WindowsPowerShell\v1.0\Modules\

[Environment]::GetEnvironmentVariable('PSModulePath','User') -split ';'

[Environment]::GetEnvironmentVariable('PSModulePath','Machine') -split ';'
C:\Windows\system32\WindowsPowerShell\v1.0\Modules\

The first command line translates to $env:PSModulePath. When PowerShell loads, it dynamically sets this session only variable equal to the following:

$home\Documents\WindowsPowerShell\Modules ($home is something like C:\Users\username)
$pshome\Modules ($pshome is the location of PowerShell, typically C:\Windows\System32\WindowsPowerShell\v1.0)

Based on the output above it also appends the contents of the system wide, SYSTEM environment variable for PSModulePath.

The second command line retrieves USER environment variable called PSModulePath (available to all users of the system), and as you can see there is none set. If there were one, it would prefixed to $env:PSModulePath (i.e. it would be the first path visible in the output of the first command).

The last command here retrieves the SYSTEM environment variable called PSModulePath. It is also present in the GUI here:

Capture

$env:PSModulePath is a rather important variable that is used not only for querying for available modules, but also module autoloading and tab completion of commands.

Sadly after manually installing PowerCLI (version 6.0.0-3056836) by running the exe as a local administrator as many would do, the behaviour of PowerShell on load is changed (some might call it broken!). Running the same query above we see:

[Environment]::GetEnvironmentVariable('PSModulePath') -split ';'
C:\Windows\system32\WindowsPowerShell\v1.0\Modules\

[Environment]::GetEnvironmentVariable('PSModulePath','User') -split ';'

[Environment]::GetEnvironmentVariable('PSModulePath','Machine') -split ';'
C:\Windows\system32\WindowsPowerShell\v1.0\Modules\
C:\Program Files (x86)\VMware\Infrastructure\vSphere PowerCLI\Modules

The SYSTEM environment variable for PSModulePath is modified, nothing is created for the USER environment variable for PSModulePath, and critically PowerShell no longer performs any customisation of $env:PSModulePath! The paths:

C:\Users\DefaultAdmin\Documents\WindowsPowerShell\Modules
C:\Program Files\WindowsPowerShell\Modules


are now gone.

This post on the VMware Community Forums 7 months ago seems to suggest the behaviour of PowerCLI on Windows 8.1 is different (or perhaps an older version of PowerCLI?):

The PowerCLI 6.0 installer does create a USER PSModulePath variable if one does not already exist.

… we can see at least on Windows 10, under the current version, it does not.

There is advice on the VMware PowerCLI blog under the v6 release which suggests:

#Save the current value in the $p variable.
$p = [Environment]::GetEnvironmentVariable("PSModulePath")

#Add the new path to the $p variable. Begin with a semi-colon separator.
$p += ";C:\Program Files (x86)\VMware\Infrastructure\vSphere PowerCLI\Modules\"

#Add the paths in $p to the PSModulePath value.
[Environment]::SetEnvironmentVariable("PSModulePath",$p)

The problem with this is that when calling SetEnvironmentVariable there is no target specified so it only updates the local session variable (i.e. $env:PSModulePath). This would therefore be required in every script (not ideal). Also we’ve established that installing PowerCLI has broken the population of this variable somehow. The best workaround is already in the comments; running the above code but targeting the SYSTEM variable for PSModulePath (i.e. [Environment]::SetEnvironmentVariable("PSModulePath",$p,'Machine')).

Now, if I repeat this process after installing the Netapp Data ONTAP PowerShell Toolkit (3.2.1.68) (accepting a UAC prompt), I get the perfect behaviour:

[Environment]::GetEnvironmentVariable('PSModulePath') -split ';'
C:\Users\DefaultAdmin\Documents\WindowsPowerShell\Modules
C:\Program Files (x86)\NetApp\Data ONTAP PowerShell Toolkit\
C:\Program Files\WindowsPowerShell\Modules
C:\Windows\system32\WindowsPowerShell\v1.0\Modules\

[Environment]::GetEnvironmentVariable('PSModulePath','User') -split ';'

[Environment]::GetEnvironmentVariable('PSModulePath','Machine') -split ';'
C:\Program Files (x86)\NetApp\Data ONTAP PowerShell Toolkit\
C:\Windows\system32\WindowsPowerShell\v1.0\Modules\
PS C:\Users\DefaultAdmin>

I think the appropriate clever people at VMware and NetApp need to have a chat with each other :)

Note: If I right click the PowerCLI installer and “Run as Administrator” I get the same behaviour.
Note: The Cisco UCS PowerTool also breaks the PSModule behaviour (using version CiscoUcs-PowerTool-1.5.1.0).

Catching Virtual Machine questions with PowerCLI

As part of our VM-template, template-VM conversion process at work (which I’ve automated via a webpage as discussed in my previous post “Executing Powershell using PHP and IIS“), I had to find a way to handle the VM question “This VM has questions that must be answered before the operation can continue” when attempting to power the VM on via Start-VM.

I found that handling the question was not possible using a simple try/catch block (the explanation can be found in this excellent post by Clint Bergman). I was however able to catch it using the following block:

try
{
	try
	{
		Start-VM -VM $serverName -ErrorAction Stop -ErrorVariable custErr
	}
	catch [System.Management.Automation.ActionPreferenceStopException]
	{
		throw $_.Exception
	}
}
catch [VMware.VimAutomation.ViCore.Types.V1.ErrorHandling.VMBlockedByQuestionException]
{
	Write-Output "Power on operation triggered a VMBlockedByQuestionException. Answering question with `"I moved it`". <br />"
	Get-VMQuestion -VM $serverName | Set-VMQuestion –Option "I moved it" -Confirm:$false	
}

It may seem like a lot of work to convert between a VM and template (and vice versa) but in our environment there are a couple of critical steps to the conversation process that must be followed to ensure deploys from these templates don’t break. As there are users outside of the team who update various templates I wanted to provide an easy way for everyone to ensure consistency when doing this, and the best way to do this is of course, automation :)