Catching Virtual Machine questions with PowerCLI

As part of our VM-template, template-VM conversion process at work (which I’ve automated via a webpage as discussed in my previous post “Executing Powershell using PHP and IIS“), I had to find a way to handle the VM question “This VM has questions that must be answered before the operation can continue” when attempting to power the VM on via Start-VM.

I found that handling the question was not possible using a simple try/catch block (the explanation can be found in this excellent post by Clint Bergman). I was however able to catch it using the following block:

try
{
	try
	{
		Start-VM -VM $serverName -ErrorAction Stop -ErrorVariable custErr
	}
	catch [System.Management.Automation.ActionPreferenceStopException]
	{
		throw $_.Exception
	}
}
catch [VMware.VimAutomation.ViCore.Types.V1.ErrorHandling.VMBlockedByQuestionException]
{
	Write-Output "Power on operation triggered a VMBlockedByQuestionException. Answering question with `"I moved it`". <br />"
	Get-VMQuestion -VM $serverName | Set-VMQuestion –Option "I moved it" -Confirm:$false	
}

It may seem like a lot of work to convert between a VM and template (and vice versa) but in our environment there are a couple of critical steps to the conversation process that must be followed to ensure deploys from these templates don’t break. As there are users outside of the team who update various templates I wanted to provide an easy way for everyone to ensure consistency when doing this, and the best way to do this is of course, automation :)

Executing PowerShell using PHP and IIS

This is an article on how to develop a PHP page to execute a PowerShell script on IIS 7.5 (7.0 is fine) as logged on user. In short, it makes use of shell_exec in PHP to launch PowerShell, grab the output and display it to the browser. It details the server setup as I have not tested this on other configurations, and provides the most basic possible PHP and PowerShell script for you to test it out. Hopefully some find this useful!

 

Introduction:

I’ve been using PowerShell fairly regularly at work for about a year now and it’s fantastic. Many tasks that our team have to perform are now scripted using PowerShell, and we’re actively trying to turn away from VBS. Given the comparative ease of this language and its integration with the .NET framework, we started toying with the idea of being able to provide an IIS website front end to our service desk to execute PowerShell scripts behind the scenes.

Essentially we wanted a website that would allow for an Active Directory user to login over HTTPS, enter some data into a web page, submit the data to a digitally signed PowerShell script as parameters and have it execute on the very same server as the logged on user, finally returning the result back to the web page. Initially the solution needed to execute PowerShell scripts that would make at least make use of the Microsoft Active Directory Module and VMware’s PowerCLI Snap-in, later expanding to utilise the Exchange 2010 and Netapp DataONTAP snap-ins.

After initially looking at a Perl wrapper for executing PowerShell with a colleague and making little progress, I decided to have a shot at doing it with PHP. I have a little experience with PHP and within 30 minutes or so I had it working using shell_exec. Game on!

 

Server Configuration:

The working solution is based on the following platform:

  • Windows Server 2008 R2 with IIS 7.5.
  • PowerShell v2 (the default on the above OS).
  • .NET 4 (not necessary but this is our setup).
  • PHP (x64 preferred, though x86 will work – discussed below).
  • Visual C++ 2008 SP1 Redistributable Package – choose the package to match the version of PHP (x64) (x86).
  • Active Directory Web Services installed on one of our 2008 DCs. This allows for the use of the Active Directory module (cmdlets) on a 2008 R2 or Windows 7 machine in a non 2008 R2 domain controller environment.

I won’t cover securely signing PowerShell scripts here as it’s widely documented. If you wish to do this, it’s useful to remember to run Set-ExecutionPolicy on both versions:

  • C:\Windows\SysWOW64\WindowsPowerShell\v1.0\Powershell.exe
  • C:\Windows\System32\WindowsPowerShell\v1.0\Powershell.exe

1. Install IIS with the following:

  • CGI
  • Basic Authentication
  • Recommended: URL authorization (if you want to limit access to particular Active Directory users or groups).
  • Optional: IP and Domain Restrictions (if you want to limit access to a particular IP range on your LAN).
  • Install any Windows patches that may be needed by adding IIS to your server.

2. Install PHP:
Note: PHP.net only compile and officially support x86 versions of PHP. Unfortunately this means that the x86 version of PowerShell is launched and may prove a limitation for you; in our case we wanted to load Exchange 2010 snap-ins which require the x64 version of PowerShell, and so needed x64 PHP. Anindya over at http://www.anindya.com/ very kindly compiles these as x64 when PHP releases a new version.

  1. Download and extract the latest “VC9 x64 Non Thread Safe” version of PHP from http://www.anindya.com/. Put this somewhere you want to keep PHP (e.g. F:\PHP).
  2. UPDATE: PHP.net now do x64 builds. Use the latest x64 non thread safe version from here: http://windows.php.net/download/
  3. Install PHP Manager from http://phpmanager.codeplex.com/.
  4. Add the location of PHP (e.g. F:\PHP;) to the Path environment variable.

3. Configure IIS:

  1. Within IIS Manager, right click on “Sites” and choose “Add Web Site…”. When trying to think of an acronym for the test website I came up with WAM (Web Automated Management) and that stuck. Specify your website name, a physical path (e.g. C:\inetpub\wwwroot\wam), hostname and IP address binding.
  2. Select the website and within the Authentication module for the new site, enable Basic and disable the Anonymous authentication. Remember, if your website is not secured using HTTPS, passwords will be sent over the network in plain text. This may not worry you but I need to point it out. There are many guides online on how to create a self signed SSL certificate for your IIS server.
  3. Again within the website, select the PHP Manager module and click “Register new PHP version” and locate your php-cgi.exe executable.

4. Configure PHP:

  1. Open PHP.ini. You will likely have to change some of these depending on your setup. I’ve described what I changed by specifying the initial value which you can search for, then the pipe symbol and the new value (e.g. search for | modify to):
    • error_log = “C:\Windows\Temp\php-5.4.3_errors.log” | error_log = “F:\php-errors\php-5.4.3_errors.log”
    • max_execution_time = 300 | max_execution_time = 600
    • display_errors = Off | display_errors = On
    • date.timezone = “Europe/Minsk” = | date.timezone = Europe/London
  2. Grant the local IIS_IUSRS group modify NTFS permission on the log folder for the PHP error log.

 

Writing the PHP Page:

Now, our PHP pages are a little more complicated than the example I’m going to give here. We only display certain pages to certain groups of users. We validate our submitted data using PHP functions. We return information to the screen if required fields aren’t filled in before submitting the form. We use jQuery mobile to present a mobile version to mobile users (I love this ability). The list goes on. You can make your PHP as advanced (but user friendly!) as you want; that’s the fun in developing websites :) Here I’ll just show how you can write an extremely simple page to pass the data to a PowerShell script. If you want to take things into a production environment I suggest you expand on this as we have done, but this serves as a proof of concept :)

Example PHP Page:

Let’s call this page “get-process.php”, and save it into the root of the created website:

&lt;!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd"&gt;
&lt;html&gt;
&lt;head&gt;
&lt;title&gt;Testing PowerShell&lt;/title&gt;
&lt;/head&gt;
&lt;body&gt;
&lt;?php

// If there was no submit variable passed to the script (i.e. user has visited the page without clicking submit), display the form:
if(!isset($_POST["submit"]))
{
?&gt;
&lt;form name="testForm" id="testForm" action="get-process.php" method="post" /&gt;
Your name: &lt;input type="text" name="username" id="username" maxlength="20" /&gt;&lt;br /&gt;
&lt;input type="submit" name="submit" id="submit" value="Do stuff" /&gt;
&lt;/form&gt;
&lt;?php
}
// Else if submit was pressed, check if all of the required variables have a value:
elseif((isset($_POST["submit"])) &amp;&amp; (!empty($_POST["username"])))
{
// Get the variables submitted by POST in order to pass them to the PowerShell script:
$username = $_POST["username"];
// Best practice tip: We run out POST data through a custom regex function to clean any unwanted characters, e.g.:
// $username = cleanData($_POST["username"]);

// Path to the PowerShell script. Remember double backslashes:
$psScriptPath = "F:\\get-process.ps1";

// Execute the PowerShell script, passing the parameters:
$query = shell_exec("powershell -command $psScriptPath -username '$username'&lt; NUL");
echo $query;
}
// Else the user hit submit without all required fields being filled out:
else
{
echo "Sorry, you did not complete all required fields. Please go back and try again.";
}
?&gt;
&lt;/body&gt;
&lt;/html&gt;

 

Writing the PowerShell Script:

Call this get-process.ps1. Save it to the location specified in the above PHP script (it can be anywhere you like, but I recommend they are outside the website root. We actually store ours on a different drive). Naming conventions are useful and in this instance it’s handy to use the same name for both PHP and PS1 script for troubleshooting.

#*=============================================================================
#* Script Name: get-process.ps1
#* Created: 	2012-01-01
#* Author: 	Robin Malik
#* Purpose: 	This is a simple script that executes get-process.
#*
#*=============================================================================

#*=============================================================================
#* PARAMETER DECLARATION
#*=============================================================================
param(
[string]$username
)
#*=============================================================================
#* REVISION HISTORY
#*=============================================================================
#* Date:
#* Author:
#* Purpose:
#*=============================================================================

#*=============================================================================
#* IMPORT LIBRARIES
#*=============================================================================

#*=============================================================================
#* PARAMETERS
#*=============================================================================

#*=============================================================================
#* INITIALISE VARIABLES
#*=============================================================================
# Increase buffer width/height to avoid PowerShell from wrapping the text before
# sending it back to PHP (this results in weird spaces).
$pshost = Get-Host
$pswindow = $pshost.ui.rawui
$newsize = $pswindow.buffersize
$newsize.height = 3000
$newsize.width = 400
$pswindow.buffersize = $newsize

#*=============================================================================
#* EXCEPTION HANDLER
#*=============================================================================

#*=============================================================================
#* FUNCTION LISTINGS
#*=============================================================================

#*=============================================================================
#* Function: 	function1
#* Created: 	2012-01-01
#* Author: 	My Name
#* Purpose: 	This function does X Y Z
#* =============================================================================

#*=============================================================================
#* END OF FUNCTION LISTINGS
#*=============================================================================

#*=============================================================================
#* SCRIPT BODY
#*=============================================================================
Write-Output "Hello $username &lt;br /&gt;"

# Get a list of running processes:
$processes = Get-Process

# Write them out into a table with the columns you desire:
Write-Output "&lt;table&gt;"
Write-Output "&lt;thead&gt;"
Write-Output "	&lt;tr&gt;"
Write-Output "		&lt;th&gt;Process Name&lt;/th&gt;"
Write-Output "		&lt;th&gt;Id&lt;/th&gt;"
Write-Output "		&lt;th&gt;CPU&lt;/th&gt;"
Write-Output "	&lt;/tr&gt;"
Write-Output "&lt;/thead&gt;"
Write-Output "&lt;tfoot&gt;"
Write-Output "	&lt;tr&gt;"
Write-Output "		&lt;td&gt;&amp;nbsp;&lt;/td&gt;"
Write-Output "		&lt;td&gt;&amp;nbsp;&lt;/td&gt;"
Write-Output "		&lt;td&gt;&amp;nbsp;&lt;/td&gt;"
Write-Output "	&lt;/tr&gt;"
Write-Output "&lt;/tfoot&gt;"
Write-Output "&lt;tbody&gt;"
foreach($process in $processes)
{
Write-Output "	&lt;tr&gt;"
Write-Output "		&lt;td&gt;$($process.Name)&lt;/td&gt;"
Write-Output "		&lt;td&gt;$($process.Id)&lt;/td&gt;"
Write-Output "		&lt;td&gt;$($process.CPU)&lt;/td&gt;"
Write-Output "	&lt;/tr&gt;"
}
Write-Output "&lt;/tbody&gt;"
Write-Output "&lt;/table&gt;"
#*=============================================================================
#* END SCRIPT BODY
#*=============================================================================

#*=============================================================================
#* END OF SCRIPT
#*=============================================================================

 

Use Cases:

That’s all there is to it! You can visit the webpage, enter your name and submit the form. What you do from here is really up to you and only limited by your talent and imagination :) We’ve created pages that save an incredible amount of work for both us and the service desk and so finally, some examples on how we use them which may (or may not) inspire you.

It’s worth noting here that where applicable, the PowerShell scripts also log this data to our “changelog” website (a bespoke system/website to record changes made to servers/devices) via an API call using System.Net.WebClient. This saves a *lot* of fiddly manual work, remembering to record the changes after you’ve made them(!) and allows for a consistent method of recording changes for specific actions.

Just some of the pages we have in use at the moment allow for:

  • Creation of different kinds of AD accounts to be created (service accounts, admin accounts, remote accounts).
  • Resetting a user’s password (generating a new one).
  • Retrieval of detailed information on an AD user (e.g. name, phone number, email, true last logon, locked out status, enabled status, Exchange email quota, mailbox size, home drive usage etc.).
  • Adding/removing users to/from AD groups or local groups on computers/servers.
  • Retrieval of detailed server information (installed software, local admins/remote desktop users, configured IP addresses, accessible shares etc.).
  • Granting full access and send as permissions on an Exchange mailbox to another user.
  • Increasing a user’s email quota.
  • Resetting the permissions on a user’s home drive (calling setacl.exe).
  • Changing of a user’s display name.

 

Caveats:

  • Obviously the webpage is going to wait while the PowerShell script is executing. I’ve had to increase the max_timeout value in php.ini from 300 to 600 to allow for 10 minutes of PHP execution time before quitting. I encounted this when resetting the permissions on a user’s home drive that was *quite large*. If the PHP timeout value is reached, you’ll receive an error 500. The PowerShell process will continue to run and complete, but it’s nice to avoid this.
  • If you want your displayed output to be nice, you will need to write HTML within PowerShell. In the above PowerShell script you will notice use of HTML so that when PHP displays the result we get a table.
  • If you’re considering using Start-Transcript to log within your PowerShell script, it doesn’t work. Launching PowerShell via shell_exec *within a PHP page* (not from the PHP command line) will result in a transcript file that has a header, footer, but no content.

Importing VEM500-201201140102-BG-release.zip to VUM results in an error

In order to install cross_cisco-vem-v140-4.2.1.1.5.1.0-3.0.1.vib to our vCenter 5 / ESXi 5 / Cisco 1000V environment we attempted to add the following VEM to VMware Update Manager (VUM): VEM500-201201140102-BG-release.zip (extracted from Nexus1000v.4.2.1.SV1.5.1.zip). Unfortunately on first attempt we received an error along the lines of “invalid vendor code CSCO in patch metadata, another vendor code with different code already exists…” (I didn’t save the error and only googled it in partiality so I can’t copy it here I’m afraid). In short, it stopped us from importing the patch.

This was solved by extracting the VEM500-201201140102-BG-release.zip, opening index.xml (wordpad is suitable) and modifying the code XML node from CSCO to csco. We then rezipped the folder contents and imported it successfully.

Thanks to http://www.thatcouldbeaproblem.com/?p=341 for the nod in the right direction :)

Speeding up Get-WinEvent in Powershell by using FilterHashTable

At work I’ve been working on a website to collate various bits of reporting information about our infrastructure. I wanted one of these reports to be selected eventlog entries of our servers, split up by WSUS patch phases (i.e. one report per AD security group). The idea behind this was that we could arrive at work in the morning knowing that Phase X had patched overnight and take a look at a jQuery sortable HTML table showing any issues with servers in that phase/group. If this was quick enough, we could even execute it live against a single server via another website I’ve been working on (I’ll post about that later). I also want to say that while have a Solarwinds monitoring solution (APM) and their Windows based log forwarder application, the forwarded eventlogs are sent to a SQL database as syslog messages which simply don’t have the same level of detail as a Powershell event object. Anyway back on topic.

Two approaches sprang to mind:

  • Invoke-Command by way of WinRM
  • The remote capabilities of either Get-WinEvent or Get-EventLog

Rolling out WinRM is an ongoing project and so the latter it was. A read around online and Get-WinEvent was touted a quickest especially when querying remote computers so I started with that. I constructed my query to retrieve any errors or critical messages in the application eventlog since 4am which is the time of patching:

$computerName = "remoteserver"
# Create a timestamp after which to retrieve events. This should be from 4am on the current day:
$currDatetime = Get-Date
$day = $currDatetime.Day
$month = $currDatetime.Month
$year = $currDatetime.Year
$patchDateTime = New-Object -TypeName System.DateTime($year,$month,$day,04,00,00)
$appLog = Get-WinEvent -LogName "Application" -ComputerName $computerName -ErrorAction SilentlyContinue | Where-Object { ( ($_.LevelDisplayName -eq "Error" -or $_.LevelDisplayName -eq "Critical") -and ($_.TimeCreated -ge $patchDateTime) ) } | Select-Object TimeCreated,LogName,ProviderName,Id,LevelDisplayName,Message

It took forever to run. Ok not quite, but it took 11 minutes! Very slow.

“Back to basics” I thought. Let’s test the execution time of a simpler query on my local machine:

$yesterday = (get-date) - (new-timespan -day 1)
Measure-Command -Expression { Get-WinEventLog -LogName "Application" -ErrorAction SilentlyContinue | Where-Object { ($_.TimeCreated -ge $yesterday) -and ($_.LevelDisplayName -eq "Error") } }

22 seconds for 1 result (out of 104 records in the last 24 hours). Hmmm.

How about against the original server?:

Measure-Command -Expression { Get-WinEventLog -LogName "Application" -ErrorAction SilentlyContinue | Where-Object { ($_.TimeCreated -ge $yesterday) -and ($_.LevelDisplayName -eq "Error") } }

330 seconds (5.5 minutes) for 2 results (out of 128 records in the last 24 hours). Very disappointing. We have 140 servers and querying them all was not going to happen using this approach and standard sequential / synchronous processing. While it would be possible to save time by using asynchronous processing (background jobs or multiple threads) I was certain that this command should be orders of magnitude faster.

I did a little more reading and discovered the -FilterHashTable parameter of the Get-WinEvent cmdlet. This filters the objects while being retrieved on the server, rather than retrieving all the objects and then filtering as happens with Where-Object.

Get-Help Get-WinEvent -Parameter FilterHashTable

showed the key:value pairs accepted by the parameter. The user friendly “LevelDisplayName” key was not one of these, but luckily “Levels” of events (error,warning,information etc.) are also given the “Level” property which is accepted by FilterHashTable. “Error” = Level 2. If you want to know how to see this sort of information the easiest way is to double click an eventlog entry, click the “details” tab and then select XML view.

A new, equivalent query was born using this new method and executed against my local computer:

Measure-Command -Expression { Get-WinEvent -FilterHashTable @{LogName='Application'; Level=2; StartTime=$yesterday} -ErrorAction SilentlyContinue }

0.09 seconds! Some 244x faster. I stripped off the Measure-Command to find it was indeed pulling back the same single error record as previously (without -FilterHashtable). Fantastic!

Now against the remote server:

Measure-Command -Expression { Get-WinEvent -FilterHashTable @{LogName='Application'; Level=2; StartTime=$yesterday} -ErrorAction SilentlyContinue -ComputerName $remoteserver }

0.43 seconds.

So back to my original command. As we’re not allowed duplicate keys in a hash table two queries were needed:

$appErrors = Get-WinEvent -FilterHashTable @{LogName='Application'; Level=2; StartTime=$yesterday} -ErrorAction SilentlyContinue -ComputerName $remoteserver | Select-Object TimeCreated,LogName,ProviderName,Id,LevelDisplayName,Message
$appCritical = Get-WinEvent -FilterHashTable @{LogName='Application'; Level=1; StartTime=$yesterday} -ErrorAction SilentlyContinue -ComputerName $remoteserver | Select-Object TimeCreated,LogName,ProviderName,Id,LevelDisplayName,Message
# Combine and sort the arrays
$appCombined = $appErrors + $appCritical | Sort TimeCreated

1.85 seconds. Outstanding :)

I then went on my merry way about creating a fully fledged script. Ultimately it worked very well and against a batch of 20 servers took 3.89 minutes with some 18,000 records returned, although it did complain with many “There are no more endpoints available from the endpoint mapper.” error messages which I noticed were against our 2003 servers. It appears that Get-WinEvent doesn’t work against 2003 but at least from a 2008R2 server it happily works against 2008+ with -FilterHashTable. I may have to construct a different, equivalent Get-EventLog query purely for the 2003 servers.