Accessing the Internet from PowerShell: Get-WebPage Part1

So last time we looked at accessing the web from PowerShell, we looked a simple function which returned our external IP address, we then used that function to make a script to automatically alert us when our external IP address changed. This was a pretty simple function and script, but I want to spend some time discussing some of this further and build a full CMDLet.

In later posts I want to discuss how I believe you should design your PowerShell code, including CMDLets, functions, scripts and modules, but I want to touch on some of this today.

In the get-externalip function, we have simply created the .Net framework webclient object, set up some headers, and the called the downloadstring method. This is fine for this simple script, but it isn't great design in the long term. What we need is a CMDLet that downloads a URL from the internet (or maybe a local source) and then make use of that CMDLet. The CMDLet should enable us to have a powerful framework where we can easily leverage all of the webclient class functionality, but in a refined and controlled PowerShell manner.

We should also ensure that we provide all the help and documentation with our CMDLets, so anyone can use them. For this we will use the PowerShell Comment Based Help Syntax, something that is an over looked part of PowerShell. Seriously, Microsoft should be encouraging more use of this, every language has something like similar to this, it’s not something special in PowerShell, but if every PowerShell developer used the comment based help syntax, the world would be a much better place.

Let’s take a look at a CMDLet that will get the HTML representation of a page, this CMDLet will also allow us to set things like the proxy server, headers, credentials for the remote page and the user agent!

The CMDLet below is based upon my Infrastructure Saturday 2012 presentation on PowerShell.

Whoa, quite a bit here, around 100 lines of code! Most of which though is documentation. As you can see, we have the start of the function definition, then the comment based help syntax. This syntax is providing documentation for the get-help CMDLet, including descriptions, parameters, inputs, outputs and examples.

Next we have the CMDLetBinding, we are not specifying anything extra here, just telling PowerShell that this is a CMDLet.

After this we have parameters, there all look pretty simple, only one is mandatory, URL. I have also specified types for all of the parameters. Whilst the specification of types is option, I am not doing this wherever possible, it reduces errors and provides very useful input validation.

Now we have the body of the CMDLet. As expected, we are making a new instance of the WebClient class. Then we have some if statements. If these optional parameters have been specified, then we need to set the various parts of the WebClient object.

I hard set the encoding, it makes things simple an easy.

We then create an object to hold the result, and within a try {} catch{} block will call download string. If we do catch any errors, simply throw them to the calling code. We don’t particularly care about handling errors here, the calling code should; however its crucial to handle errors in a controlled manner.

Finally return the resulting page.

Next time we will look at some examples and some further discussion of the code.

Update to Hashing Post

Just decided to make the cmdlet in the article: PowerShell File Hashing , a little more clear by renaming it to Get-FileHash, it makes more sense in the long run in my head, and explains what the CmdLet does.

Git to Azure

Just a quick guide on publishing web content to Windows Azure using Git!

First things first, make sure you have TortoiseGit and its dependencies installed, and create a new Windows Azure web site in the management console.

For this example, we will be publishing out own mirror of the http://isitup.org site. The sites code is available from GitHub here: https://github.com/sjparkinson/isitup

Let's begin!

Part 1 - Configuring Windows Azure for Git

1. Login to https://manage.windowsazure.com
2. Select the website you wish to use Git to publish to
3. Select "Setup Git publishing"
4. If you have never specified deployment credentials, select "reset deployment credentials" and specify a username and password
5. Note down the Git Clone URL

Part 2 - Pulling and Pushing with Git

1. In your local working copy folder, right click and select "Git Clone"
2. The URL will be "https://github.com/sjparkinson/isitup"
3. The directory will be the base URL for the Git working copy/repository for me "c:\users\defaultusername\workinprogress"
4. Hit ok
5. wait for a sucess message, and hit close
6. Right click on the isitup folder, select TortoiseGit, and then Push
7. select "Arbitary URL" enter the URL noted earlier
8. Hit OK
9. Enter your password
10. Wait for completion
11. DONE!!!

One I prepared earlier http://isitup.azurewebsites.net

PowerShell file hashing

So, I was reading the Microsoft Security Response Centres post on verifying your Windows Update files, available here, and saw their comments about using PowerShell to verify the upates. They used some code from Mike Wilbur's Blog, here, which is a nice simple little bit of code to get your SHA256 hashes for files.

I decided that the world needed something a bit more, and started to see if we could make a cmdlet that used the pipeline, and also checked the faile we were passing to the cmdlet actually existed. Whilst I was pokeing around in the .Net framework, I also noticed that SHA1, SHA384, SHA512, MD5 and RIPEMD160 was also available to us.

So I spent some time, and have developed the following piece of code:

To use the code, copy the code above into a .ps1 file, and then import-module yourfile.ps1, the function is called get-hash.

As you can see, I am using the comment based help, and there are some nice examples included.

Infrastructure Saturday 2012

I loved presenting at Infrastructure Saturday, as promised to you all, my presentations are available from the links below (and from the infrastructuresaturday.org website). 

PowerShell

Malware - What!

And as a bonus, my 2011 presentation!

Certificate Services

What is coming up:

• I will be talking about Web access and PowerShell
• Web Functions Library
• PushOver.Net and PowerShell
• Malware, malware, malware!
• new base template script with huge amount of extensibility for other notification types!