Accessing the Internet from PowerShell: Getting Page Content

In this post, I am going to cover some more of the DownloadString() method of the .Net WebClient that we briefly touched on in my earlier post.

I am going to show you a function that will return your current external/public IP address.

There a numerous services out there that allow a user to get their external IP Address, most have two flavours, one which will return a nicely formatted page with text like “Your IP address is 123.123.123.123” whilst others are designed for automation and will simply return a plain text formatted page with only the IP address 123.123.123.123. Of course, you could make your own service and host it somewhere yourself. It is very simple to make your own page which can do this with a single line of C# in a ASP.Net page (.ASPX),  C# code in a Web Event Handler (.ASHX), PHP page or even classic ASP (.ASP) page.

A later post will cover making your own page, for today, we will test with one of the already existing services listed below:

• http://whatismyip.com –> These guys provide a good service, they make use of a separate automation page located at http://automation.whatismyip.com/n09230945.asp
• http://icanhazip.com/ 
• http://whatismyip.Akamai.com/ 
• http://b10m.swal.org/ip

WARNING NOTES

1. Some malware uses these sites to work out your IP address, some proxy servers and antivirus solutions may block access to their site.
2. Don’t hit this page more than once every 300s or you might get blocked.

The Function

Let’s start by writing a function that will perform the following:

1. Take an optional value of the URL, this will be the URL of a service which returns our external IP address as plain text. Like any of the ones listed above. If we don’t specify one, use icanhazip.com
2. Set the UserAgent to be that of an Internet Explorer User
3. Call the DownloadString function with the URL

Let’s take a look at a function which would return our current IP address:

So we can see this is a pretty standard PowerShell Function. The function starts off with a param definition, there is a single parameter $ipcheckurl, for once I am being a nice programmer and specifying the type as a String. This is an optional parameter,  and if there is no value for this parameter provided we will use the value of http://icanhazip.com .

The body of the function is pretty simple. Let’s make a WebClient object, then we will set the UserAgent header to that of a Windows Internet Explorer UserAgent, and finally, we call the DownloadString method, specifying our URL and return it to the calling code.

That’s all pretty simple right?

Before we continue, notice I didn’t add any error handling? That is because if anything goes wrong, I want the calling code to know that something has gone wrong. anything calling this will need to handle any errors that might occur (for example, page not found, name could not be resolved, server busy etc).

The Script

Now lets look at a script we could use to monitor our external IP address, it will perform the following:

1. Read a file containing the previously recorded external IP address, if non exists, the script should continue.
2. Call our previously defined function and store the returned IP address in a variable. We will also safely handle any errors that occur.
3. Compare the two, if they match, then the external IP address hasn’t changed, we don’t need to do anything, but we might want to. If they don’t match, then the external IP address has changed and we should make a fuss!
4. If the address has changed, we need to store the IP address back into the file.

Lets Take a look:

I am not going to go into much detail, this is a pretty simple script. I am assuming that the function that was previously discussed is available, you could put it into either a module or just defined in the script file.

So, now you know how to watch for your external IP address changing!

Accessing the Internet from PowerShell: WebClient options

This will be very quick, as there isn't a lot in terms of options for the WebClient. When accessing the internet, there is often the need to change certain parameters to meet out needs.

Proxy Server Settings

The WebClient class is very smart, by default it will make use of your Internet Explorer/System proxy settings, thus your scripts which use this object will automatically get what ever proxy you have configured there. This is handy, as you will not have to rewrite any scripts you might have if you ever need to have them run from behind a (corporate) proxy server.

There has only been a few instances where I have needed to every change the proxy settings that were in use, and in that case I wanted to switch off any proxy support. To do this, simply set the proxy to $null

 

Headers

By default, there are a number of headers that .Net will put into a web request for us, we can overwrite those by adding entries to the headers property.

Some Examples include:

The first one is setting the content type for the request, this is important when we are submitting data to a web page, as we will see later.

The second one is setting the user-agent header, this one is basically what browser will be reported to the server when we connect to it. By Default, there will be now user-agent specified in the header, and this can cause us issues as a number of sites expect valid browser user-agent strings. A number of security products will also expect this (Especially cloudflare). Another reason to set something different is to make troubleshooting easier, all servers will log the agent-string, and if you are looking at your web server logs to determine why a script isn't working, then this will can be of assistance.

 

Encoding

There is a number of different ways to encode the content of a web page, if you are submitting content to a web page, check if it is expecting a certain encoding format. Most these days expect UTF8, to ensure the WebClient uses this, specify the following:

Now you know how to change some of the settings for the WebClient, you can start to make use of the client. Next time we will look at downloading web pages and files.

Accessing the Internet from PowerShell: Net.WebClient

The first thing we need to understand in order to make PowerShell talk with web content, is a .Net framework object, WebClient. WebClient is found in the System.Net namespace and provides us, the humble PowerShell developer with a simplified method for accessing HTTP/HTTPS/FTP content. Yes, you did just read that correctly, not only will our code be able to make use of HTTP(S) based content but it will also allow us to hit FTP Servers as well!

So lets get a PowerShell window open, and get an instance of the WebClient object. Simply make a new-object of type Net.WebClient and put it in a variable!

Congratulations, you have completed the first step.

Now lets take a look at what the WebClient offers us:

WOAH, that is quite a few methods and properties there.

For now, as I am just introducing the WebClient to you, lets just try to download a single page. Lets put the URL we want into a variable, and then call DownloadString, with a parameter of the URL variable:

The output, if everything has worked and your internet is running, will start with something like:

What the hell? that doesn’t look like Google.com does in Internet Explorer! Yes, that's because we will be seeing the un-rendered HTML, so things will be a little less friendly.

That is it for todays lesson, try downloading some pages yourself and see how you go. The next lesson will cover setting up some optional settings for WebClient such as controlling proxy server access and other context information.

Accessing the Internet from PowerShell

I had many potential titles for this series of posts, as yes, this is the first of a series. The aim of the series is to slowly guide you through interacting with web pages from within PowerShell, extending to downloading and uploading files, posting data to forms and interacting with WebDAV. My aim is to start by talking about each individual task, accessing a page, posting/uploading, download files, etc; cover off some of the issues we might encounter or what things to be on the look out for when working within a corporate environment (authentication and proxy servers) and finally bring everything together into 1) a working PasteBin scrapper that will look for the occurrences of certain works and notify us and 2) a PowerShell module which will allow us to quickly perform all of the above mention web/internet related tasks. I will provide a tonne of examples along the way to assist in helping you understand all of the different concepts that are involved.