Accessing the Internet from PowerShell: WebClient options

This will be very quick, as there isn't a lot in terms of options for the WebClient. When accessing the internet, there is often the need to change certain parameters to meet out needs.

Proxy Server Settings

The WebClient class is very smart, by default it will make use of your Internet Explorer/System proxy settings, thus your scripts which use this object will automatically get what ever proxy you have configured there. This is handy, as you will not have to rewrite any scripts you might have if you ever need to have them run from behind a (corporate) proxy server.

There has only been a few instances where I have needed to every change the proxy settings that were in use, and in that case I wanted to switch off any proxy support. To do this, simply set the proxy to $null

 

Headers

By default, there are a number of headers that .Net will put into a web request for us, we can overwrite those by adding entries to the headers property.

Some Examples include:

The first one is setting the content type for the request, this is important when we are submitting data to a web page, as we will see later.

The second one is setting the user-agent header, this one is basically what browser will be reported to the server when we connect to it. By Default, there will be now user-agent specified in the header, and this can cause us issues as a number of sites expect valid browser user-agent strings. A number of security products will also expect this (Especially cloudflare). Another reason to set something different is to make troubleshooting easier, all servers will log the agent-string, and if you are looking at your web server logs to determine why a script isn't working, then this will can be of assistance.

 

Encoding

There is a number of different ways to encode the content of a web page, if you are submitting content to a web page, check if it is expecting a certain encoding format. Most these days expect UTF8, to ensure the WebClient uses this, specify the following:

Now you know how to change some of the settings for the WebClient, you can start to make use of the client. Next time we will look at downloading web pages and files.

Accessing the Internet from PowerShell: Net.WebClient

The first thing we need to understand in order to make PowerShell talk with web content, is a .Net framework object, WebClient. WebClient is found in the System.Net namespace and provides us, the humble PowerShell developer with a simplified method for accessing HTTP/HTTPS/FTP content. Yes, you did just read that correctly, not only will our code be able to make use of HTTP(S) based content but it will also allow us to hit FTP Servers as well!

So lets get a PowerShell window open, and get an instance of the WebClient object. Simply make a new-object of type Net.WebClient and put it in a variable!

Congratulations, you have completed the first step.

Now lets take a look at what the WebClient offers us:

WOAH, that is quite a few methods and properties there.

For now, as I am just introducing the WebClient to you, lets just try to download a single page. Lets put the URL we want into a variable, and then call DownloadString, with a parameter of the URL variable:

The output, if everything has worked and your internet is running, will start with something like:

What the hell? that doesn’t look like Google.com does in Internet Explorer! Yes, that's because we will be seeing the un-rendered HTML, so things will be a little less friendly.

That is it for todays lesson, try downloading some pages yourself and see how you go. The next lesson will cover setting up some optional settings for WebClient such as controlling proxy server access and other context information.

Accessing the Internet from PowerShell

I had many potential titles for this series of posts, as yes, this is the first of a series. The aim of the series is to slowly guide you through interacting with web pages from within PowerShell, extending to downloading and uploading files, posting data to forms and interacting with WebDAV. My aim is to start by talking about each individual task, accessing a page, posting/uploading, download files, etc; cover off some of the issues we might encounter or what things to be on the look out for when working within a corporate environment (authentication and proxy servers) and finally bring everything together into 1) a working PasteBin scrapper that will look for the occurrences of certain works and notify us and 2) a PowerShell module which will allow us to quickly perform all of the above mention web/internet related tasks. I will provide a tonne of examples along the way to assist in helping you understand all of the different concepts that are involved.

WebDav on Apache

Sounds simple enough, but this is just an intro post for some other cool PowerShell related posts.

One thing I needed was a WebDav server, and well, why not use my handy little Qube server.

If you want to set up the Apache WebDav module (in this case on NetBSD) simply:

1. Ensure you have the apache DAV module installed
2. We need a folder for the DavLock database, i did the following:
   
3. Once you have the folders, add the following lines to your http.conf. Note this will assume that the folder webdav is stored wherever the rest of your htdocs are, so you need to make the folder or create a symbolic link from there to where you want WebDav files to be stored.
   
4. Restart/Reload the Apache Service
5. DONE!

Notes:

I should probably explain the lines we are entering in the httpd.conf file, as they will cause you some confusion if I don't.

The first line is specifying where we want the DavLockDB, in this case, we want the files to be called DavLock, in the folder we created earlier.

We then have the usual Apache location tags, in this case we want the location to be /webdav. Change this to suit your needs. You could also use Directory tags.

The next three lines are to do with authentication and authorisation. I only ever perform authentication with HTTP Digest (the first line), and I always where possible create a new authentication realm which Apache calls the AuthName. Finally we need to have a list of usernames and passwords, AuthUserFile will point to where we have a file of usernames and passwords setup in the correct method for Apache.

The file section, is something that was covered in the Apache documentation, and to be frank, I don't fully understand as the Apache configuration layout is still perplexing to me. What I know is that this line is limiting who can access the webdav location, in this case, valid-user means anyone who authenticates successfully and isn't a guest will have access. I am sorry I couldn't be any more helpful.

So now we have all of this sorted, I will be able to show you all soon, some of the cool PowerShell stuff I have been investigating. As a teaser, I have been looking at the various was PowerShell can integrate with web applications, as well as new methods for PowerShell to copy and save data that it creates.

I have also been working on some updates to my standard template, minor tweaks in formatting really, as well as some fun stuff around making the copy script I use every day an even more powerful and flexible tool.

PrimalScript 2012

There is a new version of PrimalScript 2012 available. As a long time user of PrimalScript for all of my scripting work, I was quite excited to see the latest version. 

From first impression, everything looks very polished and the Ribbion is very easy to use. Navigation is very quick and easy.

I am running the 30 day trial, apparently my upgrade assurance expired which is a bit of a shame, if there are no bugs, will definitely pay the small fee to upgrade to the latest version.

All in all, worth taking a look at, and still the best PowerShell IDE to date.