Accessing the Internet from PowerShell: Download Files Part 2

So, last time we looked at download a file, and looked at what a CMDLet to do this should look like.

And here it is:

Most of the layout of this CMDLet should be familiar to you. There is the comment based help, then the CMDLetBinding. We then have a list of parameters, URL is the only one which is mandatory, and the others are all optional. URL is also set to accept the value from the pipeline, this is the first step to meeting the first requirement of our CMDLet. The filename and directory parameters are for specifying the intended destination files name and location. Finally clobber is a switch, meaning it is either on or off. Clobber will be the way the user of the CMDLet tells it to overwrite files.

Next you will notice a Begin {} block of code. In here we are going to be creating the WebClient object and setting all of its options. Begin{} blocks will only get executed once, no matter how many items are passed in on the pipeline. Begin{} will occur before any input data is processed. End{} which isn’t used, is only executed once all the processing is completed.

A Process{} block is next. Process{} will be executed for every item passed in on the pipeline to the CMDLet. In this block we spend time working out the destination filename and path, then we will test to see if it exists. If the file already exists, then we can either, raise a warning that we are overwriting the file, or throw an error. This will be determined by the clobber parameter.

Finally we are going to write the full url and destination path to verbose output and download the file. We will throw any errors encountered back to the calling code.

Let’s look at how to use the cmdlet:

So, now you have it. You can download files.

Accessing the Internet from PowerShell: Download Files Part 1

Today we are going to download files from the internet, and other few places as a continuation of the series on the WebClient class in the .Net framework. The WebClient class has a method called DownloadFile, which as the name says, allows us to download files from many sources including HTTP, HTTPS, FTP and \\server\share locations. The aim will be by the end of this to have a CMDLet allowing us to download files from these various locations.

So, how does the DownloadFile method work? Here is an example:

The only issue with DownloadFile is that we need to specify a filename as the destination. This is something we will need to work around/with.

So if we wanted design a CMDLet around this, what should we consider? Firstly, let’s just ignore headers and encoding types and all that junk and assume we will handle that. What else is there? Well for a CMDLet that downloads files, we probably want to be able to use it in the pipeline, that is, I might want to pipe an array or list of URLs to that CMDLet and have it download them all. Next I don't want to have to specify the destination file name and folder all the time, but that doesn't mean I might not want to specify one or the other down the track. Finally, I hate downloading a file twice, or accidently overwriting a destination file. I want the CMDLet to not overwrite files unless I tell it to, and if it is, warn me about it.

So here are the requirements:

• ·         We want to be able to pipe URLs to the CMDLet so they can be downloaded
• ·         Destination filename optional. CMDLet to determine filename if none specified
• ·         Destination folder/directory optional. We might want to send a file to a different directory than the current working directory.
• ·         By default, do not overwrite files. If we tell the CMDLet to overwrite files, it should warn us

Well supporting the pipeline when writing a CMDLet is easy, we simply use the begin{} process {} end {} syntax. If we do things in a clever way, we should be able to reduce the amount of work we are doing for a large list of downloads.

The second requirement, optionally specifying the filename to download the file to...that can be a little more difficult! The trick is to have two parameters, filename and directory. Depending on what is or isn’t specified, we can do 4 different things.

1. 1.       If both are specified -> join the two together
2. 2.       If no filename or destination directory is specified -> the destination is the current directory (converted from .) joined with the "leaf" part of the URL
3. 3.       If no filename is specified, but a directory is -> the destination is the specified directory joined with the "leaf" part of the URL
4. 4.       If filename is specified but a directory is not -> The destination  is the current directory (converted from .) joined with the specified filename

The final requirement, not overwriting files by default, is also pretty simple, we can test to see if a file exist, if it doesn't then we are ok to download the file to the destination, otherwise we need a switch parameter, something like -clobber, to tell the CMDLet if its ok to overwrite the file. If it’s not ok, we will throw an error, if it is, we will use the write-warning to make sure we know that a file is being overwritten.

We will look at the cmdlet, next time.

Advanced Parameter Validation (revisiting get-filehash)

I am currently working on the rebuild of my template for all of my PowerShell scripts, trying to do things "better" this time around. I have learnt a few things over the last hour, one of which I don't understand (but everything works if I do it, so I will keep doing it), and another I just had to write about.

What is the great new thing? Well I wish i had known about it earlier, its the advanced parmater validation features.

Let's take a look at the first few lines of the get-filehash (original article, update) method I wrote about previously.

Now, I removed the comment based help syntax for brevity, but it is a little long, and there is the annoying enum defining the valid hash values to deal with as well, and then the section validating the file is a path etc. What if we could clean this all up?

Lets see:

Its a lot cleaner!

What we have is a ValidateScript, which is performing a test-path on the input. I am using the -pathtype 'leaf' to ensure we are hashing a file, not a folder. Then we have a ValidateSet, here we are saying, lets confirm the value for that parameter is in this array.

If either of these validations fail, we will get an error message along the lines of:

Get-FileHash : Cannot validate argument on parameter 'File'. The "Test-Path $_ -PathType 'leaf'" validation script for
the argument with value "D:\WIP\powershell\FileHash\filehash.jljdklf" did not return true. Determine why the
validation script failed and then try the command again.
At line:1 char:20
+ Get-FileHash -File D:\WIP\powershell\FileHash\filehash.jljdklf
+                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidData: (:) [Get-FileHash], ParameterBindingValidationException
    + FullyQualifiedErrorId : ParameterArgumentValidationError,Get-FileHash
Get-FileHash : Cannot validate argument on parameter 'File'. The "Test-Path $_ -PathType 'leaf'" validation script forthe argument with value "D:\WIP\powershell\FileHash\filehash.jljdklf" did not return true. Determine why thevalidation script failed and then try the command again.At line:1 char:20+ Get-FileHash -File D:\WIP\powershell\FileHash\filehash.jljdklf+                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~    + CategoryInfo          : InvalidData: (:) [Get-FileHash], ParameterBindingValidationException    + FullyQualifiedErrorId : ParameterArgumentValidationError,Get-FileHash

Not bad is it?

Another option to check out is ValidateRange. As you can guess, it allows you to validate the parameter against a range. An example would be [ValidateRange(1,10)], which will validate the paramater to ensure it is between 1 and 10! 

Here is the new get-filehash in full...

Note on PowerShell/Web Series

Just a quick note on the current series I am running on accessing the Internet from PowerShell. If you are looking at PowerShell V3, you will note the invoke-webrequest CMDLet. Whilst it is probably a million times more powerful than my humble set of web functions, it can be a little complicated. I am going to explore it over the coming weeks, and will report back my findings.

Accessing the Internet from PowerShell: Get-WebPage Part2

Welcome back.

So last time we looked at the get-webpage CMDLet, but first, you may notice that it is different from this version of this CMDLet to that I showed off at Infrastructure Saturday; your right, it has been updated. The update is to make supplying user-agent headers more easily.

Let’s go back to the get-externalip function we defined a few months ago. Previously it was:

Now if we make use of the cmdlet we just finished:

Pretty neat? 

I will leave it here, explore the cmdlet on your own.