Definition of PowerShell Invoke-Webrequest
In PowerShell Invoke-WebRequest cmdlet which is a part of Microsoft.PowerShell.Utility Module is one of the Web scrapping methods (the other method is Invoke-RestMethod and in cmd and other OS it is known as CURL) which sends HTTP and HTTPS request to the requested webpage or URI which is called Endpoint and retrieves, deletes, updates content in the form of Headers, Images, Links and other HTML significant elements and for that it uses the multiple methods like GET, POST, PUT, DELETE, etc to send the request, modify the content or to delete the content of the webpage. Invoke-WebRequest is also known to work with REST API.
Syntax:
Invoke-WebRequest
[-UseBasicParsing]
[-Uri] <Uri>
[-WebSession <WebRequestSession>]
[-SessionVariable <String>]
[-AllowUnencryptedAuthentication]
[-Authentication <WebAuthenticationType>]
[-Credential <PSCredential>]
[-UseDefaultCredentials]
[-CertificateThumbprint <String>]
[-Certificate <X509Certificate>]
[-SkipCertificateCheck]
[-SslProtocol <WebSslProtocol>]
[-Token <SecureString>]
[-UserAgent <String>]
[-DisableKeepAlive]
[-TimeoutSec <Int32>]
[-Headers <IDictionary>]
[-MaximumRedirection <Int32>]
[-MaximumRetryCount <Int32>]
[-RetryIntervalSec <Int32>]
[-Method <WebRequestMethod>]
[-Proxy <Uri>]
[-NoProxy]
[-CustomMethod <String>]
[-ProxyCredential <PSCredential>]
[-ProxyUseDefaultCredentials]
[-Body <Object>]
[-Form <IDictionary>]
[-ContentType <String>]
[-TransferEncoding <String>]
[-InFile <String>]
[-OutFile <String>]
[-PassThru]
[-Resume]
[-SkipHttpErrorCheck]
[-PreserveAuthorizationOnRedirect]
[-SkipHeaderValidation]
[<CommonParameters>]
How does the Invoke-WebRequest method work in PowerShell?
From PowerShell v3.0 onwards PowerShell has added cmdlet Invoke-WebRequest which ships with Microsoft.PowerShell.Utility module to work with WebPages. We can retrieve, remove or update the content of the webpages using PowerShell, apparently this will not be the same experience as page navigation using GUI but it is to automate the web requests and get rid of tedious browsing and manual daily tasks.
Invoke-WebRequest sends a request to the URI (Uniform Resource Identifier) which is also called Endpoint and retrieves the data from the Web Page. It directly works with the URL or with the REST API because some websites allow modifying the content using only APIs.
When we work with the APIs, the output data is generally in JSON or XML format for example,
https://yts.mx/api/v2/list_movies.json
or
https://yts.mx/api/v2/list_movies.xml
In the above case, once the content is retrieved we can convert the content to JSON or XML method using PowerShell.
Examples
Let us discuss examples of PowerShell Invoke-Webrequest.
a.Invoke-WebRequest to retrieve the data from the Webpage.
We can provide the URI to retrieve the content from the webpage. There is a Content and RawContent Property, you can use those properties to retrieve the data. It depends on the webpage, what kind of data we are retrieving. Data can be in HTML format or can be in JSON or XML format.
$req = Invoke-WebRequest -Uri https://theautomationcode.com/feed/
$req
\
Once you run the above command you can see the different Properties like StatusCode, StatusDescription, Content, RawContent, Forms, Headers, Images, InputFields, Links, ParsedHtml, RawContentLength, etc.
You can also check the properties of this command using Get-Member command.
$req | gm -MemberType Properties
Output:
To retrieve the content,
$req = Invoke-WebRequest -Uri "https://theautomationcode.com/feed/"
$req.Content
b.Invoke-WebRequest to Check the Website Status code.
When a client sends the request to the endpoint, it sends the handshake status with the various status codes which shows the website availability or any error on the webpage.
You can find various codes with their status message below the link.
https://en.wikipedia.org/wiki/List_of_HTTP_status_codes
To retrieve the status messages using Invoke-WebRequest we need to use the StatusCode as a reference property. For example,
try{
$uri = "http://theautomationcode.com"
$response = Invoke-WebRequest -Uri $uri -Method Get -ErrorAction Stop
Write-Output "Status Code : $($response.statuscode)"
}
catch{
Write-Output "Status Code : $($_.Exception.Response.StatusCode.Value__)"
}
Output:
If we mention a website that does not exist, it will give a different status code. For the below URI,
$uri = "http://theautomationcode.com/failed"
Output:
c. Invoke-WebRequest to retrieve images and Links from the Webpage.
We can retrieve images and Link from the webpage using Invoke-WebRequest command. To retrieve images from the webpage and store them to the specific location us the below command,
try{
$wc = New-Object System.Net.WebClient
$uri = "http://theautomationcode.com"
$imgs = (Invoke-WebRequest -Uri $uri -ErrorAction Stop).Images
$count = 0
foreach($imgurl in $imgs.src){
$wc.DownloadFile($imgurl,"C:\temp\WebImages\img$Count.jpg")
$count++
}
}
catch{
Write-Output "Status Code : $($_.Exception.Response.StatusCode.Value__)"
}
Output:
Invoke-WebRequest to get links from the WebPage.
$uri = "http://theautomationcode.com"
$response = Invoke-WebRequest -Uri $uri
$response.Links.href
Output:
d.Working With Forms in Invoke-Webrequest.
Forms are the pages that stores the information about Input items such as username and Password. We can use forms to submit the username and the password using the POST method.
For example, We will use the Reddit login page and submit credentials there. To retrieve the login page we have URL below for Reddit.
$Response = Invoke-WebRequest https://www.reddit.com/login/ -SessionVariable sv
$Response.Forms
Here we have used Forms Property to retrieve our needed information.
Once you run the above command, you can see the POST method there and we will use it.
$Response = Invoke-WebRequest https://www.reddit.com/login/ -SessionVariable sv
$form = $Response.Forms[2]
$form.Fields['loginUsername'] = 'reddit'
$form.Fields['loginPassword'] = 'redditpass'
Invoke-WebRequest $Response -Body $form -WebSession $sv -Method Post
The above script is an example of the Web Forms and POST method.
e. Invoke-WebRequest to get the feeds from the webpage.
Sometimes we just need to retrieve the content of the webpage. For example, Feeds from the webpage might be in the JSON or XML format. In the below example, we will retrieve Reddit RSS feeds from the URL https://www.reddit.com/.rss then later convert them into XML or JSON format to retrieve the feeds in the proper format in the console. For example,
$response = Invoke-WebRequest "https://www.reddit.com/.rss" -Method Get
$response.Content
Once we run the above command, it retrieves the XML output as shown below.
$response = Invoke-WebRequest "https://www.reddit.com/.rss" -Method Get
$feeds = [XML]$response.Content | Select -ExpandProperty Feed
$feeds.Entry | Select Title, Updated
Output:
Conclusion
Invoke-Webrequest is a very useful command to automate web requests and for working with the APIs. There are many free APIs like weather API, movies database, finance APIs that allow integrating with the web request, and in that case, we can retrieve, update, post, or delete the data from the requested APIs.
Recommended Articles
This is a guide to PowerShell Invoke-Webrequest. Here we discuss Definition, How does the Invoke-WebRequest method work in PowerShell? and examples with code implementation. You may also have a look at the following articles to learn more –
2 Online Courses | 1 Hands-on Project | 4+ Hours | Verifiable Certificate of Completion
4.5
View Course
Related Courses
