Part 3 Intro
In Part 1, I covered the primary changes in the actual code base of the PowerShell Core Web Cmdlets Invoke-RestMethod and Invoke-WebRequest and how those changes manifest themselves in the PowerShell user experience.
In Part 2, I covered outstanding issues as well missing and/or deprecated features.
In Part 3, I will cover new features available in PowerShell Core 6.0.0 Invoke-RestMethod & Invoke-WebRequest. I will also cover future plans for the cmdlets.
If you have not read Part 1 and Part 2, please do so before reading Part 3. This blog series goes in depth and requires a great many words to do so. To save space I will not repeat some information and will assume the reader as read Part 1 and Part 2.
A quick bit of news: PowerShell Core v6.0.0-rc.2 was released. Unless any blocking issues are discovered, this will be the final RC release and the next release will be GA in January 2018.
Table of Contents
- Part 1
- Intro
- Under The Hood
- The Move from WebRequest to HttpClient
- Strict header Parsing
- HttpWebResponse is now HttpResponseMessage
- Headers Values are Now String Arrays
- Content Headers are Separated
- WebHeaderCollection is now HttpResponseHeaders or HttpContentHeaders
- Error Handling (404 etc.)
- Part 1 Conclusion
- Part 2
- Part 2 Intro
- Table of Contents
- Missing Features, Deprecated Features, and Outstanding Issues
- .NET Core and PowerShell Core 6.0.0
- No FTP: or FILE: Support
- Basic Parsing Only
- No New-WebServiceProxy Cmdlet
- macOS Partial Support for SSL/TLS/Certificate Features
- No Certificate Authentication support for macOS
- Painful -CertificateThumbprint support on Linux
- No SSL 3.0 Support
- No ServicePointManager Support
- No Custom Certificate Validation Support
- Part 2 Conclusion
- Part 3 (This Post)
- Part 3 Intro
- Table of Contents
- New Web Cmdlet Features and Fixes in PowerShell Core 6.0.0
- New Parameters
- Updated Default User-Agents
- Authentication
- Improved Support for Multiple Response Headers With the Same Name
- Improved BasicHtmlWebResponseObject.Headers Performance
- Response Header Support for Invoke-RestMethod
- -SslProtocol Parameter
- Custom HTTP Method Support
- Proxy Bypass Support
- Support for Authorization on Redirect
- Support for Accessing Sites with Expired, Self-Signed, Malformed, or Untrusted Certificates
- Automated Pagination With Invoke-RestMethod
- Relation Links Parsed by Invoke-WebRequest
- Support for multipart/form-data Submissions
- Single Value Null JSON Literal Support for Invoke-RestMethod
- Conclusion
New Web Cmdlet Features and Fixes in PowerShell Core 6.0.0
It's Christmas time and Santa Claus has delivered a sleigh-load of presents to the good little boys, girls, and other genders of the PowerShell community. I am very excited about many new of the features in PowerShell Core 6.0.0. So many breaking changes were made for the better and so many new features have been added. If I could cover them all it would take a year and could fill a book.
This blog series is focused on just the Web Cmdlets. If you have read parts 1 and 2 you can see just how radically they have changed, and they are only a small (but important) part of PowerShell. In this part of the series I will pile on all the goodies we are delivering to the Web Cmdlets in PowerShell Core 6.0.0.
New Parameters
Most of the new features of Invoke-RestMethod and Invoke-WebRequest reveal themselves as new parameters available on the cmdlets. I will go into all of these in depth, but I want to show them all together so you can see just how impressive this list is:
New parameters in both Invoke-RestMethod and Invoke-WebRequest:
In addition to the new parameters available to both cmdlets, Invoke-RestMethod has theese additional new parameters:
That makes for 12 total new parameters!
You may be wondering if this is too much or if the web cmdlets are being overcomplicated. Those of us who have worked in the Linux/Unix side of the world have had curl and wget. Those 2 binaries are extremely flexible and can be used for wide variety web scenarios. By contrast, Invoke-RestMethod and Invoke-WebRequest are severely lacking, or at least they have been in my opinion. Their simplicity has made them rigid and in some cases useless.
But, do we need all these new features? I believe we really do. This is not just to play "catch up" with curl and wget. I'm not seeking 100% parity with them. However, the modern IT paradigm has been shifting to more and more REST based services used to manage everything from internal on-premise services to cloud based SaaS, PaaS and IaaS. If PowerShell is to survive as a modern scripting language, it needs immaculate support for web requests and it needs to be flexible enough to work with the various web API implementations. These new features will enable module authors to more easily abstract REST services as native PowerShell commands.
That has been my focus in contributing to the Web Cmdlets as well as why I got involved to begin with. I am not a developer and I am very new to C# (less than a year), but I use PowerShell heavily to work with many web based APIs both personally and professionally. For work, I am using Microsoft Graph and many Office 365 & Azure REST APIs to manage and audit our Office 365 and Azure environments. Outside of work, I work closely with with Reddit API. My desire to contribute to the PowerShell Core Web Cmdlets was driven by frustration and disappointment with the state of Invoke-RestMethod and Invoke-WebRequest in Windows PowerShell 5.1.
So, I do believe these new features are needed. I understand that this raises the complexity of the cmdlets and in some cases breaks the "do one thing" principle, but, rather than have 100's of separate tiny tools to do related tasks, having a small number that are flexible enough to deal with the majority of API requirements in the wild seems to be the better option.
Updated Default User-Agents
Now that PowerShell Core is cross-platform, it was time to make some changes to the User-Agent HTTP request header that Invoke-RestMethod and Invoke-WebRequest sends by default. In Windows PowerShell 5.1 this is what is being sent to servers from my Windows 10 system:
Some things to note: The Windows NT and WindowsPowerShell where hard coded values in the Microsoft.PowerShell.Commands.PSUserAgent class. Those obviously needed to change. Here is what is sent now:
Windows:
Linux (line breaks added to readability):
macOS (line breaks added to readability)::
You can see that the details are a bit more dynamic. The Platform identifier for Windows now includes the major and minor version. For all Linux and Unix distributions, Linux is the Platform Identifier. On macOS the Platform Identifier is Macintosh.The Platform Version Info now includes more detailed information for all platforms. Finally, it now says just PowerShell instead of WindowsPowerShell.
A change in User-Agent headers means that web servers may potentially serve different content to Invoke-RestMethod and Invoke-WebRequest. You may see different responses on Windows PowerShell 5.1 and PowerShell Core 6.0.0 on Windows as well as on Linux and macOS. Using the User-Agent to determine what kind of content to serve (for example, a desktop vs mobile version of a site) is not an exact science. While there is an RFC for User-Agent headers, it is very much irrelevant to what is used in the wild. This shouldn't be a problem for APIs, but may be for web scraping from normal web sites. My advice is to find a bowser that renders the page in a way you like, visit https://httpbin.org/headers, then use that User-Agent with the -UserAgent parameter.
If you want to see what User-Agent your Invoke-RestMethod and Invoke-WebRequest will send by default you can do one of 2 things.
1) Use https://httpbin.org/headers:
2) Use reflection
Also, don't forget that the Microsoft.PowerShell.Commands.PSUserAgent class provides several options as well:
Chrome
Firefox
Internet Explorer
Opera
Safari
For example:
For more information, see Pull Requests #4914, #4937, and #5256.
Authentication
In PowerShell Core 6.0.0 we have added an -Authentication parameter that will perform explicit authentication requests. What I mean by explicit is that the authentication is sent without being challenged. In Windows PowerShell 5.1 you could use the -Credential parameter for Basic Authentication, however, it would only send the Authorization: Basic header only if it had first received a WWW-Authenticate: Basic response header from the remote endpoint.
I already have an in-depth blog on this new feature at https://get-powershellblog.blogspot.com/2017/10/new-powershell-core-feature-basic-and.html so I suggest reading that. I will provide a brief review and some examples here of the new options, but for in-depth coverage please read the previous entry. Also, there is one addendum to the previous blog: The legacy -Credential parameter will now produce an error when you attempt to use it over HTTP to be consistent with the newer authentication options. The relevant Pull requests are #5052 & #5402.
Another quick note, if you find that -Authentication is too long, you can use the shorter -Auth.
No Authentication
This is the default behavior of the cmdlets when no -Authentication option is supplied, but it is also included as an explicit option should you need it.
Basic Authentication
The Basic option for the -Authentication parameter provides RFC-7617 Basic Authentication. This was available in Windows PowerShell 5.1, but only for challenge based authentication. Many modern APIs do not send challenge responses and expect you to provide your Basic Authentication up front. -Authentication Basic requires the -Credential parameter and does not support the -UseDefaultCredentials parameter.
For the below examples, the username is user and the password is passwd.
How to do this in Windows 5.1:
How to do it in PowerShell 6.0.0
Much easier!
Bearer/OAuth Authentication
Welcome to the modern Internet where an ever increasing majority of APIs and websites function on OAuth Authentication! Going into how OAuth works is out of scope for this series, but you can read up on RFC-6749. To accommodate this, we have added the Bearer and OAuth options to the -Authentication parameter. Both options do the exact same thing but are included as separate options for convenience. They both implement the RFC- 6750 Authorization: Bearer header.
The Bearer and OAuth options require the new -Token parameter. The -Token parameter requires a SecureString. OAuth tokens are supposed to be short-lived secrets. Like other secrets (such as passwords) they should not be in plain-text in your code. Also, some API's can have up to a 6 month expiration on OAuth tokens making them closer to traditional passwords instead of short-lived secrets.
For the following examples you can use any text you want for the OAuth token.
In Windows PowerShell 5.1
In PowerShell Core 6.0.0
The SecureString does complicate things somewhat, but, you can now send it as a parameter instead of as a header.
Please note that this does not manage your OAuth access token lifecycle. You will still need to request tokens yourself as well as refresh them when they expire.
Authentication Over HTTP vs HTTPS
Attempts to send secrets over HTTP instead of HTTPS will now result in an error. Here is an example using the previous Basic Authentication example but changing the URI to http instead of https.
This results in the following error:
There are 2 ways around this. The first and recommended solution is to switch to using HTTPS. There is really no excuse to not use HTTPS any more. If you have a legacy system that only supports HTTP or only supports older deprecated SSL protocols, you should place those behind reverse proxies that can support HTTPS and TLS (preferably 1.2).
But, we understand that some times that is just not an option or you are testing something. The second and unsafe option is to use the new -AllowUnencryptedAuthentication parameter.
Again, you should avoid doing this unless necessary, as you will be sending secrets over an unencrypted protocol.
Improved Support for Multiple Response Headers With the Same Name
As mentioned in part 1, HttpClient treats multiple response headers with the same name differently and response headers are now an array of strings. This means that we can now properly reconstitute them in BasicHtmlWebResponseObject.RawContent. Relevant Pull Request: #4494
The following example has a URI that causes HttpBin.org to return the following response headers:
Example:
Result on PS Core 6.0.0:
Result on Windows PowerShell 5.1:
You can see that PowerShell Core 6.0.0 properly shows the separate headers but Windows PowerShell 5.1 concatenates the separate headers into one line.
As a refresher, this improvement also applies to the Headers property:
On PowerShell Core 6.0.0
On Windows PowerShell 5.1
This should significantly improve parsing response headers from APIs. Please note that APIs and websites may still chose to send single delimiter separated header instead of multiple headers with the same name. In those cases, there will be a single array element with the delimiter separated string.
Improved BasicHtmlWebResponseObject.Headers Performance
The BasicHtmlWebResponseObject.Headers returned from Invoke-WebRequest transforms the headers object returned from the .NET API into a more PowerShell friendly Dictionary. This is the case for both Windows PowerShell 5.1 and PowerShell Core 6.0.0 even though the type of objects it parses is different. In Windows PowerShell 5.1 there was a bit of inefficient code for generating the Dictionary. Every time you accessed the property, a new Dictionary would be generated. This means a small excess of memory and CPU usage as well as some confusion.
Take the following code in Windows PowerShell 5.1:
The result is null and there are no errors. However this works as expected:
Result:
The reason the first block of code doesn't work is that the second time you access the Dictionary, a completely new Dictionary was created from the original response object.
In Pull Request #4853 this was fixed so that the Dictionary is only created once on the first access and then the same Dictionary is returned on all subsequent accesses.
The equivalent code in PowerShell Core 6.0.0:
Result:
This isn't going to result in a serious performance gain and whether or not users actually modify the headers dictionary is debatable. But, every little bit counts in my book!
Response Header Support for Invoke-RestMethod
Modern API's return response headers that range from comical, to useful, to absolutely critical. For an example of a critical response header, several APIs return the X-RateLimit-Remaining header which determines the number of API calls remaining for the current reset period before the client will be blocked from making API requests. This is to prevent the client from making too many API calls in too short a period of time. Other APIs use response headers to return information about where you are at in paged collection or when the current OAuth token will expire.
In Windows PowerShell 5.1, you have no way of retrieving the response headers on a successful call to Invoke-RestMethod. If you are working with an API in Windows PowerShell 5.1 and need to parse response headers, you have to fall back to Invoke-WebRequest, parse the headers, determine the content type, and then convert the content to a PowerShell object yourself. Basically, you are performing all the tasks that Invoke-RestMethod does for you just so you can inspect the Response Headers.
In Pull Request #4888 we added the -ResponseHeadersVariable parameter to address this problem. This features is covered in depth in my New PowerShell Core Feature: Invoke-RestMethod -ResponseHeadersVariable blog entry, so I will cover it only briefly here. This parameter works similar to the -SessionVariable parameter in that you supply a string for and a variable with that name will be created in the calling scope containing the same Response Headers Dictionary you would see on a similar Invoke-WebRequest call. If -ResponseHeadersVariable is too long for your tastes, there is also the -RHV alias you can use.
Here is an example:
Result:
Note that setting $Headers to null is not required and is done only to demonstrate that variable is empty before the call to Invoke-RestMethod. Also note that this parameter is only available on Invoke-RestMethod and is not available on Invoke-WebRequest. Invoke-WebRequest already returns the same headers as a property on the BasicHtmlWebResponseObject object.
-SslProtocol Parameter
Sometimes you find yourself in a position where you need to deliver code that meets certain security requirements. One increasingly common requirement is that network calls be made only over certain TLS versions such as TLS 1.1 and higher. To accommodate that need, Pull Request #5329 added the -SslProtocol parameter for both Invoke-RestMethod and Invoke-WebRequest.
The -SslProtocol parameter accepts a Microsoft.PowerShell.Commands.WebSslProtocol, which is a newly added Flag Enum. This is, admittedly, an odd choice for a PowerShell cmdlet parameter type, but the reason is that it can also accept a System.Security.Authentication.SslProtocols which is commonly used for this kind of setting. This allows for a single configuration setting to be used against multiple APIs. We created a separate Flags Enum, for this to ensure only the supported protocols can be supplied. However, you do not need to use this advanced feature. You can simply supply a string representation of the option or options.
The available options are as follows:
- Default - This is the default setting used when -SslProtocol is not supplied. It uses the underlying defaults for the system which tend to be Tls, Tls11, and Tls12 all being allowed.
- Tls - Use TLS 1.0
- Tls11 - Use TLS 1.1
- Tls12 - Use TLS 1.2
Here is an example of this in action:
You can specify multiple flags on some platforms. Currently this is not supported on macOS due to some underlying .NET Core limitations. We have also seen that some combinations are not supported on some Linux distributions. You can specify multiple options in two ways: using a string or using a binary-or:
Note that when using a string representation of multiple options, you place the comma and space inside the string. This is not an array of strings. As stated in Part 2 SSL 1.0, 2.0 and 3.0 are not supported.
Custom HTTP Method Support
Some APIs (IoT devices in particular) have a requirement for custom HTTP request methods. These are methods that fall outside the ones allowed by Microsoft.PowerShell.Commands.WebRequestMethod (GET, POST, HEAD, DELETE, MERGE, OPTION, PUT, and TRACE). Let's say you have an IoT water filtration device that has a reservoir tank that is managed through the /api/reservoir/ endpoint. In order to purge (dump) water from the reservoir, the IoT device expects the PURGE verb to be sent to the api/reservoir/ endpoint. In Windows PowerShell 5.1 this was not possible with the Web Cmdlets. However, in Pull Request #3142 the -CustomMethod parameter was added to Invoke-RestMethod and Invoke-WebRequest for PowerShell Core 6.0.0.
Example:
Result:
Proxy Bypass Support
There are times where you have a system-configured web proxy, but that proxy is not required or desirable for your Web Cmdlets calls. In Windows PowerShell 5.1 you were at the mercy of the system proxy settings. In Pull request #3447 the -NoProxy parameter was added for Invoke-RestMethod and Invoke-WebRequest.
Showing a demo of this is somewhat difficult as there are no publicly available proxies to test with that I'm aware of. But, I will walk you through briefly setting up a local proxy if you really want to see this in action. First, this assumes that you have installed node.js and npm which I will not cover here.
Prep and start the proxy server:
Now you will need to set your proxy in Internet Explorer to 127.0.0.1 port 9999 and enable it. In another PowerShell Session test a normal web request across the proxy:
On the node session you should see something like this:
and in the PowerShell Session you should see something like this:
Now that we know the proxy is working we can bypass it with the -NoProxy parameter:
You should not see any new entries in the proxy session window because the cmdlet has bypassed the proxy. Now that you are done testing you can close the console you were running the proxy in and revert your proxy setting in Internet Explorer.
If you are thinking this is a security hole, it really isn't. If the user has the ability to bypass an administrator configured proxy in the .NET API (either directly or through an application configured to do so), then this is no different. It's somewhat difficult to keep a user from bypassing a proxy on their system unless you lock it down so they cannot use anything but Internet Explorer. There are just too many ways around the settings and not every program even supports or respects the system proxy settings. Usually, you effectively block this at the network level making it impossible to make web requests against anything but the proxy.
Support for Authorization on Redirect
Some websites and APIs provide generalized login endpoints that will redirect the client to the correct authentication endpoint that will ultimately process the request. This isn't a common scenario in my experience and it is often discouraged. It's discouraged because the potential exists for abusing the client's trust in an endpoint to redirect the client to the endpoint of a bad actor which then collects the authorization details from the client. To prevent this abuse, Invoke-RestMethod and Invoke-WebRequest do not allow the Authorization request header to be sent when the endpoint responds with a redirect response.
However unsafe and discouraged this may be, there are websites and APIs out there that require this. To make it possible to use those resources, Pull Request #3885 added the -PreserveAuthorizationOnRedirect parameter to Invoke-RestMethod and Invoke-WebRequest.
To see this in action, let's first do a normal redirect with an Authorization request header. You can do this in both PowerShell Core 6.0.0 and Windows PowerShell 5.1.
The result is null because the redirected endpoint did not receive the Authorization header as it was stripped by Invoke-RestMethod. On PowerShell Core 6.0.0 do the following:
Result:
The -PreserveAuthorizationOnRedirect parameter bypasses the default behavior and the redirected endpoint receives the Authorization header. Again, you should use this with caution because it has the potential to be abused.
Support for Accessing Sites with Expired, Self-Signed, Malformed, or Untrusted Certificates
This feature has been deeply needed for a long time and is still a common question on PowerShell forums and slack channels. I have heard of users switching their code to PowerShell Core just to take advantage of this new feature. So, what is this awesome new feature? It's the ability to work with sites that have self-signed certificates.
This is a common scenario: you have a network appliance, IoT device, or test site setup and it is using a self-signed certificate to provide HTTPS access. You implicitly trust the endpoint because it is yours so you don't necessarily care that the certificate is not issued by a trusted CA. You are either testing something out or in a chicken-and-egg scenario where you need to initially trust the certificate before you can apply a certificate issued by a trusted CA. In a web browser you can chose to ignore the certificate warnings and add a temporary trust. But in Windows PowerShell 5.1 there was no easy way to do this.
This not to say that there is no way to do thisin Windows PowerShell 5.1. One option is to add the certificate to the Trusted Root Authorities cert store. That's not really a good idea, so there is a programmatic solution. Since Windows PowerShell 5.1 uses WebRequest and WebRequest honors settings in System.Net.ServicePointManager, you can do something like this:
That's ugly and not at all apparent what is going on. There are other alternatives such as creating a small .NET class with a delegate that returns true. There is incorrect documentation out there that says you can assign a ScriptBlock that returns true, but that doesn't seem to work in recent versions.
The solution in PowerShell Core 6.0.0 is the -SkipCertificateCheck parameter added in Pull request #2006 for both Invoke-RestMethod and Invoke-WebRequest.
First, let's try a site with a self-signed certificate and see what happens:
Result:
A Security error means the certificate used by the HTTPS endpoint is not trusted. In this case it is because the certificate is self-signed, but the same error occurs on sites with expired, malformed, and untrusted certificates. Now, let's see this new feature in action on PowerShell Core 6.0.0:
Result:
This is a feature you really shouldn't use unless you have to and you should definitely not be setting this as a default parameter value. This could allow a bad actor to impersonate an endpoint and steal credentials or confidential data. Trust is an essential part of encryption, not just an unnecessary annoyance.
Instead of relying on this feature, you should work on getting a trusted certificate on to or in front of the endpoint. If the endpoint doesn't allow you to set a trusted certificate, you can always put the endpoint in front of an HTTPS reverse proxy. Let the proxy handle the trust issues. If you are testing, that is all the more reason to use properly trusted certs. You should validate how things will work in production where hopefully you are using properly trusted certs. Even if this means setting up a development PKI, you should not be relying on self-signed certs.
But, those are "perfect world" scenarios. So, please just use this feature sparingly and with caution.
Automated Pagination With Invoke-RestMethod
This feature is one of my favorites from all of PowerShell Core 6.0.0. Some APIs take advantage of RFC-5988 Link response headers when returning pages of results. Take the GitHub issues API as an example. If you go to https://api.github.com/repos/powershell/powershell/issues you will see it returns 30 open issues from the PowerShell Gallery. If you go to https://github.com/PowerShell/PowerShell/issues you can see that there are over 1,000 open issues, definitely more than 30. The API is returning paged results.
If you look at the response headers from https://api.github.com/repos/powershell/powershell/issues you will see the following (I have added line breaks for readability):
The API provides a URI to the next page and last page of results. if you go to https://api.github.com/repositories/49609581/issues?page=2 and inspect the response headers, you will see this:
In addition to the next page and last page links, it provides the previous page and first page links. These link headers are what API clients use to parse and create next, back, first, and last links or to pull in all results from all pages.
In Windows PowerShell 5.1, in order to leverage an API's Link header pagination, you had to use Invoke-WebRequest. As mentioned before, Windows PowerShell 5.1's Invoke-RestMethod does not provide a method for inspecting response headers. That means the process in Windows PowerShell 5.1 went something like this:
- Use Invoke-WebRequest to grab a page of results
- Perform error detection
- Detect the Content-Type (JSON/XML/Text)
- Serialize a PowerShell object from the content (e.g. ConvertFrom-Json).
- Check for the existence of Link header
- Use Regex to parse the Link headers to find the URI for the next page
- Use the parsed URI and repeat the above
That is bunch of work and large part of it would normally be handled by Invoke-RestMethod.
In Pull Request #3828 the -FollowRelLink and -MaximumFollowRelLink parameters were added to Invoke-RestMethod. This allows for Invoke-RestMethod to automatically follow the next page links. -FollowRelLink enables the behavior and -MaximumFollowRelLink sets the maximum number of links to follow.
Here is an example of it in action:
Result:
This may seem a bit counter-intuitive at first. You might expect that instead of returning an array of arrays, Invoke-RestMethod would just return the members of each page as a single array. There is a good reason why this is not done. A cardinal rule for any API client is that is should maintain parity with the remote API. This means that if the remote endpoint returns an array, the API client should also return an array and not a string, dictionary or other object type. In this example, the GitHub API returns an array of issues for each page. Thus Invoke-RestMethod maintains parity and returns an array for each page.
If you want to normalize the data so it is not an array of arrays but instead a single array containing all issues you can do the following:
Result:
At this time I don't think we should bake a normalization capability into the Invoke-RestMethod. It would add some needless complication to the code and working with it in its current form is simple enough. To me, it is far more important to maintain parity.
There is an open issue on this that will be fixed in a future version. Issue #5667 points out a white space parsing issue in the current regex used to parse the link headers. One issue we did get fixed before 6.0.0 was support for multiple Link response headers being returned by an API (Pull request #5265). One possible feature addition for the future is to add the ability choose what link to follow. For example, supply the last page link and follow the previous page links to grab the last 3 pages.
Relation Links Parsed by Invoke-WebRequest
Invoke-WebRequest also has a new feature for the RFC-5988 Link response headers discussed in the previous section. The BasicHtmlWebResponseObject object returned by Invoke-WebRequest now includes a RelationLink Dictionary property. This is a Dictionary where the keys are the relation link type (next, first, last, prev, etc.) and the values are the URIs.
As an example, we can use the https://api.github.com/repositories/49609581/issues?page=2 page from the previous section:
Result:
Even though Invoke-WebRequest does not support automatic pagination, you can at least easily access the links. For example:
This was also added in Pull Request #3828.
Support for multipart/form-data Submissions
Many APIs and Websites require a multipart/form-data submission for uploading files or making mixed data and text form submissions. In Windows PowerShell 5.1 it was not possible to do this with Invoke-RestMethod and Invoke-WebRequest. multipart/form-data submissions required a workaround and direct calls to .NET APIs. In PowerShell Core 6.0.0 we have added initial support for multipart/form-data submissions. This is only the first phase of support. It is still a bit ugly and requires some .NET API calls but it lays the groundwork for an improved implementation planned for 6.1.0.
I cover this in depth in my Multipart/form-data Support for Invoke-WebRequest and Invoke-RestMethod in PowerShell Core article. Rather than go in-depth here I will only provide an example of submitting text and a file.
Example:
Result:
Basically you can now supply a MultipartFormDataContent object as the -Body parameter value. We completed the specification for the simplified version of this but it was completed too late to be accepted for 6.0.0. This is what you can tentatively look forward to in 6.1.0:
The current -Body parameter implementation in 6.0.0 will remain and you may still need to use it for more advanced multipart/form-data submissions, but the common uses will be further simplified. Notice the use of Get-Item. The -Form parameter will require a System.IO.FileInfo for files to distinguish file fields from text/string fields.
Single Value Null JSON Literal Support for Invoke-RestMethod
This is a breaking change fix in PowerShell Core 6.0.0. JSON supports single values as valid JSON. Usually, an API will return a dictionary or an array. Usually, they will return an array even if the array only contains a single value. But, a single JSON string, integer, or literal (true, false, or null) by itself alone is considered valid JSON. Windows PowerShell 5.1 works properly if the single value JSON is anything but null. When null is returned from an API, Invoke-RestMethod in Windows PowerShell 5.1 interprets it as the string "null" instead of $null. This behavior breaks parity with the API and requires logic to detect and correct in your own PowerShell code.
This has been fixed in Pull Request #5338. Run the following in both Windows PowerShell 5.1 and PowerShell Core 6.0.0:
Windows PowerShell 5.1 Result:
PowerShell Core 6.0.0 Result:
A quick note on empty responses: A 0 byte response or a white-space-only string is not valid JSON. JSON must be a string, integer, literal, dictionary, or array. As such, Invoke-RestMethod does not return $null for a 0 byte response or a white-space-only string. Instead, it returns either an empty string or a string that contains the white-space. This is by design and intended behavior. If you have an endpoint that returns 0 byte results or white-space-only strings that they document as being null, you will need to do your own null parsing. I recommend you contact the API owners and point them to the JSON RFCs. There is some confusion because an earlier RFC revision did not allow for single value literals and as a result it has persisted as a myth that they are not supported. This is why empty responses have been incorrectly used to represent null.
Conclusion
I hope you have enjoyed this in-depth series on the PowerShell Core 6.0.0 Web Cmdlets. As you can see, so many things have been changed. Hopefully I have covered everything and haven't missed anything. If you notice something is missing, please reach out to me and I will add it.
The general operation of Invoke-RestMethod and Invoke-WebRequest has mostly not changed on the surface, but these 2 cmdlets are radically different in PowerShell Core 6.0.0. We have more features in store for the future too. As mentioned, an Invoke-Download like cmdlet will be added as the de facto file download cmdlet. We also plan for the simplified multipart/form-data support. Custom Certificate Validation is also slated for future versions. Another idea floating around is to add a webstream focused cmdlet for working with things like log streams and other stream based web request. There are also several issues open (such as RFC compliant JSON character encoding detection) that will be fixed in future versions.
Thanks for reading and Happy Holidays!
