Part 2 Intro
In Part 1, I covered the primary changes in the actual code base of the PowerShell Core Web Cmdlets Invoke-RestMethod and Invoke-WebRequest and how those changes manifest themselves in the PowerShell user experience.
In Part 2, I will cover outstanding issues as well missing and/or deprecated features. Some of this will be an extension of Part 1 because some of the feature reduction is due mostly to the switch to HttpClient. Any plans or fixes I mention in this section are tentative and may change when and/or if they are implemented.
If you have not read Part 1, please do so before reading Part 2. This blog series goes in depth and requires a great many words to do so. To save space I will not repeat some information and will assume the reader as read Part 1.
Also, this will be my first blog post as a new Microsoft MVP. I wanted to take a quick moment to thank the people who nominated me and the PowerShell community for your support in getting me here. It is a great honor and I am overjoyed! Thank you!
Table of Contents
- Part 1
- Intro
- Under The Hood
- The Move from WebRequest to HttpClient
- Strict header Parsing
- HttpWebResponse is now HttpResponseMessage
- Headers Values are Now String Arrays
- Content Headers are Separated
- WebHeaderCollection is now HttpResponseHeaders or HttpContentHeaders
- Error Handling (404 etc.)
- Part 1 Conclusion
- Part 2 (This Post)
- Part 2 Intro
- Table of Contents
- Missing Features, Deprecated Features, and Outstanding Issues
- .NET Core and PowerShell Core 6.0.0
- No FTP: or FILE: Support
- Basic Parsing Only
- No New-WebServiceProxy Cmdlet
- macOS Partial Support for SSL/TLS/Certificate Features
- No Certificate Authentication support for macOS
- Painful -CertificateThumbprint support on Linux
- No SSL 3.0 Support
- No ServicePointManager Support
- No Custom Certificate Validation Support
- Part 2 Conclusion
- Part 3
- Part 3 Intro
- Table of Contents
- New Web Cmdlet Features and Fixes in PowerShell Core 6.0.0
- New Parameters
- Updated Default User-Agents
- Authentication
- Improved Support for Multiple Response Headers With the Same Name
- Improved BasicHtmlWebResponseObject.Headers Performance
- Response Header Support for Invoke-RestMethod
- -SslProtocol Parameter
- Custom HTTP Method Support
- Proxy Bypass Support
- Support for Authorization on Redirect
- Support for Accessing Sites with Expired, Self-Signed, Malformed, or Untrusted Certificates
- Automated Pagination With Invoke-RestMethod
- Relation Links Parsed by Invoke-WebRequest
- Support for multipart/form-data Submissions
- Single Value Null JSON Literal Support for Invoke-RestMethod
- Conclusion
Missing Features, Deprecated Features, and Outstanding Issues
I wish I could say that the PowerShell Core 6.0.0 web cmdlets have complete backwards compatibility. The reality is that they certainly do not. While I personally believe the Web Cmdlets in PowerShell Core are vastly superior to their Windows PowerShell ancestors, I would be dishonest if I didn't speak on missing features, deprecated features, and outstanding issues present.
.NET Core and PowerShell Core 6.0.0
PowerShell Core 6.0.0 is a different beast from Windows PowerShell 2, 3, 4, 5.0, and 5.1 for many reasons. The primary driver for most differences is the move to .NET Core. Before I go into to details about missing and deprecated features, I feel it is necessary to first explain some of the deep underlying architecture of PowerShell as well as explain the "go forward" strategy for the language in an attempt to alleviate concerns as well as to explain why having missing features is a good thing.
Many PowerShell users may not even need to know this to use PowerShell, but .NET is a huge part of the PowerShell language. You can only do things in PowerShell that are possible in .NET. PowerShell cmdlets are essentially easy ways to use .NET. You can even access .NET directly in several ways from within PowerShell. .NET truly puts the "Power" in "PowerShell".
PowerShell Core is built on .NET Core. Windows PowerShell 5.1 and older is built on .NET Framework. For decent primer on what that means, you can watch this video. The key takeaway is that .NET Core is a cross-platform open source subset of .NET Framework. For the PowerShell User, this doesn't change much in how you use PowerShell, but it does change what PowerShell can do. Since .NET Core is a subset of .NET Framework, not everything is available in PowerShell Core that was available in Windows PowerShell. Also, since .NET Core is cross-platform, many of the Windows specific features on PowerShell are also not immediately available.
What this means for you, the PowerShell user, is that 6.0.0 is a very significant change from Windows PowerShell 5.1. This often rubs PowerShell users the wrong way. Up until now, major version releases of PowerShell (e.g. 4.0 to 5.0) basically meant little-to-no loss of existing features with near 100% backwards and forwards compatibility. They amounted to bug fixes, performance enhancements, and new feature dumps. This is actually somewhat uncommon for programming and scripting languages, in my experience. Usually, a major version means significant refactoring is required on your code to work with the new version. In a sense, PowerShell users have been lucky and 3.0 to 5.1 is not a huge refactoring ordeal in most cases.
PowerShell Core 6.0.0 is a true major version release. Many breaking changes were introduced. But, breaking changes aren't always bad. Some undesirable features of the language have persisted because they could not be changed without breaking functionality between versions of PowerShell. PowerShell Core gave us, the PowerShell Community, a chance we never had before: to directly influence the feature set and future of PowerShell. Many community contributors, myself included, fixed many of our long standing gripes with the language. The move to open source and open development in PowerShell is not just for show. Community members and official PowerShell Team members have worked side by side to make 6.0.0 possible. There are open communication channels between the PowerShell community and Microsoft in a way that never existed before.
Windows PowerShell is not going away, at least not yet. You can run PowerShell Core 6.0.0 side-by-side with Windows PowerShell 5.1. This was something you could not do before as when you installed Windows PowerShell 5.0 it would completely replace Windows PowerShell 4.0. You can even run multiple version of PowerShell Core side-by-side (6.0.0, dev/daily builds, 6.1.0, etc.). Best of all, if you want a feature PowerShell wont provide, you can fork the project, add the feature yourself, build it, and run your own! For things that are not available in PowerShell Core, you can still make use of Windows PowerShell. Yes, eventually everything will move to PowerShell Core and all of the missing Windows PowerShell features will either be integrated, deprecated, or obsoleted. But that is a process that will happen over time.
And that is the future of the PowerShell language: cross-platform, community involved, open source, open development, modern application life cycle, and side-by-side versions.
While this part of the series may seem like a huge let down as I go into things that don't work or are missing in PowerShell Core's web cmdlets, please come back for Part 3 where I will go into all of the exciting new features and fixes. If you use the web cmdlets with any regularity you will forgive the missing features once you see all the goodies PowerShell Core 6.0.0 has in store for you.
No FTP: or FILE: Support
If you recall from Part 1, PowerShell Core 6.0.0 now uses HttpClient instead of WebRequest. The WebRequest API used in Windows PowerShell 5.1 was a generic API that supported multiple protocols. HttpClient is HTTP specific, which means it only support http:// and https:// URLs. What this means for the PowerShell user is that Invoke-RestMethod and Invoke-WebRequest also only support http:// and https:// URLs and that ftp:// and file:// URLs are no longer supported.
Example:
Results in 5.1:
Results in 6.0.0:
The Web Cmdlets will likely not support ftp:// and file:// URLs in the future either. For file:// it is probably not a huge loss as it is primarily a convenience for GUI browsers. It might have been useful in 5.1 when HTML parsing was built into Invoke-WebRequest, but that is not the case in 6.0.0 (more on that later).
For ftp://, this is a bit of a gap. While FTP has been slowly falling out of favor with HTTP taking over for almost al web related tasks, FTP is still around and is sometimes the only means of procuring a file from the web. I have opened issue #5491 to track this. There are 2 ways which may be available in the future.
First, I feel there is a missing cmdlet for generalized downloading. Invoke-RestMethod is specialized for serializing objects returned from web APIs. Invoke-WebRequest is specialized for dealing with HTML pages. Both provide download capabilities, but they are not optimal for doing so and trying to improve their download performance is painful. All of their value is in what they do with HTTP content after it has been attained, so they make for somewhat undesirable download cmdlets.
To address both the lack of FTP support and this downloading shortfall, I envision a new cmdlet specialized for downloading. It would be very basic. The only parameters would be a URI, credentials, output path, and possibly proxy related settings. It will be simple because the idea is to make it expandable to include more protocols in the future (e.g. TFTP, SFTP, SSH, etc.). So it wont have all the web cmdlet fluff. It will take at minimum the URI, similar to wget, and would save the file to the current location unless a path is provided.
Possible Syntax:
Theoretical Example:
Second, a neat idea is to add a SHiPS based solution for FTP. Being able to use commands like Get-Content, Get-ChildItem, Remove-Item, New-Item etc against FTP would be really cool, IMO.
Basic Parsing Only
In Windows PowerShell 5.1, Invoke-WebRequest could be used to parse HTML. This provided an object oriented way to navigate the HTML DOM. As a real basic example, here is how to get the path to the Google logo on google.com:
The way this was accomplished was using Internet Explorer's HTML rendering engine. Anyone who has installed a Windows 2012 R2 Server, launched a PowerShell console, and then ran Invoke-WebRequest without configuring IE beforehand has figured this much out. Obviously, Internet Explorer is not available on Linux and macOS. Parsing like this wouldn't have been available cross-platform.
This also means you can no longer use the Forms property do do something like this:
In PowerShell 5.1, when you didn't want or need the overhead associated with HTML Parsing, you supplied the -UseBasicParsing parameter. In PowerShell Core 6.0.0 all calls to Invoke-WebRequest are permanently set to -UseBasicParsing. The -UseBasicParsing parameter is now hidden and setting it to true or false will have no effect on the operation of the cmdlet. It is being kept for backwards compatibility with user scripts only.
This means that the objects returned by Invoke-WebRequest are always Microsoft.PowerShell.Commands.BasicHtmlWebResponseObject. PowerShell Core no longer has the HtmlWebResponseObject class.
Result in 5.1:
Result in 6.0.0
What is the future of HTML parsing in PowerShell? The future lies in a new cmdlet that has yet to be written: ConvertFrom-Html. This will be an all purpose HTML-to-Object conversion cmdlet. It will use AngleSharp to achieve this. This means that HTML parsing is not returning to the Web Cmdlets, but, that solution that will work with more than just the Web Cmdlets will be available. I'm hoping we can get this added by 6.1.0. I have not been personally involved in this effort, though. For now, if you need to parse HTML, you can do so on windows by calling out to Windows PowerShell. At this time, I do not have good solution for Linux and macOS users.
In Pull Request #5376, we removed the ParsedHtml and Forms properties from the formatting data. In the 6.0.0 betas these properties were still visible because the format data in 5.1, which applied to both BasicHtmlWebResponseObject and HtmlWebResponseObject, creating "dummy" properties in the default display that were always null. This resulted in quite a bit of confusion as to why parsing does not work. I expect the lack of parsing to continue to cause confusion, so please help spread the word this this feature is deprecated in PowerShell Core and that a replacement solution will come.
No New-WebServiceProxy Cmdlet
One cmdlet that is completely unavailable in PowerShell Core is New-WebServiceProxy.This was not technically a Web Cmdlet as it lives in a different PowerShell module and doesn't share any of the same code base as Invoke-RestMethod and Invoke-WebRequest.However, many consider it as part of the same family anyway, so I felt I should include it here.
I have never personally used New-WebServiceProxy for anything other than academic purposes. I do know it is used for SOAP services, which were very popular for web APIs for a time. REST has begun to dominate the Web API space, but SOAP is not completely gone yet and many legacy systems still only have SOAP.
Result:
Result in 6.0.0:
The future of this functionality is uncertain. In Windows PowerShell 5.1, New-WebServiceProxy used System.Web.Services.dll.The current line from Microsoft is that the System.Web namespace and the related libraries won't be ported to .NET Core from .NET Framework (with a very small number of exceptions). For clarity, that is a .NET Core decision, not a PowerShell decision. These libraries on Windows are some of the few .NET Framework libraries that can't be manually loaded into. .NET Core either. There is apparently thick dependencies on the OS and IIS infrastructure in these libraries, which makes them incompatible with .NET Core. With that said, that means we can't make use of them in PowerShell Core, even on Windows.
That doesn't mean the cmdlet is completely dead and functionality completely gone for all time. Currently, adding this functionality is "up for grabs" to anyone in the community who wants it. It is not a priority for the PowerShell Team and there haven't been any requests for it to be available in Core. If there was a sufficient outcry from customers that this was needed, it would show up on their radar. Until then, this will be a community effort.
If you are interested in possibly adding this, the relevant APIs/ Packages for this functionality in .NET Core are System.ServiceModel.Primitives, System.ServiceModel.Http, BasicHttpBinding, BasicHttpsBinding, NetHttpBinding, NetHttpsBinding, System.ServiceModel.NetTcp, and NetTcpBinding. My quick research on it revealed adding this functionality was possible, but that it might not work exactly as it did in 5.1. That might call for using a new Cmdlet name to prevent confusion.
macOS Partial Support for SSL/TLS/Certificate Features
I have to admit that my least favorite part of working with the Web Cmdlets so far has been macOS compatibility. First, I want to make it clear that while I do personally dislike macOS, I'm not bashing it as an OS. However, my general dislike for macOS is not what drives my dislike for it in dealing with the Web Cmdlets. The reason macOS has been unpleasant has more to do with the CoreFX (.NET Core) implementation of the HttpClient on macOS coupled with some oddities (for better or worse) on macOS.
CoreFX wraps libcurl on Linux and macOS when making calls via HttpClient. I personally love curl and libcurl and have always had nothing but positive experiences working with them in Linux. 99% of the time in CoreFX I get the same great experience, except when libcurl has been configured without OpenSSL.CoreFX seems to have settled on making some of the advanced features only work when OpenSSL is used as the crypto provider for libcurl. macOS does not use OpenSSL for its crypto provider by default, but you can change that if you want. If you use brew to install PowerShell on macOS, the formula includes installing the OpenSSL version of curl.
When OpenSSL is not used, any of the settings on HttpClientHandler concerning certificate validation or SSL protocols become either completely unusable or only work in certain circumstances. But, even when OpenSSL is used, I have seen strange behavior with these features.
To give you a concrete example, I recently added the -SslProtocol feature to the web cmdlets (more on that feature in Part 3). During the Pull Request it was discovered that macOS doesn't appear to support multiple protocols being supplied to HttpClientHandler.SslProtocols. This limitation isn't documented and appears to be a bug. In any case, we decided to mark the tests for that as pending on macOS.
The PowerShell project uses Travis CI for testing Linux and macOS builds. Recently, Travis CI updated their default image for macOS and deprecated the image that PowerShell was using. This update caused some of the -SslProtocol tests to fail. Rather than use the macOS provided libcurl, we had to force PowerShell to use the one provided by brew. If you are seeing weird issues with the web cmdlets on macOS try the following in bash or or another shell besides PowerShell:
If that clears up your issues, you will want to add the following to your .bash_profile to ensure PowerShell Core always uses the correct version of libcurl and OpenSSL.
Some of this same behavior can be observed on Linux systems that don't have an OpenSSL version of libcurl. I believe the current version CentOS 7 doesn't install the OpenSSL version of libcurl by default. This is almost always fixed there by adding the OpenSSL version of libcurl.
What this means to the PowerShell user is that they may see different or unexpected behavior on TLS/SSL/Certificate features of the Web Cmdlets on macOS (and sometimes Linux) based on a wide number of factors that are not immediately obvious. There isn't much that can be done about this other than to warn users. I have an update to the documentation that calls out that these features may not be supported on all platforms.
No Certificate Authentication Support for macOS
Because of some of the same issues called out in the previous section, client certificate authentication is not available in macOS. There is apparently no way to make this work in the raw .NET APIs. In CoreFX Pull Request #17011 .NET Core removed the reliance on OpenSSL because macOS is moving away from OpenSSL. Also, CoreFX moved their certificate implementation on macOS to use Apple's Security.Framework. While this is great for the vast majority of CoreFX by making it stable and less dependent on third party libraries, it does mean some things fall out of the realm of possibility. Apparently, due to how libcurl handles certificates and an incompatibility with Security.Framework there is no way for CoreFX to present a certificate in a useable fashion for libcurl to consume.
If you were to do the following
You will get the following error on macOS:
The version of libcurl and the SSL backend may be different depending on what version of macOS you are on, or if you are using the libcurl provided by brew or MacPorts.
Painful -CertificateThumbprint Support on Linux
The -CertificateThumbprint feature allows you to use the thumbprint of a certificate in your certificate store to initiate client certificate authentication. Currently, the certificate store is not implemented by default on Linux. Doing this
Will result in this error:
If you happened to have used a .NET Core application that has initialized this for you then you won't see that error. You can initialize your own store with the following:
You can obtain ClientCert.pfx from here. This self-signed certificate is used for client certificate authentication tests in the PowerShell project.
You will need to use the .NET APIs to add and remove certificates to and form the store. How to use the .NET APIs to manage the certificate store is out of scope for this blog series. The store does create a folder:
However, directly manipulating that folder is not supported by .NET Core. I would recommend against trying to manually modify this folder and its contents. On Windows you can use the Cert: PSDrive or the Certificates MMC snap-in to manage your client certificates. However, per Issue #1865, the Cert: PSDrive is currently not implemented on non-Windows platforms.
While .NET Core on macOS uses the macOS Keychain as the certificate store, client certificate authentication is still not supported on macOS as mentioned in the previous section.
No SSL 3.0 Support
I'm not sure whether to consider this an improvement or a loss, but Invoke-RestMethod and Invoke-WebRequest do not support SSL 3.0 in PowerShell Core. Hopefully, you don't have anything still running on SSL 3.0. I should add that SSL 1.0 and 2.0 are also not supported, but I don't believe they were supported in 5.1. I don't have any easy way to confirm that.
This is because HttpClient does not support SSL 3.0 in .NET Core. It wasn't a conscious decision made by the PowerShell team, but a side effect of the move to HttpClient.
TLS 1.0, 1.1, and 1.2 are supported. TLS 1.3/2.0 is not supported yet. However, I don't believe it is in wide use yet and at last check some browsers don't support it either.
What this means is that if you have legacy systems where the crypto protocols cannot be updated, you won't be able to access them directly with the web cmdlets. My recommendation for legacy systems with obsolete protocols or none at all (HTTP-only) is to place them behind reverse proxies that can manage the crypto for you. You can always upgrade and replace reverse proxies to suit your needs.They don't require much in the way of resources (usually) an there are many free options with tons of documentation and community support. There really is no good excuse for legacy crypto protocols, in my professional opinion.
No ServicePointManager Support
Another side effect of moving to HttpClient is that the options under System.Net.ServicePointManager have no effect on Invoke-RestMethod and Invoke-WebRequest in PowerShell Core. In Windows PowerShell 5.1 this was used to control certificate trusts and security settings on the Web Cmdlets. For example, you could do the following to trust all HTTPS Certificates, including self-signed, expired, and those issued by untrusted Certificate Authorities:
I lifted that code lovingly from Patrick Meinecke.
Some of the other settings that are used by PowerShell scripts in the wild include the following
HttpClient does not make use of any of these settings. Instead, these features are implemented on System.Net.Http.HttpClientHandler. However, HttpClientHandler is defined and controlled only within the Web Cmdlets code at runtime and cannot be accessed or modified by the PowerShell user. In time, these features may be made available as parameters on the web cmdlets. I also have a plan to possibly add some process level settings to the cmdlets where these settings may also be exposed.
No Custom Certificate Validation Support
Without access to [System.Net.ServicePointManager]::ServerCertificateValidationCallback , There is no way to do any custom certificate validation. Currently, the only option is to supply the -SkipCertificateCheck parameter (more on that feature in Part 3). That will trust all certificates, but wont allow you to trust a specific CA only, or deny a cert with a specific thumbprint, or ignore cert errors only or a specific site. In Windows PowerShell 5.1, a PowerShell user could create custom C# delegates via Add-Type to set as the ServerCertificateValidationCallback and implement their own restrictive or permissive certificate polices.
The current plan is to add a -CertidficateValidationScript parameter to the cmdlets which will accept a ScriptBlock that will be run to perform the certificate validation. This will have some caveats. First, since the underlying .NET calls this code in a separate thread, the code in the ScriptBlock will need to be self contained. That means any variables, functions, aliases, modules, etc. defined in the current scope will not work in this ScriptBlock. It also will not support $using: variables.
You can read more about this in Issue #4899. I had a Pull Request open for this, but due to some code cleanup and PowerShell Committee review decisions, the code needed to be almost completely rewritten. I will try to get this feature added to 6.1.0.
Part 2 Conclusion
There are a bunch of open issues on the web cmdlets, but many of those are issues that are also present in Windows PowerShell 5.1. I wanted to keep this limited to issues that are new in PowerShell Core 6.0.0. Some of the other issues opened are debatable or not completely reproducible.
Part 3 will be out in a few weeks. Part 3 will cover all of the fun goodies added to the PowerShell Core 6.0.0 Web Cmdlets with working examples of how to use them. I may have another PowerSMells article between now and then as well. I will be working on those between all of the cleanup work I have planned for the Web Cmdlets. I want to use my free time this holiday season to address the backlog of issues and cleanup activities for the Web Cmdlets and their tests. The goal is to make contributions to the cmdlets easier and the tests more stable & less reliant on 3rd party web sites.
Stay Tuned!
