Intro
Recently the PowerShell Team published their PowerShell Core 6.1 Roadmap. Several questions have come up about 6.1 plans for Invoke-WebRequest and Invoke-RestMethod. You may have noticed there is no mention of the Web Cmdlets in their roadmap. That is because the majority of the feature work for the cmdlets is being done by yours truly. As I am a community member and not a Microsoft employee, it would make sense that my work is not included in their blog. Anyway, I felt it was a good idea to share what I have in store for the next 6 months.
I should warn that this is more accurately called "Mark Kraus's PowerShell Core 6.1 Web Cmdlets Wish List" rather than a roadmap. I only have 2 features that are pretty much approved. The rest will depend on whether or not the changes actually get accepted. Also, I'm not part of any team working on this. It's just me and the issue queue back log of feature requests and bugs. If your company wants to hire me to work full time on the Web Cmdlets, then hit me up. Otherwise, this is all a side project for me and life happens. So please don't hold me to any deadlines! :)
Simplified multipart/form-data Support
Status: Complete
This is continuation of the work done in 6.0.0. This one will definitely be in 6.1 as it was all ready merged into master from #5972. I will have a blog post on this after I finish documenting it. If you look at the header image for this blog post you can see a quick demo of it. Basically, we've added a -Form parameter to Invoke-WebRequest and Invoke-RestMethod that accepts strings and files and will submit the strings and files as multipart/form-data to endpoints. This includes support for multiple files or strings for the same form field.
https://github.com/PowerShell/PowerShell/pull/5972
Partial File Download Resume
Status: Completed (Would you like to know more?)
This one is new to the list. I didn't originally plan to investigate this until after 6.1, but, there have been multiple requests for it. Basically, if you have a download that is interrupted, Invoke-WebRequest and Invoke-RestMethod will be able to resume where the download left off. Let's say if you downloaded 8 GB of a 10 GB file when your internet died, you will be able to download the file again but instead of redownloading the whole 10 GB, it will pickup and download the remaining 2 GB. I haven't even done a POC (proof of concept) on this yet, so don't get your hopes up just yet. However, this is next in my feature queue to look into adding.
https://github.com/PowerShell/PowerShell/issues/5964
Custom Certificate Validation
Status: Moved to 6.2.0
Update: I am awaiting a few features in future versions of .NET Core which might address some of the pinpoints in macOS. I also waiting on the addition of experimental feature gating. I will consider this for 6.2.0.
Shortly before 6.0.0 went GA, the PowerShell Committee made some final decisions on the implementation of custom certificate validation. I was doing a bunch of refactor work and the existing PR could not be rebased. I will pick this one up after the file resume feature. I may hold off on this until .NET Core adds a managed implementation of HttpClient that isn't dependent on libcurl. As it stands now, any custom certificate validation will not work on macOS due to .NET Core limitations. I'm a bit torn on adding features that are not fully x-plat. Anyway, this feature will allow users to supply their own custom certificate validation ScriptBlock that will either pass or fail SSL/TLS server certificates based on whatever criteria the user defines. This would, for example, allow you to accept self-signed certificates but only with specific thumbprints.
https://github.com/PowerShell/PowerShell/issues/4899
Extend -SkipHeaderValidation Support to -ContentType
Status: Completed
Currently, -SkipHeaderValidation only suppresses RFC compliance checking on -Headers and -UserAgent. We need to extend that to -ContentType. For one, if you needed to send some non-RFC-compliant Content-Type request header, there is currently no way to do so. For another, there is a bug in .NET Core where even RFC-compliant Content-Types are being rejected.
https://github.com/PowerShell/PowerShell/pull/6018
Fix Response Encoding Detection on application/json per RFC
Status: Completed
Currently, the Web Cmdlets default to iso-8859-1 encoding when the server does not supply a content encoding. This is problematic for application/json responses as, per RFC 4627, a charset is not required and must be Unicode. There is a specifically defined method we need to implement for detecting UTF-8, UTF-16LE, UTF-16BE, UTF-32LE, or UTF32-BE with UTF-8 being the default if detection fails. I hope to be able to fix this, but, it qualifies as a breaking change even though it fixes bad behavior. I hope to have it in 6.1, but the breaking change aspect might be problematic.
https://github.com/PowerShell/PowerShell/issues/5530
Fix Redirects to Relative URI's when Authorization Header Present
Status: Completed in #6325
There is currently a minor esoteric bug when an Authorization header is present, the server redirects to a relative URI, and -PreserveAuthorizationOnRedirect is not used the redirect will fail. This will need a good relative-to-absolute URI mashing solution.
https://github.com/PowerShell/PowerShell/issues/5967
Web Cmdlet Error Message Makeover
Status: Ongoing
Seriously, the error messages for the Web Cmdlets are bad and we should feel bad. I certainly do. I plan to give all of these some tender love and care for 6.1.
https://github.com/PowerShell/PowerShell/issues/5140
Web Cmdlet Code Cleanup and Refactor Project
Status: Ongoing
This is ongoing work. This isn't sexy feature work so I don't bring it up often as it bores people to tears. Take a look at the Web Cmdlet code base. it is frightening and intimidating. It is not easy to work with. The code is messy, all over the place, and has the flavors of some 40 different programmers mashed into it. I took it upon myself to start fixing all this nonsense. Some major work to remove the Full CLR code went in before 6.0.0 released and I have made some small headway since. This will be ongoing and those who want to pitch in and help are very welcome. The goal is to make this code easy for the community to jump in and assist with new features and bug fixes.
https://github.com/PowerShell/PowerShell/issues/5610
Web Cmdlet Tests Cleanup and Refactor
Status: Ongoing
I have to admit that the tests are much better now than they were when I started contributing. I recently completed moving all tests to my WebListener project. This makes all of the tests no longer reliant on any external web sites. It also makes all tests fully functional across all platforms in-box. They are much more stable now than they have been in the past. There is, however, still a bunch of cleanup to be done. Every test needs to be inspected to see if it is still needed, if the test is still valid, and to identify redundant tests. In addition there is a bunch of cleanup to do in terms of helper functions and code consistency. This work is pretty easy though so if you are looking to help out this is a good place to start.
https://github.com/PowerShell/PowerShell/issues/5669
Web Cmdlet Automatic Retry Feature
Status: In Progress
We need to add the ability to retry when a request fails. For example, if an API returns that it is busy or even a temporary 404 while waiting for deployment content to distribute. This work is currently not being done by me, but by one of the PowerShell Team members. I'm not sure if this will make it to 6.1. There are some, erm, logistical issues with implementing this as well as testing it.
https://github.com/PowerShell/PowerShell/pull/5760
Want to Help?
So, if you are interested in helping out on the Web Cmdlets, please reach out to me. My contact information is in the side links. I would be happy to coordinate with other community members on making the Web Cmdlets better. If you have questions about the Invoke-WebRequest and Invoke-RestMethod code base, please ask! I feel close to being expert on it at this point so I probably have an answer.
Conclusion
So that's it for now. I'm not sure that is really enough to fill the whole 6 months so I will probably have more. later. I have some RFCs to draft for some session spanning features I want to implement as well as work on a download-centric cmdlet that doesn't share the Web Cmdlet code base (for inclusion in micro-footprint installs of PS Core planed for the future). Those definitely won't make it to 6.1 unless a miracle happens, but work for them has already begun. I also want to look at making some of the Web Cmdlet API's public, but who knows how well that will work out. *shrugs. Anyway, plenty planned, lots to do but that's my PowerShell Core 6.1 Web Cmdlet Roadmap.
