Call For Presenters: The PowerShell Conference Book Volume 2

Calling All PowerShell and DevOps Authors, Presenters, Bloggers, and Enthusiasts!

The Call for Presenters (authors) is open for “The PowerShell Conference Book: Volume 2” until Friday May 31 at 11:00 PM EST. We are looking for one chapter per author on the topics of PowerShell and DevOps. All proceeds for the book will go to the PowerShell + DevOps Global Summit OnRamp Scholarships.
Those interested should submit a chapter abstract to the PSConfBook 2 CFP form at http://bit.ly/PSConfBook2CFP before the deadline. Authors may submit multiple abstracts, but only one abstract per author will be selected for inclusion in the book.

On to New Endeavors: My New Role at LinkedIn

On May 18th I posted a tweet that made public my intent to find a new job. I’m happy to announce that search is over!

I have accepted an offer at LinkedIn with a tentative start date in the coming weeks.

I have waited to announce this publicly until I was sure there were no roadblocks. It has been a kind of open secret as I haven’t shied away from sharing it with anyone who asked directly.

This blog is to share that news and a little bit about the circumstances around the change.

Why Invoke-RestMethod and ConvertFrom-Json Have Funky Pipelines

Intro

At least once a week I am either directly asked to @-mentioned into a question about why Invoke-RestMethod and/or ConvertFrom-Json don’t play nice with the pipeline. I decided to put this into a blog post so I can just point people to it in the future.

Note that this blog entry was written while PowerShell 6.0.2 was current and 6.1.0 was on the way. This behavior may change in future versions.

New Feature for 6.1: Resume Partial Downloads with Invoke-WebRequest and Invoke-RestMethod

Intro

I’m excited to announce a new feature for Invoke-WebRequest and Invoke-RestMethod that will ship with PowerShell Core 6.1.0: Resume Downloads!

This is a feature that has been requested many times throughout the years and I’m please to say that it will be included in the next release of PowerShell Core. You should be able to preview the feature in 6.1.0-preview.2.

You can se the code changes in PR #6447.

Peanut Butter and Chocolate: Azure Functions CI/CD Pipeline with AWS CodeCommit (Part 6 of 6)

Part 6

In Part 5 we configured the AWS CodeCommit to trigger the AWS Lambda when a commit is made to the master branch of the repository. Effectively, our CI/CD pipeline is in place. To use it properly, we first need to add a cc2af.yml configuration file. After the configuration file is there, we can push out first Azure Function to our AWS CodeCommit repository and our AWS Lambda will be triggered and start a manual deployment on the Azure Functions Web App.

We will finish out the series in this post with demonstrating an automatic deployment from AWS CodeCommit to Azure Functions and triggering our Azure Function all from PowerShell.

Series Table of Contents

• Part 1 of 6: Project Introduction and Overview
  
• Part 2 of 6: Project Settings, Authentication, Azure Function App Deployment, and AWS Code Commit Deployment
  
• Part 3 of 6: Creating AWS IAM User for Git, Generating HTTPS Git Credentials, Configuring Azure Web App External Git Deployment, and Initial Manual Deployment
• Part 4 of 6: Create AWS IAM Role, Deploy the C# .NET Core 2.0 AWS Lambda, and Create an AWS KMS Key
• Part 5 of 6: Add AWS CodeCommit Trigger and Encrypt Secrets with AWS KMS
• Part 6 of 6 (This post): Generate Configuration File, Automatic deployment from AWS CodeCommit to Azure Functions, Trigger the Azure Function from PowerShell, and Series Conclusion

Peanut Butter and Chocolate: Azure Functions CI/CD Pipeline with AWS CodeCommit (Part 5 of 6)

Part 5

Sorry for the delay between part 4 and 5! I was at the Microsoft MVP Summit this past week and didn’t have time to devote towards updating. This series is nearing completion with just a few more parts to go.

In Part 4 we published the AWS Lambda and created the AWS KMS Key that will be used for encrypting and decrypting secrets. In Part 5 we will configured the AWS CodeCommit repository trigger to invoke the AWS Lambda and encrypt our secrets to store in in the cc2af.yml file.

Series Table of Contents

• Part 1 of 6: Project Introduction and Overview
  
• Part 2 of 6: Project Settings, Authentication, Azure Function App Deployment, and AWS Code Commit Deployment
  
• Part 3 of 6: Creating AWS IAM User for Git, Generating HTTPS Git Credentials, Configuring Azure Web App External Git Deployment, and Initial Manual Deployment
• Part 4 of 6: Create AWS IAM Role, Deploy the C# .NET Core 2.0 AWS Lambda, and Create an AWS KMS Key
• Part 5 of 6 (This Post): Add AWS CodeCommit Trigger and Encrypt Secrets with AWS KMS
• Part 6 of 6: Generate Configuration File, Automatic deployment from AWS CodeCommit to Azure Functions, Trigger the Azure Function from PowerShell, and Series Conclusion

Peanut Butter and Chocolate: Azure Functions CI/CD Pipeline with AWS CodeCommit (Part 4 of 6)

Part 4

In Part 3 we successfully made the first glue between Azure Functions and AWS CodeCommit by making it possible to manually trigger the Azure Functions Web App to pull from the AWS CodeCommit repository. Obviously, a manual pull is not ideal. It is certainly not a Continuous Delivery.

In Part 4 we lay the groundwork for the 2nd piece of glue between Azure Functions and AWS CodeCommit. In order to automatically trigger a pull AWS CodeCommit from Azure Functions, we need an AWS Lambda. AWS Lambda and Azure Functions are somewhat analogous. They serve almost identical purposes in their respective clouds. We also need to create a KMS key that will be used for encrypting and decrypting secrets.

Series Table of Contents

• Part 1 of 6: Project Introduction and Overview
  
• Part 2 of 6: Project Settings, Authentication, Azure Function App Deployment, and AWS Code Commit Deployment
  
• Part 3 of 6: Creating AWS IAM User for Git, Generating HTTPS Git Credentials, Configuring Azure Web App External Git Deployment, and Initial Manual Deployment
• Part 4 of 6 (This Post): Create AWS IAM Role, Deploy the C# .NET Core 2.0 AWS Lambda, and Create an AWS KMS Key
• Part 5 of 6: Add AWS CodeCommit Trigger and Encrypt Secrets with AWS KMS
• Part 6 of 6: Generate Configuration File, Automatic deployment from AWS CodeCommit to Azure Functions, Trigger the Azure Function from PowerShell, and Series Conclusion

Peanut Butter and Chocolate: Azure Functions CI/CD Pipeline with AWS CodeCommit (Part 3 of 6)

Part 3

In Part 2 we created the Azure Functions Web App and the AWS CodeCommit repository. In Part 3 we will make the initial deployment from AWS CodeCommit to Azure Functions. To do that we need to create an AWS IAM User Account, grant it access to the CodeCommit repository, generate HTTPS Git Credentials for the user, and configure the Azure Functions Web App external git deployment. By the end of this post, we will be able to manually deploy from AWS CodeCommit to Azure Functions on demand. This is a critical step to make automating the process possible.

This part will be short and sweet. I want to keep the relevant pieces together regardless of their length.

Series Table of Contents

• Part 1 of 6: Project Introduction and Overview
  
• Part 2 of 6: Project Settings, Authentication, Azure Function App Deployment, and AWS Code Commit Deployment
  
• Part 3 of 6 (This Post): Creating AWS IAM User for Git, Generating HTTPS Git Credentials, Configuring Azure Web App External Git Deployment, and Initial Manual Deployment
• Part 4 of 6: Create AWS IAM Role, Deploy the C# .NET Core 2.0 AWS Lambda, and Create an AWS KMS Key
• Part 5 of 6: Add AWS CodeCommit Trigger and Encrypt Secrets with AWS KMS
• Part 6 of 6: Generate Configuration File, Automatic deployment from AWS CodeCommit to Azure Functions, Trigger the Azure Function from PowerShell, and Series Conclusion

Peanut Butter and Chocolate: Azure Functions CI/CD Pipeline with AWS CodeCommit (Part 1 of 6)

Source: 1981 Reese's Peanut Butter Cup Advertisement

Intro

This blog series will cover a Proof of Concept (POC) Project for creating a PowerShell-based Azure Functions CI/CD pipeline where the code is stored in AWS CodeCommit git-based version control system. The pipeline will be created and deployed using Windows PowerShell 5.1. Every step of the pipeline deployment process will be verified with Pester tests. The result of the project will be the ability to push changes to an AWS CodeCommit repository and those changes will be automatically deployed to Azure Functions.

This blog series is targeted at intermediate level PowerShell users and basic PowerShell concepts will not be described in detail. Also, this series will require some basic understanding of both Azure and AWS clouds and their PowerShell based management. Git and C# .NET Core are also leveraged in this project but they will not be covered in depth as this is a PowerShell-centric blog. Readers need only be familiar with basic concepts of git and C#.

Series Table of Contents

• Part 1 of 6 (This Post): Project Introduction and Overview
  
• Part 2 of 6: Project Settings, Authentication, Azure Function App Deployment, and AWS Code Commit Deployment
  
• Part 3 of 6: Creating AWS IAM User for Git, Generating HTTPS Git Credentials, Configuring Azure Web App External Git Deployment, and Initial Manual Deployment
• Part 4 of 6: Create AWS IAM Role, Deploy the C# .NET Core 2.0 AWS Lambda, and Create an AWS KMS Key
• Part 5 of 6: Add AWS CodeCommit Trigger and Encrypt Secrets with AWS KMS
• Part 6 of 6: Generate Configuration File, Automatic deployment from AWS CodeCommit to Azure Functions, Trigger the Azure Function from PowerShell, and Series Conclusion

PowerShell Core 6.1 Web Cmdlets Roadmap

Intro

Recently the PowerShell Team published their PowerShell Core 6.1 Roadmap. Several questions have come up about 6.1 plans for Invoke-WebRequest and Invoke-RestMethod. You may have noticed there is no mention of the Web Cmdlets in their roadmap. That is because the majority of the feature work for the cmdlets is being done by yours truly. As I am a community member and not a Microsoft employee, it would make sense that my work is not included in their blog. Anyway, I felt it was a good idea to share what I have in store for the next 6 months.

I should warn that this is more accurately called "Mark Kraus's PowerShell Core 6.1 Web Cmdlets Wish List" rather than a roadmap. I only have 2 features that are pretty much approved. The rest will depend on whether or not the changes actually get accepted. Also, I'm not part of any team working on this. It's just me and the issue queue back log of feature requests and bugs. If your company wants to hire me to work full time on the Web Cmdlets, then hit me up. Otherwise, this is all a side project for me and life happens. So please don't hold me to any deadlines! :)

PowerShell Core Web Cmdlets in Depth (Part 3)

Part 3 Intro

In Part 1, I covered the primary changes in the actual code base of the PowerShell Core Web Cmdlets Invoke-RestMethod and Invoke-WebRequest and how those changes manifest themselves in the PowerShell user experience.

In Part 2, I covered outstanding issues as well missing and/or deprecated features.

In Part 3, I will cover new features available in PowerShell Core 6.0.0 Invoke-RestMethod & Invoke-WebRequest. I will also cover future plans for the cmdlets.

If you have not read Part 1 and Part 2, please do so before reading Part 3. This blog series goes in depth and requires a great many words to do so. To save space I will not repeat some information and will assume the reader as read Part 1 and Part 2.

A quick bit of news: PowerShell Core v6.0.0-rc.2 was released. Unless any blocking issues are discovered, this will be the final RC release and the next release will be GA in January 2018.

PowerShell Core Web Cmdlets in Depth (Part 2)

Part 2 Intro

In Part 1, I covered the primary changes in the actual code base of the PowerShell Core Web Cmdlets Invoke-RestMethod and Invoke-WebRequest and how those changes manifest themselves in the PowerShell user experience.

In Part 2, I will cover outstanding issues as well missing and/or deprecated features. Some of this will be an extension of Part 1 because some of the feature reduction is due mostly to the switch to HttpClient. Any plans or fixes I mention in this section are tentative and may change when and/or if they are implemented.

If you have not read Part 1, please do so before reading Part 2. This blog series goes in depth and requires a great many words to do so. To save space I will not repeat some information and will assume the reader as read Part 1.

Also, this will be my first blog post as a new Microsoft MVP. I wanted to take a quick moment to thank the people who nominated me and the PowerShell community for your support in getting me here. It is a great honor and I am overjoyed! Thank you!

PowerShell Core Web Cmdlets in Depth (Part 1)

Intro

I recently spoke at the North Texas PC Users Group's PowerShell Special Interest Group on the topic of the Web Cmdlets in PowerShell Core. I spoke for full hour because there is just so much new and different about Invoke-RestMethod and Invoke-WebRequest between Windows PowerShell 5.1 and PowerShell Core 6.0.0. In fact, because I was limited to an hour, I couldn't go as in depth or cover as many things as I would have liked. This blog series will cover what I covered in that presentation and more. At this time I plan to have 3 parts with a possible 4th as an addendum should anything change between now and GA.

New PowerShell Core Feature: Invoke-RestMethod -ResponseHeadersVariable

Pictured: The new -ResponseHeadersVariable parameter in action

Intro

I wanted to share with you another new feature added to the PowerShell Core Invoke-RestMethod Cmdlet: The -ResponseHeadersVariable parameter.

This feature will be available starting with PowerShell Core v6.0.0-beta.8 and is available now in the nightly builds or if you build it yourself from master.

Multipart/form-data Support for Invoke-WebRequest and Invoke-RestMethod in PowerShell Core

Pictured: A packet capture of a current build of PowerShell Core submitting a multipart/form-data POST request from Invoke-WebRequest.

Intro

Over the past few months I have been donating a generous portion of my spare time to help improve the Web Cmdlets (Invoke-WebRequest and Invoke-RestMethod) in PowerShell Core. This is partly because I want and need certain functionality for both personal and work related projects. It is also because I have had some minor gripes about these Cmdlets for some time.

One common ask I have seen repeated in just about every PowerShell forum is multipart/form-data support. It seems like a reasonable thing to ask when there are many endpoints that will only work with a multipart/form-data submission. There is an open issue (#2112) on the PowerShell GitHub echoing the same request. It was brought to my attention and I decided to give it a serious look.

The result is that PowerShell Core now has partial multipart/form-data support in both Web Cmdlets. This change didn't make the cut for 6.0.0-beta.7 but it will be available starting in 6.0.0-beta.8 and is available now if you build it manually or grab the latest nightly build.

This blog will cover some of the challenges involved in supporting multipart/form-data, how to make use of this new feature, and about future plans for additional support.

Because typing multipart/form-data is annoying, I will be shortening it to just multipart. Please don't let this be mistaken for other multipart submission methods.

Also, I will be referring collectively to Invoke-WebRequest and Invoke-RestMethod as Web Cmdlets. In this case, there is no need to call out each command as they offer the same base functionality for multipart support.

Porting PSRAW to PowerShell Core: Lessons Learned

Intro

I took a significant break from commits to my PSRAW project. I have spent that time learning more about Open Source projects and Object Oriented Programming. Before I could move the project forward I had some architectural decisions to make and I don't have quite enough knowledge to make those decisions yet, but I'm getting there.

On July 14th, this blog from Microsoft dropped a bit of a bombshell on the PowerShell community by making it clear the path forward is in PowerShell Core.Windows PowerShell will still be developed/maintained, but the primary focus will be PowerShell Core. There was also a call to test out PowerShell Gallery modules. I put off the leap to Core for awhile, but it seemed that now was the time. My module is still young and flexible and I suspected that most of it would work on Core.

Shortly after that blog post, I  unleashed the Kraken and created a new branch in my local PSRAW repo named CoreRefactor, installed PowerShellCore 6.0.0-beta.4, switched VS Code to use PowerShell Core for the integrated terminal, and fired up my pester tests. Thus flowed a glorious sea of red failures and errors. This kicked off 2 weeks of refactoring. I wanted to share what I have learned from the experience.

At the time of this post PowerShellCore 6.0.0-beta.5 has just been released. I just completed my test against it. So everything in this post is at least relevant to that release.

How Safe Are Your Strings?

Source: https://www.facebook.com/knitsforlife

Intro

A question about using plain text strings for passwords was recently asked on /r/PowerShell. The poster was making a wrapper for LastPass’s CLI and wanted to know if they should be using [System.Security.SecureString] objects. This question gets asked often and my stock answer is to always use [SecureString] objects to house secrets in memory regardless of how frequently the secret is converted from or to a plain text string.

My stock answer has had some pushback in the past. The problem is that when you do convert a [SecureString] to a normal string, that string object now exists in memory as plain text. If you know anything about how the CLR garbage collector works, you will know that the string may even hang around in memory long after the variable that housed it has been destroyed or overwritten in the code. Effectively, once you convert a [SecureString] to a normal string, the plain text secret can reside in memory until the program/script exits. The argument against using [SecureString] objects that will be converted to and from plain text is that it adds a level of complexity to the code for no effective gain.

This argument makes my eye twitch every time I see it. I’m a huge proponent of layered security and believe that security should be baked into every level of the stack every with chance possible. The idea is that we never know where our code will end up and we do not want our code to be the weak link in the chain. While the blame rests with the person who uses your insecure code in their sensitive environment, I don’t think we are totally without fault if we didn’t make an effort to be secure in the first place.

This is especially true with password manager wrappers. I have reviewed no less than 40 PowerShell based password manager wrapper modules and scripts in the past 2 years. The overwhelming majority of them are not using [SecureString] objects. When the lack of [SecureString] usage is pointed out that “inefficient complexity” argument  invariably rears its head.

“Mark, [SecureString] objects should never be converted to plain text the first place!” Let me remind you PowerShell is glue. It is being used to glue together various APIs. Many of these APIs are not Windows native or local and therefore don’t accept [SecureString] objects. This makes it necessary to convert the [SecureString] to plaintext and either submit it as plain text or encode it in some way. Also, sometimes we are accepting secrets from APIs, such as OAuth access tokens, and we don’t want these sitting around as plain text. It becomes necessary to convert back and forth quite a bit.

I wanted to see what can be done about this and to get a deeper understanding of the problem myself. In this post I will go in-depth with when and where [SecureString] objects give up their plain text secrets and how we can add some security around that process. This post will likely be a stretch for those new to PowerShell and is not intended as an introductory how-to.

Let’s Kill Write-Output

Intro

I think it is time we put down our old friend Write-Output for good. That’s right, Write-Output and not Write-Host. You should definitely not be using Write-Host outside of functions beginning with the Show- verb in console driven PowerShell apps, but I’m not here to kill off Write-Host. No, my target is Write-Output.

What is this craziness I’m speaking? Well, if I went back in time to 10 months ago and told myself not to use Write-Output, I would think my future self had gone mad. In fact, my journey towards murdering Write-Output began around that time.  There was a post in the /r/PowerShell  (sadly, I can’t find it in my history) where I was “correcting” someone for not using Write-Output. This prompted another user to link me to this discussion. At first I rejected the notion, but over time I came to embrace it. And today, I think I was crazy for not killing it off sooner.

What I’m saying here wont be new or revolutionary. My only hope with this post is to expose others to the issues with Write-Output and contribute to its demise.

What’s my beef with Write-Output? There are three concerns I have about using Write-Output: Performance, “Security”, and a “False Sense of Security”.

Before you begin, please understand that this is not an introduction-level PowerShell topic. This post was written with experienced PowerShell users and those who provide guidance and training to novice PowerShell users in mind. Also, this post is not specifically about console output, but about the use of the Write-Output command in general and its common use for “putting objects in the pipeline” (e.g. to “return” objects from a function). Most of it applies whether the objects ultimately go to the console or not. The only section dealing with console output is the Text Output section. Even then, the output for the targeted scenarios is more often a log file than it is a console.

Permission Granted: Using SharePoint Online, Flow, Azure Automation and PowerShell to Automate OneDrive for Business Permission Requests

Intro

A few weeks ago a request came through to create a group that would have full access to all OneDrive for Business accounts in our Office 365 Tenant. I’m am patently against blanket access to things, even for administrators. It turns out the goal was to enable our Service Desk staff to manage user’s OneDrive’s as we ramp up our adoption rate through various “to the cloud” projects in the works.

We have a very small team who have admin rights to our SharePoint Online and we are wary of granting frontline technicians admin rights to it as it is a complex beast and there is sensitive data to consider. Currently, all requests to access another user’s OneDrive requires an escalation to that small team. This creates a constraint that isn’t much of a problem today, but will become one as our adoption rate grows.

I identified two user stories we needed to support:

1. IT Technicians needing temporary access to a user’s OneDrive to assist them with various tasks
2. Users needing to access the OneDrive of another user permanently (e.g. a manager needing access to the OneDrive of an employee on Extended leave)

We already have automation built into our leaver process which grants managers access to their leaver subordinate’s OneDrive for 30 days. Unfortunately, that functionality is tightly coupled with the leaver process so it can’t really be used for these two user stories.

I am a big fan of SharePoint lists. They make it really easy to make a web based form and tracking mechanism that can support RBAC. I’m also real big on PowerShell automation. I have quite a bit of automation involving both in production. One thing I don’t like is that most of this automation is on a scheduled basis and not a trigger basis. I noticed that SharePoint Online now has triggers for Microsoft Flow and that Microsoft Flow added the ability to run Azure Automation Runbooks. So that’s where I decided to go.

This blog will cover the temporary admin access solution. It’s intended to be more of an overview and not a deep dive or tutorial. This is a PowerShell blog, but most of this post will be taken up by Flow as it is the glue of the solution. However, I won’t go into great detail of Flow either.

Out-Default: Secrets Revealed

Intro

You may or may not know that Out-Default plays a very important but silent role in your everyday use of of PowerShell. If you don't know the role it plays, you may want to check out this post by Jeffry Snover. I had always had my suspicions about how the Console Host in PowerShell works, but during a recent discussion on /r/PowerShell I was linked to Snover's post and had many of my suspicions confirmed directly from the source.

But, after reading, re-reading, and re-re-reading the post I was left wondering about the specifics of when Out-Default is actually called. It wasn't clear to me if it was only attached to a line of input from the console, or attached to the end of every line of every piece of processed code, or if this done in a non-interactive session or not. I decided it would be fun to investigate this using PowerShell itself and even more fun to share my process and findings.

Test Methodology

One of my favorite things about PowerShell is how easy it is to investigate and manipulate everything in PowerShell except for the language primitives. Pretty much any function, cmdlet, or alias can be "hijacked". What I mean by "hijacked" is that you can replace even core functions like Out-Default with your own code. While this obviously isn't always a good idea, has a few security implications, and some major performance impact implications, it also has some amazing investigative use-cases.

The setup for this investigation is simple. I'm going to create a proxy function to wrap around Out-Default. Then I'm going to increment a Global variable each time the proxy function is called and update the contents of a file. This is akin to early web site visitor or hit counters. The reason for using the file instead of just looking at the variable itself is that we are trying to reduce the observer effect. If we were just to look at the variable contents every time and that happened to trigger a call to Out-Default, it would update the variable and pollute our results.The variable needs to be Global scoped because we need to test Out-Default being called in all scopes.

We will then run a series of lines of code. The same tests will be done from the PowerShell console, PowerShell ISE, and from a non-interactive session. For both the PowerShell Console and ISE we will test running individual lines, grouped lines, and multiple commands run on the same line separated by semi colons. After each test we will investigate the output file and the number contained therein should reveal the number of times Out-Default was run.