Source: https://www.facebook.com/knitsforlife
Intro
A question about using plain text strings for passwords was recently asked on /r/PowerShell. The poster was making a wrapper for LastPass’s CLI and wanted to know if they should be using [System.Security.SecureString] objects. This question gets asked often and my stock answer is to always use [SecureString] objects to house secrets in memory regardless of how frequently the secret is converted from or to a plain text string.
My stock answer has had some pushback in the past. The problem is that when you do convert a [SecureString] to a normal string, that string object now exists in memory as plain text. If you know anything about how the CLR garbage collector works, you will know that the string may even hang around in memory long after the variable that housed it has been destroyed or overwritten in the code. Effectively, once you convert a [SecureString] to a normal string, the plain text secret can reside in memory until the program/script exits. The argument against using [SecureString] objects that will be converted to and from plain text is that it adds a level of complexity to the code for no effective gain.
This argument makes my eye twitch every time I see it. I’m a huge proponent of layered security and believe that security should be baked into every level of the stack every with chance possible. The idea is that we never know where our code will end up and we do not want our code to be the weak link in the chain. While the blame rests with the person who uses your insecure code in their sensitive environment, I don’t think we are totally without fault if we didn’t make an effort to be secure in the first place.
This is especially true with password manager wrappers. I have reviewed no less than 40 PowerShell based password manager wrapper modules and scripts in the past 2 years. The overwhelming majority of them are not using [SecureString] objects. When the lack of [SecureString] usage is pointed out that “inefficient complexity” argument invariably rears its head.
“Mark, [SecureString] objects should never be converted to plain text the first place!” Let me remind you PowerShell is glue. It is being used to glue together various APIs. Many of these APIs are not Windows native or local and therefore don’t accept [SecureString] objects. This makes it necessary to convert the [SecureString] to plaintext and either submit it as plain text or encode it in some way. Also, sometimes we are accepting secrets from APIs, such as OAuth access tokens, and we don’t want these sitting around as plain text. It becomes necessary to convert back and forth quite a bit.
I wanted to see what can be done about this and to get a deeper understanding of the problem myself. In this post I will go in-depth with when and where [SecureString] objects give up their plain text secrets and how we can add some security around that process. This post will likely be a stretch for those new to PowerShell and is not intended as an introductory how-to.
Stealing Secrets from PowerShell Processes
Before we get into the code we need a way of seeing plain text strings in-memory of a PowerShell session. We also need a valid testing environment to simulate script execution and an attacker sniffing memory. To do this I will be using Cheat Engine (CE). Really, we could use any debugger, however, I happen to be more comfortable and experienced with CE.
The general workflow will be like this
- Start CE
- Start cmd.exe
- Runpowershell.exe -nologo -noprofile -file c:\temp\StringTest.ps1
- Attach CE to the PowerShell process
- Scan memory and test
- End the PowerShell session
- Change the code in StringTest.ps1
- Repeat steps 3 through 8
As a test of this, I set StringTest.ps1 to this
and ran the PowerShell command from step 3.
This allows me to pause PowerShell and get CE attached to the process. CE lists the PID in hex format so I have PowerShell converting the PID to hex to make it easier to see which process I should attach CE to.
Before continuing, I want to see if the secret is in memory. If you look at the PowerShell code, you will notice I define the $PlainText variable after the initial Read-Host. In theory, this variable does not exist yet. But, if we scan for testtesttest in CE, we can see the secret is already in memory.
That makes sense, if you think about it. The entire code of the script itself is copied into memory. Even if the variable has no yet been defined, the plain text secret is already there in memory.To my surprise it is there twice. I originally thought this might be because of script being loaded into memory and again as it is converted to MSIL, however, hitting enter to continue and thus populating the $PlainText variable did not add a third entry.
This indicates to me that assignment operations are done out of order when they are static. While interesting, this doesn’t effect my testing. What I have done is to successfully scan a plain text string from the PowerShell process’s memory. I hit enter to end the PowerShell session and we can see in CE that the values have disappeared as part of the exit cleanup.
Emulating a Password Manager
PowerShell has a pretty awesome feature baked in called CLIXML. It’s a great way to store and retrieve PowerShell objects to and from the disk. It has benefits over JSON and CSV in that it maintains the type definitions fairly well. It’s not perfect, but one of its major uses is to securely store secrets, You can import and export [SecureString] objects and they will remain encrypted in the XML file in a manner that can only be decrypted by the original user who exported it and only on the original computer where it was exported.
I am going to use this to simulate the Password Manager scenario. To do so I run the following in PowerShell:
For the user I’m just putting gibberish as I don’t care. But for the password, I’m using testtesttest. This will be our secret that we scan for from here on. The Password property of a [PSCredential] is a [SecureString]. The [PSCredential] class provides a convenient GetNetworkCredential() method for retrieving the plain text version of the [SecureString]. I tend to use [PSCredential] for secrets even when the username is not populated just so I can make use of this method.
I updated the StringTest.ps1 to this
Then I ran the PowerShell command and attached CE to the PowerShell process.
I scan for the secret and this time it is not found in memory at startup.
I hit enter to continue and then scan again.
The secret is nice and safe inside the [SecureString]. Now we need to test having it give up its secret. I updated the StringTest.ps1 to the following:
I run the PowerShell command, attach CE to the process and then scan after the [SecureString] is in memory but before it has been converted to plaintext and stored in the $String variable.
As before, the secret is nice and safe. I now hit enter to progress the script and load the secret as plain text into the $String variable, and then I scan for the secret.
We can see that the secret is now visible as plain text in memory.
At this point we have a valid testing process for when a [SecureString] has given up its plain text secret and have a good starting point investigate possible ways to clear that string from memory.
Removing Variables Doesn’t Remove Them From Memory
The first thing I wanted to test was various ways of “deleting” variables. Logically, one would assume that once a variable is “deleted” it should no longer be in memory. That’s not exactly the case with the .NET CLR that PowerShell runs on. There is a garbage collection process that runs to clean up unused memory. This is all a bit beyond me as I only have the most basic understanding. The important part for PowerShell, however, is that clean up is not immediate and while variables may no longer be accessible in the PowerShell code, the actual values they held still exist in memory until the CLR’s garbage collection decides to take out the trash.
This is easier to understand with a demonstration or four.
Remove-Variable
I updated the StringTest.ps1 to the following:
Following my established workflow, I scanned the memory after the variable has been “deleted” by Remove-Variable.
Even though the variable has been “deleted” the value it held is still in memory. I’m not sure when it would actually disappear from memory, but after 10 minutes it was still there. I won’t be waiting on the other tasks.
Clear-Item
Next on the list of things to try is using Clear-Item to “delete” the variable.
Same process as before except using this in StringTest.ps1:
The scan after “deleting” the variable:
Well, here’s something interesting. Not only did Clear-Item not remove the value from memory, it actually duplicated it for some reason. I don’t have a good theory for why that is. I might dig into the source code at a later date and try to figure out what causes this. In any case, if you ever thought Clear-Item was helping you secure code you were sorely mistaken.
Assigning $Null
Ok, so “deleting” the variable doesn’t seem to work. So, what about writing over it? In this test, I’m assigning $Null to $String in an attempt to overwrite the value.
StringTest.ps1:
Scan result:
Still no dice.
Assigning Gibberish
Perhaps $Null is special in some way. In this test I’m assigning a gibberish string to try and overwrite the value in memory.
StringTest.ps1:
Scan result:
Again we see that the secret still persists in memory.
Perhaps They Are Right?
This is the crux of the argument against [SecureString] objects. The level of complexity it adds to the code appears to offer no real value if that [SecureString] is ever converted to plain text. Once it is in memory as plain text it often lingers indefinitely. Or does it?
Surely Scopes Shall Save Us!
One piece of advice I see parroted all over the place is that if the variable is properly scoped it will be collected properly when the scope is removed. The theory is that variables in a scope, such as a function or cmdlet, are removed from memory when the scope ends (such as when the function returns back to the calling scope). So far I have been testing directly in the “global” scope. Obviously, the global scope is more persistent than a function. We are good programmers who create lots of functions that Do One Thing and one of those One Things our functions do is convert the [SecureString] to a normal string. We are safe! Right?
Simple Function Test
This test moves the storing of the secret in plain text to a function scoped $String variable. In theory, this variable no longer exists after the function ends.
StringTest.ps1:
Scan result:
Clearly, the value is still in memory after the function scope has been destroyed. I want to note that I created this test and ran it before writing this section and then took the screenshot. At least 5 minutes have passed between my execution and the above screenshot.
Thermonuclear Function Test
Ok, it’s still in memory. But, I didn’t do anything to try and remove it from memory. Let’s try to nuke it from orbit just to be sure. I will combine all the other tests (except Clear-Item).
StringTest.ps1:
Scan result:
Nope, still there. The secret still persists in memory after nuking it from orbit in a function scope.
So… Again… Perhaps They are Right?
Ok. Ok. Maybe they are ri- Just kidding. They are wrong I just have yet to prove them so. It’s clear that scopes are of no more help than anything else. But when I haven’t brought out the extinction level event which will truly kill this cockroach like pest.
Time to Take Out The Trash
It’s a Wednesday and at my apartment complex, that means trash pickup day. Perfect timing for this section.
As I have mentioned, much of why the value remains in memory after the variable that housed it has been deleted is due to the CLR’s garbage collection. As it turns out, we can actually somewhat control that from within PowerShell. I say “somewhat control” because, as I understand it, the CLR will always do what it wants when it wants, but we can gently encourage it do what we want it to. It still makes the decisions, we just persuade it to make the decisions that we favor. Much like a Mafioso applying leverage to a patsy.
The secret to this is the System.GC class. This class provides us an interface to the garbage collector (GC). We can cause the garbage collector to clean up the unused memory with the following:
This calls the Collect() static method. After it is called the CLR will be persuaded to start collection ASAP. Soon after, any variables we removed or modified in PowerShell will have the lingering values cleared from memory.
Remove-Variable
StringTest.ps1:
Before collection:
After collection:
Clear-Item
StringTest.ps1:
Before collection:
Again, note the double entry when Clear-Item is used.
After collection:
Well, at least one of them goes away.
Assigning $Null
StringTest.ps1:
Before collection:
After collection:
Assigning Gibberish
StringTest.ps1:
Before collection:
After collection:
Collection in Function
In this test I use a $Null assignment and then collect before returning to the calling scope.
StringTest.ps1:
Before collection:
After collection:
Collection After Function
In this test I let the scope be destroyed and call the collection after the function has returned.
StringTest.ps1:
Before collection:
After collection:
Perhaps They Are Wrong?
They are absolutely wro- well… kind of wrong. I have been doing fresh scans for the full secret string after collection. It’s true that the full secret is gone from memory, but not all of it. If I don’t do a fresh scan in CE and just leave it watching the memory location, this is what happens:
You can see that the secret is still half there. It has been truncated but not completely overwritten. The same happens in every test.
Conclusions and Takeaways
It is possible to somewhat remove plain text strings from memory when they have been converted from [SecureString] objects. This is not a default behavior and requires a call to [System.GC]::Collect(). In my opinion, if you are storing secrets in memory, you should still do so with [SecureString] objects. You should also be making calls to [System.GC]::Collect() as soon as possible. It would be my personal preference to assign $Null to the variable and then immediately start collection.
I would do this in the scope where the variable is used and not rely on the parent scope to do it for you. If you are working with 3rd party code that is consuming your secrets, you should run collection as soon as their code returns.
We should also recognize that this is not perfect and that some of the secret may still linger in memory. With that in mind, we should limit the number of times as well as the amount of time a secret is converted to plain text. Do it right as you need to, only when absolutely necessary, and collect as soon as possible.
For myself, this means I need to make some updates to my projects to force garbage collection as i have clearly demonstrated that nulling variables and relying on scope cleanup is insufficient.
Also, never use Clear-Item to try and delete secrets.
Is this paranoia? I don’t think it is. The argument is often made that if someone is sniffing memory you have bigger things to worry about than secrets in PowerShell. My counter, however, is that we don’t know where our code will be used and regardless of the “bigger problems” we should still do our best to ensure that at least our sandbox is clean. Maybe an attacker was only able to get memory sniffing up and running and your loose code led to them gaining a password you only used once in a 5 hour process. Maybe your piece of code is only one function in the entire 40 thousand line code base. You just don’t know.
If you do know you are working with secrets, you have no excuse not to try and keep them secret, in my opinion. Just because it adds complexity or lines to your code doesn’t mean using [SecureString] objects is not practical, even for “quick and dirty” jobs. Little code snippets have a way of turning into larger code that gets integrated in places you wouldn’t imagine. So, yes, even when you are working with secrets in a “one off” script, you should still try and keep them secure, even at the cost of time and complexity doing so adds.
Finally, never type secrets in the consoles. This include ISE, the PowerShell Console, and VS Code. I was originally planning on testing this from one of them to make it easier. What I discovered is that if you ever enter a secret on a line and hit enter, that secret now exists in 3 to 5 memory locations. On Windows 10 they also persisted in my command history between sessions of the PowerShell console. It is ok to enter a password when prompted by Get-Credential. Just never enter a command like this:
My guess is that the process in the console, ISE, and VS Code which parse code for things like tab completion require that the duplicate and process every command entered. In any event, just don’t do it.
Addendum: Secrets Sent To Other Programs
/u/sk82jack asked “Did you look at what happens if you pass a converted string into another command/program without storing it to a variable? “ I did not, so I will do it here.
StringTest.ps1:
Before collection:
After collection:
It appears not everything gets cleaned up by [System.GC]::Collect(). I also tested the following:
and this as well
They both had the same results with 4 lingering entries. I’m not sure how to make those go away.
It looks like if we throw the task that sends the credential to the exe in a job, the secret does disappear:
Scan result:
Jobs are run in a separate PowerShell thread. To make sure this was actually exposing the secret temporarily in the other thread, I added a Start-Sleep into the jobs script block and then sniffed on the child process.
So when the job exits, the child thread is killed, the normal end of thread cleanup is done, and the secret is nowhere to be found in memory. Therefore, if you must pass a plaintext secret to an exe, do it as a job.