Part 6
In Part 5 we configured the AWS CodeCommit to trigger the AWS Lambda when a commit is made to the master branch of the repository. Effectively, our CI/CD pipeline is in place. To use it properly, we first need to add a cc2af.yml configuration file. After the configuration file is there, we can push out first Azure Function to our AWS CodeCommit repository and our AWS Lambda will be triggered and start a manual deployment on the Azure Functions Web App.
We will finish out the series in this post with demonstrating an automatic deployment from AWS CodeCommit to Azure Functions and triggering our Azure Function all from PowerShell.
Series Table of Contents
- Part 1 of 6: Project Introduction and Overview
- Part 2 of 6: Project Settings, Authentication, Azure Function App Deployment, and AWS Code Commit Deployment
- Part 3 of 6: Creating AWS IAM User for Git, Generating HTTPS Git Credentials, Configuring Azure Web App External Git Deployment, and Initial Manual Deployment
- Part 4 of 6: Create AWS IAM Role, Deploy the C# .NET Core 2.0 AWS Lambda, and Create an AWS KMS Key
- Part 5 of 6: Add AWS CodeCommit Trigger and Encrypt Secrets with AWS KMS
- Part 6 of 6 (This post): Generate Configuration File, Automatic deployment from AWS CodeCommit to Azure Functions, Trigger the Azure Function from PowerShell, and Series Conclusion
Generate the cc2af.yml Configuration File
The AWS Lambda consumes a cc2af.yml file from the repository. When the Lambda receives the CodeCommit event, it pulls the most recent commit from the master branch and downloads the cc2af.yml file from the repository. The name and location of this file can be changed by setting the CC2AF_CONFIG_PATH environment variable in the Lambda configuration. The name cc2af.yml is an initialism for CodeCommit To (2) Azure Functions. It is a YAML file. Hopefully I don’t need to explain what YAML is at this point. I suspect anyone who has read this far into the series is familiar with them.
The cc2af.yml file consists of the following settings:
- DeploymentUser – The Username for the Azure Functions Web App deployment.
- DeploymentPassword – The Password for the Azure Functions Web App deployment. This will be an encrypted string.
- DeploymentTriggerUrl – The deployment trigger URL for the Azure Functions Web App.
- DeploymentAppName – The name of the Azure Functions Web App.
- CodeCommitUser – The Username for the AWS IAM user HTTPS Git Credentials.
- CodeCommitPassword – The Password for the AWS IAM user HTTPS Git Credentials. This will be an encrypted string.
- CodeCommitBranch – The Branch to deploy from (this allows you to have a separate branch from master as the deployment source).
Unfortunately, the settings are case sensitive. I’d have to do more digging to figure out how to make YamlDotNet parse as case insensitive.
To create the cc2af.yml, I use a string template, pass in the required data, and then set the contents of the YAML file.
As a reminder, we changed directory to the local git repository. the cc2af.yml is being created in root of that repository.
Push the cc2af.yml and Azure Function cod to AWS Code Commit
And now, the moment we have been working for: Pushing an Azure Function to AWS CodeCommit and having it automatically deployed to Azure Functions. The demo function included in the project is a very simple one. It accepts a name and returns a string which includes the name. We need to copy the demo function to the local git repository, commit the changes, and push them to CodeCommit.
This isn’t real impressive code. It’s basically just normal git commands. The real magic is happening behind the scenes. After our push, AWS CodeCommit triggers the AWS Lambda and sends it the CodeCommit event. The Lambda then triggers the manual deployment of the Azure Functions Web App. The Azure Functions Web App then pulls the AWS CodeCommit repo in and works its deployment magic.
Verify the CI/CD Deployment
After a few minutes we can verify the Azure Function was successfully deployed to Azure Functions from AWS CodeCommit. It’s important to note there is some latency in this. There is a delay between the commit to CodeCommit and the trigger of the Lambda. There is a start up delay of the Lambda and the run time of the Lambda takes a few seconds. Then there is some delay between the Azure Functions Web App deployment trigger and when the code is live. In testing, this took anywhere from a few seconds to 5 minutes. The nature of modern workflows and event driven architecture is ripe with latency like this. I don’t think it’s such a bad thing, but it’s important to note that if you don’t wait for some time to pass before verifying the deployment, you might mistakenly think it did not succeed.
There are 3 places we need to verify the deployment: Azure Resources, The Azure Function Web App deployment log, and the AWS CloudWatch logs.
First, we grab the Azure Functions defined in the Azure Function Web App
Next, we prep some headers we will need to make calls the Azure Functions Web App Kudu management API.
Now we grab the deployment history for the Azure Functions Web App from the Kudu management API.
Finally, we grab AWS CloudWatch Log Stream for the AWS Lambda
With all the information in hand, we can use Pester to verify our first automated deployment was successful:
If all wen well, you should see this:
Display Deployment Logs
If you want to look though the logs, you can do so in both the AWS Management Console and Azure Portal. But, it’s more fun to view them with PowerShell!
You may have notice we collected the AWS CloudWatch Log Stream info but didn't use it in Pester. Technically, the deployment only really happens on the Azure side. You might be able to pull some CodeCommit event logs and see the pull being made from Azure.
Anyway, you can run the following to view the logs in PowerShell
There is a ton of output for this, so I’ve created a gist for it:
The Azure Web App Deployment Log section shows all of the steps taken in Azure to pull the commit from the AWS CodeCommit repo and deploy the Azure Function.
The AWS CloudWatch Log for Lambda section shows the Lambda execution log. You will notice the state of the application was logged. In the C# Lambda, I have a section where I take all the relevant environment and run time variables and place them in a dictionary and then convert that dictionary to JSON and write it to the console. Writing to the console in a C# lambda will result in the data being logged to CloudWatch.
I redacted the secrets and some sensitive data, but the secrets are actually encrypted with AWS KMS and the sensitive data isn’t really all that sensitive. But, I just didn’t want to take any unnecessary risks.
Retrieve the Azure Function Key
So now that we know our automated deployment was successful, we want to test our deployed Azure Function. Before we can run our Azure Function, we need to retrieve the Function Key. The Function Key is passed in to the Azure Function to grant access to run the function. Any attempt to run the function without the key will result in access to the Function being denied. This security is configured in the function.json in our PBnC function source code.
To retrieve the Function Key, we first need to retrieve the Master Key. To retrieve the Master Key, you have to use the deployment credentials. If you were starting form scratch and you wanted to get the Function Key programmatically, you would need to follow this flow: Authenticate with Azure, Retrieve the Web App Publisher Profile for the Azure Function Web App, use the Deployment Credentials to obtain the Master Key from Kudu, Use the Master Key to obtain the Function key from Kudu.
Trigger The Azure Function From PowerShell
Now that we have our Function Key, we can trigger the Azure Function. All it requires is a call to Invoke-RestMethod with the correct URL. You don’t need to be authenticated anywhere. Authentication is based solely on the presence of the Function key. Securing serverless APIs is out of scope for this blog series, but you should have a care about this and look at the different security options available for Azure Functions.
If you look at the URL we are generating here, you will notice the name query field contains “Mark Kraus”. This query field will be consumed by the Azure Function to produce output. The code query field contains our Function Key.
You may notice the function takes some time to run. if this is the first time the Azure Function has run (either ever or in a certain period of time) there is some initialization that needs to take place in the background before it is run. Subsequent runs will be quicker. You can dodge this initialization cost by using a dedicated Web App model for your Azure Function Web App. Of course, that means that rather than being on a consumption plan you would be on always-on plan. You can control the cost by stopping your Web App service when your Functions are not needed, but you then incur the same initialization costs when you need to start it again to run your functions on demand.
And now for the moment of truth… Time to get the results of all our labor up to now:
And this the result:
That’s right, we have successfully put Peanut Butter in the Chocolate!!
Clean up
I provide some clean up code commented out in the end of the Configuration.ps1. You can use this code to remove all the Azure and AWS resources created in the project. In the AWS Management Console you need to remove the HTTPS Git Credentials from the IAM User before you can remove it in PowerShell. Everything else can be cleaned up in PowerShell.
Series Conclusion
So, that’s it! We have put Peanut Butter in Chocolate by having AWS CodeCommit automatically deploy our Azure Function when a commit is made to the master branch by using an AWS Lambda. I hope this series was informative and at least somewhat entertaining.
The entire idea of cross-cloud resourcing is interesting to me. I have spent much of my career working with cross-platform solutions to cross-platform problems. It seems there will be a niche now and in the future for cross-cloud solutions.
I don’t think any company should limit themselves to one cloud provider out of blind loyalty. Use the features in the cloud provider which makes sense for your needs. If a cloud provider wants your business, they should work on feature parity with other cloud providers to make their service attractive to use for everything.
Working cross-cloud, just like working cross-platform, comes with its own costs, benefits, risks, and rewards. So, you will need to weigh all the options before designing your solution. But, please keep cross-cloud solutions on your radar. It’s doable!
I bring this up because this project was born fro a lack of feature parity in AWS with Azure Functions and Azure Automation. Yes, there are options I can do in AWS using SSM and a host of other combined services. but, none of them will be significantly more expensive than what I can do in Azure. The problem lies in AWS not having PowerShell or Windows based workloads for serverless. .NET Core is there, but many Windows APIs are only available on Windows, even for .NET Core and especially for PowerShell.
Finally, props to PowerShell! It was able to glue together the 2 clouds almost seamlessly. The PowerShell team and Jeffry Snover are not just spouting a line of bull when they say they are committed to making PowerShell cross-platform and cross-cloud. This project is a good example of that commitment coming to fruition. This wasn’t exactly easy, but it wasn’t as difficult as I originally envisioned. AWS HTTPS Git Credentials is the only barrier to fully automating this solution in PowerShell. I still need to raise a ticket with them to see about opening an API for that. But PowerShell did an amazing job!
I hope you enjoyed the series! I have a few blogs in the pipe, but I need to devote some spare time to the Web Cmdlets so I can meet my roadmap goals for PowerShell Core 6.1.0. So this blog may be a bit quiet for a few weeks.
