Automated Documentation in a CI/CD Pipeline for PowerShell Modules with PlatyPS, psake, AppVeyor, GitHub and ReadTheDocs
- Part 1: Intro, Pre-reqs, Components and Build Workflow (This Post)
- Part 2: GitHub Access Token, ReadTheDocs Account & Project, and Comment Based Help
- Part 3: mkdocs & Release Preparations and Pushing the Release
- Part 4: Monitoring the Build, PowerShell Magic, Looking Forward, and Closing Thoughts
Part1: Intro, Pre-reqs, Components and Build Workflow
Intro
One of my least favorite parts of IT is documentation. It
takes a significant amount of time to document things, sometimes longer than
the actual work you are documenting. I’m not alone. I hear complaints from all
levels about how much of a pain and bore documentation is. However, we all
depend on it. The moment things don’t go as planned or expected we start
googling for solution and digging through documentation hoping to find the
answer. So, while documentation is absolutely evil, it is a necessary evil.
With PowerShell, and other programming languages, there is
an added burden of needing to document in more than one place. For example, you
may need in-code documentation for what you are doing and why, programmatic
help systems (like PowerShell’s comment based help), external help, internal
help, and online documentation. It can get overwhelming and will often dissuade
people from documenting their code at all or at the very least result in their
only documenting in one place.
I recently published my PSMSGraph module to the PowerShell gallery. One of my pet peeves about many gallery modules is that there is just
no documentation. Some don’t even include comment based help or any useful
comment based help. You are left with a trial-and-error based approach to using
them. I wanted to avoid that for anything I publish to the gallery. I’m not
trying to put anyone down, but I just wanted to make my own module easy, more
accessible, and at least somewhat documented.
Personally, a year ago, I promised myself to commit to
comment based help on everything. Every script and function I write now include
comment based help. I really didn’t want to duplicate my efforts in documentation.
Since I always have comment based help I began searching for a method of
converting that to markdown and I stumbled across a few scripts until I found
PlatyPS. For my ConnectReddit module I had very manual process where I ran a
script to generate markdown documentation and then it had to be pushed to the
separate wiki repo. It was painful and I hated it. Since PSMSGraph is now in a
CI/CD pipeline, I spent some time integrating PlatyPS into my pipeline. Now I
don’t even need to think about the documentation part outside of keeping my
comment based help updated.
This 4 part blog series will cover how to implement this in your
pipeline. Once done you will have automated function online documentation as
part of your pipeline as well as an automatically managed Change Log based on
your Release notes.
Before you Begin
Please note that this blog series is not intended as an
introduction to PowerShell. This blog is intended for those already familiar
with PowerShell.
This blog series builds on the work of Kevin Marquette and Warren
Frame. They both have excellent blogs on how to implement a CI/CD pipeline for
module development. This blog is written with the assumption that you are
familiar with GitHub, AppVeyor, and psake. If you want to learn those things,
please stop here and go read their blogs.
Required reading:
- Kevin Marquette:
- Warren Frame
Get-Help about_comment_based_help
Make sure you have a decent understanding of those concepts and technologies before moving on with this series or you are sure to be lost. I will try my best not to rehash anything covered in their post except where my process or architecture differs.
Also, I recommend
taking a primer on both markdown and YAML. I don’t claim to be an expert on
either, but some basic familiarity with them will be needed for this.
Finally, you might want to look at the PSMSGraph project and
you will definitely want to look at the AutoDocumentsExample project that I
will be using for this series. With the PSMSGraph project you can see the
entire build process from start to finish and how it ends up looking in the
PowerShell Gallery.Components
Before I get into the nitty-gritty I figured it was best to
name and describe the basic components that will be used to accomplish the goal
of a CI/CD pipeline with automated documentation building. If you have done the
“required reading”, some of this will be a review for you.
PlatyPS
PlatyPS is a PowerShell module for generating documentation.
For our purposes, it has a very convenient capability of creating markdown help
documents for an entire module. It has a lot of other features for external
documentation that I will want to look at in the future.
AppVeyor
AppVeyor is a CI/CD build service. I like to think of it as
a cloud Jenkins. It builds, tests, and deploys code. It also happens to be free
for Open Source projects. Best of all, it supports PowerShell 5 which will be
used to run the psake build.
psake
psake is a DSL build automation module for PowerShell. It
makes it easy to create build automation processes. It makes it easier to code
your process as a set of tasks and dependencies. The /psake.ps1 in the project
is where most of the magic happens. It is run in the AppVeyor environment and
makes all our calls to the PlatyPS functions as well as recommitting our changes
ReadTheDocs
ReadTheDocs is a community supported documentation build and
host service. It supports mkdocs which turns markdown documents into HTML web
pages. It supports a range of other documentation engines, but since we are
using PlatyPS to generation in markdown, we will make use of the mkdocs engine.
If you use their service, please consider contributing time or money. You can
find more details about ways to contribute on their website.
GitHub
If this is the first you have ever heard of GitHub, this
blog post is not for you. The role GitHub obviously serves here is the public
version control repository for our code. When we commit to the repo it will
fire off the webhooks for both AppVeyor and ReadTheDocs which will
automagically fire up the build pipeline.
Overview of the AutoDocumentationExample Repository
If you read the “required reading” this structure should
look mostly familiar to you. I will go over the pieces that are different or
standout in this configuration. For convenience, I will used / to refer to the
top level of the project folder and refer to all files as their path name from
/.
/appveyor.yml
Our appveyor.yml file differs slightly in that we are also
including an access_token secure environment variable. This is a GitHub access
token you will need to generate and then convert to a secure string in your
AppVeyor account. This is used to write our changes from the build in AppVeyor
back to the repo. This can be named anything but /psake.ps1 will need to be
updated to reflect whatever name is given.
/RELEASE.md
RELEASE.md is where we will put our Release Notes for the
current release. As the .md file extension indicates, the file will be in
markdown. There will be more details on what to put here later in the Release
Preparation section. This file will be copied to the /docs/ folder and added as
the ReleaseNotes field on the module manifest. This means that the text here
will show up on the ReadTheDocs site as well as the PowerShell Gallery. psake
also prepends this to the /docs/ChangeLog.md file
/mkdocs.yml
In Kevin’s blog, he brought this up but said he would go
into detail later. Basically, this is used by ReadTheDocs’s mkdocs engine to
configure and execute the documentation build process. For our project, we do
not write to this directly. In fact, when starting your project, you do not
need to include this. psake will overwrite or create this file based on what is
in /header-mkdocs.yml , the release notes, the change log, and the function
documentation generated by PlatyPS.
/header-mkdocs.yml
This is another new file. As stated above this is what will
be used to generate /mkdocs.yml
/psake.ps1
This file is not new, but, it is different from other
projects. There are 3 tasks that are of importance to this blog: Build,
BuildDocs, and PostDeploy. These are where the magic happens for managing
release notes, change log, documentation building, and committing build changes
back to the repo.
/docs/
This is the folder where the markdown files used by
ReadTheDocs will live.
/docs/index.md
This serves as the landing page and “Home” page for the
ReadTheDocs site
/docs/RELEASE.md
This copy of the /RELEASE.md is generated by psake.
/docs/ChangeLog.md
This is the automatically maintained change log. Basically,
this has /RELEASE.md prepended to it every build.
/functions/ and /functions/*.md
The function documentation generated by PlatyPS is stored in
this folder, one per function in the module.
/Tests/Help.Tests.ps1
The Help.Tests.ps1 for this project includes verifying the
function has a HelpUri defined and at least one related link.
Build and Documentation Build Flow
Now that the component and project architecture is out of
the way, let’s talk about the project flow that is used here. My flow is
probably not ideal and will probably be reworked over time. But, it’s
functional enough for now that those looking to add automated documentation to
their pipeline could likely drag and drop this into their project. The basic flow is this:
- Modify code in project
- Update comment based help for all public functions
- Erase contents of /RELEASE.md (left from the last release) add the changes made in this release
- Git add
- Git commit
- Git push
- AppVeyor webhook triggered (ReadTheDocs is also triggered but this document build will be overwritten once AppVeyor finishes)
- /Build.ps1 sets environment and runs /psake.ps1
- /psake.ps1:
- Initializes
- Runs quick unit tests
- Build:
- Populates functions in module manifest
- Populates aliases in module manifest
- Version bumps in module manifest
- Prepend version and date to /RELEASE.md
- Prepends /RELEASE.md to /doc/ChangeLog.md
- Runs all tests (unit tests are re-run to ensure build changes have not affected them)
- BuildDocs:
- Imports the freshly built module
- Grabs content of /header-mkdocs.yml
- Copies the /RELEASE.md to /docs/RELEASE.md
- Runs New-MarkdownHelp from PlatyPS to build the function documentation in /docs/functions/
- Generates the /mkdocs.yml based on the results of the above steps.
- Deploy (if triggered by !deploy in the commit)
- PostDeploy:
- Commits all the changes made by the build back to the repo with [ci skip]
- Creates a GitHub Release if !deploy is in commit.
- GitHub repo is updated with build changes
- ReadTheDocs webhook is triggered (AppVeyor ignores because of the [ci skip] in the commit message)
- ReadTheDocs builds the documentation website
- Git Pull (You need to pull the changes made from the build process back into your local Git repo).
