Random thoughts on technology and other stuff...

April 15, 2019

Native Kubernetes Continuous Delivery Part 3

Prow: Keeping Kubernetes CI/CD Above Water

SONY RX-100 ISO 125 15.45mm ƒ/5.6 1/640
Small Fishing Boat, Genova, Italy
Photograph by Kurt Madel ©2019
CDF CI CD Tekton Kubernetes Prow Jenkins X Native K8s CD

If you are doing CI and/or CD at scale and you aren’t leveraging Native Kubernetes Continuous Delivery (Native K8s CD) then you are just doing it wrong missing out on a better way - plain and simple. And if there is one Kubernetes project that has been at the forefront of Native K8s CD and best exemplifies the why and the how of what makes Kubernetes such an excellent platform for executing CI/CD at scale - it is Prow. Prow is a Native Kubernetes CI/CD system and as far as its name goes, Prow didn’t go the Greek route for its K8s-themed name, rather it went the nautical route with prow meaning the portion of a ship’s bow (the front end of a ship) above water - and Prow has been keeping K8s CI/CD above water for several years now. And others have taken notice, resulting in Prow becoming an important tool for a number of high-profile Kubernetes-centric projects. Prow is an important tool in the toolbox of another Native K8s CD project called Tekton, going the Greek route for its name which means carpenter, that I wrote about in a previous post of this Native K8s CD series. Tekton, along with a number of other important K8s projects, use Prow for their own CI/CD automation even though there are a number of other options. So what exactly is Prow and why should you care? Before we dive into that, I believe it is important to understand the origin and evolution of Prow to provide additional insight into why K8s is such an excellent platform for CI/CD.

Where did Prow come from?

When it came to CI/CD and the creation of Prow, Kubernetes was scratching its own itch with a need to execute over 10,000 CI/CD jobs per day across over 100 repositories of code in several GitHub Organizations - and other tools just weren’t getting the job done. So, the Kubernetes Testing Special Interest Group (sig-testing) created their own tools and services, to include Prow. Prow is currently part of the Kubernetes test-infra project and is one of several Kubernetes Testing Special Interest Group (sig-testing) top-level projects. At the time of this post, Prow was not even in its own GitHub repository, rather it was just a sub-directory (and previously named ciongke)in the kubernetes/test-infra repository. And partly because of the importance of Prow to the K8s project as a whole and also because there is so much interest from other K8s related projects, Prow continues to add features and flourish. Another testament to Prows continued growth is how top-level test-infra components continue to be replaced with Prow components - with one of the latest examples of this being the replacement of Gubernator with Spyglass.

What is Prow?

Prow is a lot - certainly more than I can cover in one post - but I will do my best. Prow’s GitHub README describes it as a system, and a very complex multi-faceted system it is at that. Prow is built on a container based microservice architecture and consists of a number of single purpose components - many integrating with one another while others are more or less standalone. The Kubernetes Testing SIG describes Prow as “a CI/CD system built on Kubernetes for Kubernetes that executes jobs for building, testing, publishing and deploying.” However, that description does not highlight perhaps the most important inferred capability of Prow - a capability that is at the heart of best-of-breed CI/CD automation tools - that capability is automation that starts with code commits - and in the case of Prow it starts with a scalable stateless microservice called hookthat triggers native K8s CI/CD jobs (among a number of things that hook does via plugins). It is this GitHub automation capability that has been one of the key reasons why other K8s projects have adopted Prow for their own CI/CD. But Prow is more than just GitHub webhook automation and CI/CD job execution. Prow is also:

Comprehensive GitHub Automation

Prow is like steroids for GitHub automation.

Prow is a GitHub Webhook Listener

This may be one of the most important capabilities of Prow and most of Prow’s other components depend on it in one way or another. Prow listens for GitHub webhooks via the hook component that in turn triggers one or more configured plugins to include the trigger plugin that triggers Native K8s CI/CD jobs for GitHub PR events, and the /test and /retest ChatOps commands.

Many non-Native K8s CI/CD platforms support GitHub webhooks but in many cases they are not highly fault-tolerant, not horizontally scalable and not stateless - so they become a single-point-of-failure (SPoF) for GitHub webhooks. If a GitHub webhook is sent while such a platform is down it will be completely missed - so you might miss a merge to your deployment branch and miss a mission critical change that needs to be deployed to production.

Prow ChatOps

ChatOps are an important component of Prow’s GitHub webhook support - Prow doesn’t just listen for pushes to branches, but listens for a number of more sophisticated GitHub events, like comments on issues and PRs using ChatOps commands. Prow ChatOps allows you to do everything from approving a PR with the /approve command to telling a /joke. Many of the Prow hook plugins provide ChatOps commands that streamline and automate repository contributors’ interactions with GitHub PRs and issues - so developers can focus more on the code and not GitHub process and UI. The complete list of supported ChatOps commands is available as one of the views of the deck component.

Prow Provides More Granular Directory Level Permissions in GitHub

Prow allows you to manage ownership/permissions at a more granular level than GitHub does. Prow allows you to create an OWNERS file at the repository level, but also within any repository sub-directory. This allows you to manage who can do what at the directory level instead of just the top-level repository level. The directory specific OWNERS files are utilized by a number of hook plugins and ChatOps commands - for example the blunderbluss plugin automatically assigns PR reviewers based on OWNERS files most relevant to the file(s) being committed - but the automated reviewer selection is more sophisticated than just using the most relevant/closest OWNERS file - it also weights reviewers based on the number of lines of code that changed and in what files - once again highlighting the incredible breadth and depth of Prow’s GitHub automation.

Prow is Automated GitHub Merges

Prow’s tide component automates the bulk testing of PRs and automatically merges multiple PRs. It replaced another test-infra top-level project called mungegithub. In addition to automated testing and merging of PRs - at scale, tide also provides Promethus metrics and serves live data to Deck - the Prow reporting dashboard.

Prow Provides GitHub Organization Settings and Team Membership Management

Prow allows you to manage GitHub Organization settings and team membership with Config-as-Code (CasC) so the source of truth for your Organization configuration is in one easily auditable file, rather than relying on one or more people making manual changes via the GitHub UI or manual API calls. The peribolos component provides automated auditable support for managing these GitHub settings with a CasC org.yaml file.

Prow Provides GitHub Project Management

Prow allows you to manage GitHub projects by assigning and/or moving GitHub issues/PRs to project columns. The project plugin provides the capability for you to add an issue or PR to a GitHub project board and column.

Prow Provides Config-as-Code for Its Own Configuration

Prow is stateless - storing its configuration in a K8s ConfigMap - but you don’t have to manually create or update the Prow ConfigMap. Prow, of course, provides a way to automate its own configuration by creating the different configuration files in GitHub. Configuring Prow for your own GitHub Organization/Repository is based on a prow/config.yaml file and you can use Prow itself to update it on PR merges.

Prow provides components to manage its own configuration with GitHub - to include the ability to test configuration changes before you apply them. Create a new PR with configuration changes and it will trigger an automated configuration test. Merge the PR to the master branch and Prow will update its K8s ConfigMap to match the configuration in your GitHub repository.

  • updateconfig plugin allows Prow to update K8s ConfigMaps when Prow configuration files in a repository change.
  • checkconfig component loads the Prow configuration given with --config-path, --job-config-path and --plugin-config in order to validate it. Use checkconfig as a pre-submit for any repository holding Prow configuration to ensure that check-ins do not break anything.
  • config-bootstrapper is a component used to bootstrap a configuration that would be incrementally updated by the updateconfig plugin.

Prow is a CI/CD Job Executor

Prow is very modular and triggers jobs of different types, triggered in different ways for different execution engines and provides dashboards to track job status, job artifacts and merge status among other things.

  • Job Types - specifies how the job is triggered.
    • presubmit - runs on unmerged PRs.
    • postsubmit - runs on each new commit.
    • periodic - runs on a time-basis, unrelated to git changes.
    • batch - tests multiple unmerged PRs at the same time.
  • Job States - specifies whether the job is running or not, the status of the job.
    • triggered - means the job has been created but is not yet scheduled.
    • pending - means the job is scheduled but not yet running.
    • success - means the job completed without error (exit 0).
    • failure - means the job completed with errors (exit non-zero).
    • aborted - means that prow killed the job early (new commit pushed, perhaps).
    • error - means the job could not schedule (bad config, perhaps).
  • Execution Engines/Job Agents - specifies the controller (such as plank or jenkins-agent) that runs the job.
    • kubernetes - Prow will create a pod to run this job (default).
    • jenkins - Prow will schedule the job on an external static Jenkins instance.
    • knative-build - Prow will schedule the job via a Knative build-crd resource.
    • tekton (coming soon) - Prow will schedule the job via a Tekton CRD based PipelineRun.

Who is using Prow?

Prow provides a list of other K8s organizations/projects using Prow. But what is very interesting is the other Native K8s CD projects that leverage Prow for their own CI/CD:

What’s next?

So as you can see, and as I already mentioned above, Prow is a lot! And I definitely didn’t discuss all of Prow’s features in this post. But I would also like to share some things I would like to see next for Prow:

Most of all, I want to see you keep your own CD above water by using Prow for your own GitHub automation. But before you do that, you may want to wait for my next post in this series where I will explain how effortless it can be for you to realize the advantages of Prow for your own application delivery by using Jenkins X for your CD.

© 2019 Kurt Madel All Rights Reserved