If you are doing CI and/or CD at scale and you aren’t leveraging Native Kubernetes Continuous Delivery (Native K8s CD) then you are just
When it came to CI/CD and the creation of Prow, Kubernetes was scratching its own itch with a need to execute over 10,000 CI/CD jobs per day across over 100 repositories of code in several GitHub Organizations - and other tools just weren’t getting the job done. So, the Kubernetes Testing Special Interest Group (sig-testing) created their own tools and services, to include Prow. Prow is currently part of the Kubernetes test-infra project and is one of several Kubernetes Testing Special Interest Group (sig-testing) top-level projects. At the time of this post, Prow was not even in its own GitHub repository, rather it was just a sub-directory (and previously named ciongke)in the kubernetes/test-infra repository. And partly because of the importance of Prow to the K8s project as a whole and also because there is so much interest from other K8s related projects, Prow continues to add features and flourish. Another testament to Prows continued growth is how top-level test-infra components continue to be replaced with Prow components - with one of the latest examples of this being the replacement of Gubernator with Spyglass.
Prow is a lot - certainly more than I can cover in one post - but I will do my best. Prow’s GitHub README describes it as a system, and a very complex multi-faceted system it is at that. Prow is built on a container based microservice architecture and consists of a number of single purpose components - many integrating with one another while others are more or less standalone. The Kubernetes Testing SIG describes Prow as “a CI/CD system built on Kubernetes for Kubernetes that executes jobs for building, testing, publishing and deploying.” However, that description does not highlight perhaps the most important inferred capability of Prow - a capability that is at the heart of best-of-breed CI/CD automation tools - that capability is automation that starts with code commits - and in the case of Prow it starts with a scalable stateless microservice called
hookthat triggers native K8s CI/CD jobs (among a number of things that
hook does via plugins). It is this GitHub automation capability that has been one of the key reasons why other K8s projects have adopted Prow for their own CI/CD. But Prow is more than just GitHub webhook automation and CI/CD job execution. Prow is also:
- Comprehensive GitHub Automation
- ChatOps via simple
- Fine-grained GitHub policy and permission management via
- GitHub PR/merge automation -
- GitHub API request cache to minimize impact of GitHub API limits -
- GitHub Organization and repository membership and permissions management -
- GitHub labels management -
- GitHub branch protection configuration -
- GitHub release notes management -
- Scalable, cacheable
- GitHub bot with Prow’s bot being an active GitHub user since 2016
- ChatOps via simple
- Multi-engine CI/CD Job Execution -
- CI/CD Reporting -
- CI/CD Dashboards for viewing job history, merge status and more -
- Pluggable Artifact Viewer -
- Promethus Metrics for monitoring and alerting -
- Config-as-Code for Its Own Configuration -
- And I am sure I am still missing a bunch of things - like
Prow is like steroids for GitHub automation.
This may be one of the most important capabilities of Prow and most of Prow’s other components depend on it in one way or another. Prow listens for GitHub webhooks via the
hook component that in turn triggers one or more configured plugins to include the
trigger plugin that triggers Native K8s CI/CD jobs for GitHub PR events, and the
/retest ChatOps commands.
Many non-Native K8s CI/CD platforms support GitHub webhooks but in many cases they are not highly fault-tolerant, not horizontally scalable and not stateless - so they become a single-point-of-failure (SPoF) for GitHub webhooks. If a GitHub webhook is sent while such a platform is down it will be completely missed - so you might miss a merge to your deployment branch and miss a mission critical change that needs to be deployed to production.
ChatOps are an important component of Prow’s GitHub webhook support - Prow doesn’t just listen for pushes to branches, but listens for a number of more sophisticated GitHub events, like comments on issues and PRs using ChatOps commands. Prow ChatOps allows you to do everything from approving a PR with the
/approve command to telling a
/joke. Many of the Prow
hook plugins provide ChatOps commands that streamline and automate repository contributors’ interactions with GitHub PRs and issues - so developers can focus more on the code and not GitHub process and UI. The complete list of supported ChatOps commands is available as one of the views of the
Prow allows you to manage ownership/permissions at a more granular level than GitHub does. Prow allows you to create an
OWNERS file at the repository level, but also within any repository sub-directory. This allows you to manage who can do what at the directory level instead of just the top-level repository level. The directory specific
OWNERS files are utilized by a number of
hook plugins and ChatOps commands - for example the
blunderbluss plugin automatically assigns PR reviewers based on
OWNERS files most relevant to the file(s) being committed - but the automated reviewer selection is more sophisticated than just using the most relevant/closest
OWNERS file - it also weights reviewers based on the number of lines of code that changed and in what files - once again highlighting the incredible breadth and depth of Prow’s GitHub automation.
tide component automates the bulk testing of PRs and automatically merges multiple PRs. It replaced another
test-infra top-level project called
mungegithub. In addition to automated testing and merging of PRs - at scale, tide also provides Promethus metrics and serves live data to Deck - the Prow reporting dashboard.
Prow allows you to manage GitHub Organization settings and team membership with Config-as-Code (CasC) so the source of truth for your Organization configuration is in one easily auditable file, rather than relying on one or more people making manual changes via the GitHub UI or manual API calls. The
peribolos component provides automated auditable support for managing these GitHub settings with a CasC
Prow allows you to manage GitHub projects by assigning and/or moving GitHub issues/PRs to project columns. The
project plugin provides the capability for you to add an issue or PR to a GitHub project board and column.
Prow is stateless - storing its configuration in a K8s
ConfigMap - but you don’t have to manually create or update the Prow
ConfigMap. Prow, of course, provides a way to automate its own configuration by creating the different configuration files in GitHub.
Configuring Prow for your own GitHub Organization/Repository is based on a
prow/config.yaml file and you can use Prow itself to update it on PR merges.
Prow provides components to manage its own configuration with GitHub - to include the ability to test configuration changes before you apply them. Create a new PR with configuration changes and it will trigger an automated configuration test. Merge the PR to the
master branch and Prow will update its K8s
ConfigMap to match the configuration in your GitHub repository.
updateconfigplugin allows Prow to update K8s
ConfigMapswhen Prow configuration files in a repository change.
checkconfigcomponent loads the Prow configuration given with
--plugin-configin order to validate it. Use
checkconfigas a pre-submit for any repository holding Prow configuration to ensure that check-ins do not break anything.
config-bootstrapperis a component used to bootstrap a configuration that would be incrementally updated by the
Prow is very modular and triggers jobs of different types, triggered in different ways for different execution engines and provides dashboards to track job status, job artifacts and merge status among other things.
- Job Types - specifies how the job is triggered.
- presubmit - runs on unmerged PRs.
- postsubmit - runs on each new commit.
- periodic - runs on a time-basis, unrelated to git changes.
- batch - tests multiple unmerged PRs at the same time.
- Job States - specifies whether the job is running or not, the status of the job.
- triggered - means the job has been created but is not yet scheduled.
- pending - means the job is scheduled but not yet running.
- success - means the job completed without error (exit 0).
- failure - means the job completed with errors (exit non-zero).
- aborted - means that prow killed the job early (new commit pushed, perhaps).
- error - means the job could not schedule (bad config, perhaps).
- Execution Engines/Job Agents - specifies the controller (such as plank or jenkins-agent) that runs the job.
- kubernetes - Prow will create a pod to run this job (default).
- jenkins - Prow will schedule the job on an external static Jenkins instance.
- knative-build - Prow will schedule the job via a Knative build-crd resource.
- tekton (coming soon) - Prow will schedule the job via a Tekton CRD based PipelineRun.
- Reporting and Dashboards
- Deck provides:
- Crier detects changes to ProwJob CRDs and notifies configured clients to include:
- GitHub status context
- Gerrit comment
- Pubsub message
- Artifact viewer with Spyglass
Prow provides a list of other K8s organizations/projects using Prow. But what is very interesting is the other Native K8s CD projects that leverage Prow for their own CI/CD:
- Knative - not only provides one of the CI/CD job execution engines for Prow, it also uses Prow for its own CI.
- Tekton - currently using Prow to build and test Tekton Pipeline.
- Jenkins X - uses Prow for its own CI/CD. But it also takes Prow usage to a whole other level by seamlessly integrating Prow with the original Serverless Jenkins X and now for Jenkins X with Tekton - more about this in the next post.
- Istio - the service mesh used by Knative to support continuous deployment (among others of course), integrated Prow early on its existence.
So as you can see, and as I already mentioned above, Prow is a lot! And I definitely didn’t discuss all of Prow’s features in this post. But I would also like to share some things I would like to see next for Prow:
- I would like to see Prow become a top-level project/repository under kubernetes-sig GitHub Organization. See https://github.com/kubernetes/test-infra/issues/11782 for some details.
- I would like to see support for other Git providers - BitBucket, GitHub Enterprise and even generic git support. See:
- I would like to see Prow support Tekton as a Prow agent type/execution engine. See:
- Prow originated as a Google engineering project as displayed by Prow’s GCP slant. I would like to see better support for AWS and Azure.
- I would like to see Prow become a Continuous Delivery Foundation project.
Most of all, I want to see you keep your own CD above water by using Prow for your own GitHub automation. But before you do that, you may want to wait for my next post in this series where I will explain how effortless it can be for you to realize the advantages of Prow for your own application delivery by using Jenkins X for your CD.