This service is based on the echo template. Please view the README for details about the dev loop and how it works.
We use the GitHub-Runner-Provisioner to serve a webhook to GitHub Actions. GitHub will send any Actions events to the GRP running in Skunkworks, which will parse those events looking for workflows that request special labels in their runs-on
property.
Using the GitHub Self-Hosted Runner binaries we then spin up the custom runners in one of our supported runner providers - currently AWS and CodeMagic. Supported runners are configured in runner.go.
AWS runners are created in EC2 using the AWS SDK. See the aws_runners package for details on the implementation.
CodeMagic runners are actually CodeMagic Builds (CI jobs in their service) that then pull the GitHub Self-Hosted binaries and register themselves as ephemeral (single-use) runners - picking up a single job from the calling repo and then terminating.
Note: Before running tests, make sure you run the application with environment variable WEBHOOK_TOKEN=FAKE_TOKEN
.
You will also need to set GITHUB_TOKEN
to a PAT for the D6E Automaton. These values can all be found in the github-runner-provisioner-secrets.yaml file in Keybase - you will need to base64 decode them before use. If only running dry-runs only AWS and GitHub authentication is required.
To test the application we use targets in the Makefile. The make go-unit-tests
target will run the unit tests, and make test-runners
will run the integration tests against the dry-run endpoints. Note that to test the AWS macOS-arm64
runner you will need to set the USE_CODEMAGIC
environment variable to true
in the GRP.
Testing CodeMagic M1 & AWS ubuntu-arm64:
USE_CODEMAGIC=true GITHUB_TOKEN=<pat> go run main.go --dry-run
make test-runners
Note: You can send requests to the production client using make run-<runner tag>
Be careful when sending requests to production using an HTTP client, since the dry-run
request parameter defaults to true. This is necessary because we have no way to set GitHub to send this parameter.
Some unit tests use mocks generated by gomock. If the interface being mocked is updated, you may have to re-generate the mocks by running:
make update-go-mocks
The runner provisioner requires the following variables to be configured:
GITHUB_TOKEN
- a personal access token with admin access to the repo configuring the runners. We use theD6E-Automaton
's token in production.WEBHOOK_TOKEN
- the secret used to configure the webhook in GitHub. We use the token stored at/Keybase/team/datawireio/infra/github-runner-provisioner-secrets
CODEMAGIC_TOKEN
- the secret used to authenticate to the CodeMagic build API to trigger M1 runnersUSE_CODEMAGIC
- a boolean flag to indicate whether to use CodeMagic or AWS to provision M1 runners- AWS auth can be configured with
AWS_ACCESS_KEY_ID
andAWS_SECRET_ACCESS_KEY
or by using the AWS CLI