Last time I covered several tips on how to launch and operate a Drone CI installation. As promised, I will now reveal my hard-earned secrets on how to build, configure, and monitor DroneCI pipelines!
This assumes pipelines using the Docker runner which is the common use case (and the most useful!)
Each pipeline step can use entirely different container images
Yes, this is implicit with the use of the Docker runner, however, take a moment and consider the possibilities:
- Test execution of your code across multiple Linux distributions, versions, and language runtimes
- Use specialized container images for each step (lint, build, etc)
Use base64 encoding for complex secrets
Imagine that your pipeline requires an SSH key to do something (clone a 3rd party repo, run a command on another server, etc).
You could try to store the key in a Drone secret, but the process of accessing that environment variable from a pipeline step and writing it to a file can be cumbersome. Ideally, environment variables are just a string of characters with no formatting.
This is where base64 comes to the rescue!
base64 -w0 FILE
Use the output of the above command when creating the secret instead of using plaintext.
Then, from within .drone.yml, add a command like this:
- echo "$ENV_VAR" | base64 -d > FILE
(Note: This tip is also useful for other tools like CircleCI)
Properly create .ssh directory and files
Speaking of SSH, make sure you use these commands to enable seamless use of git clone.
- mkdir $HOME/.ssh
- echo "$SSH_KEY" | base64 -d > $HOME/.ssh/id_rsa
- ssh-keyscan -H github.com >> $HOME/.ssh/known_hosts
- chmod 600 $HOME/.ssh/known_hosts $HOME/.ssh/id_rsa
Note that ssh-keyscan can be used against any ssh-based git host (gitlab, bitbucket, etc).
Use ‘timeout’ command to prevent long-running steps
The timeout tool from coreutils allows you to set a time limit on specific steps:
timeout 600 /path/to/long-running-command
Monitor Pipeline Status and Duration Using Prometheus
Once you have your pipeline set up, it’s perfectly natural to monitor it, namely:
- Status (did it run successfully?)
- Duration (is the pipeline getting slower over time?)
When researching how to do this, I learned that Drone already had a Prometheus endpoint available for scraping! However, it only contains statistics on the Drone service.
After some further Googling, I found an article from Grafana Labs. The key info is from this passage:
“By checking environmental variables such as DRONE_BUILD_STATUS, DRONE_STAGE_STATUS, DRONE_STEP_NAME we can get really valuable information about the build status. Those environmental variables need to be exported and stored so we can query them and create meaningful dashboards and alerts for our CI builds. We have configured the Drone server to send information as webhooks to a service which listens for these webhooks. Once the listener gets the information from the webhooks, it exports metrics using a custom Prometheus exporter we’ve created and stores them in Grafana Mimir.”
That revealed that Drone provides a set of special environment variables that are available from within the pipeline. For my purposes, I care about:
- DRONE_BUILD_STATUS
- DRONE_BUILD_CREATED
- DRONE_BUILD_FINISHED
Do I need to build a special exporter and persist these metrics in Mimir, however?
This is actually one of the few examples where Prometheus Push Gateway is an appropriate tool since we are precisely ‘capturing the outcome of a service level batch job’.
Here’s a pipeline step that will send build status and duration to a Push Gateway:
- name: send-build-metrics
image: python
commands:
- |
URL="http://gateway.domain.tld:9091/metrics/job/drone-pipeline/instance/test"
if [ $DRONE_BUILD_STATUS = "success" ]
then
echo "status 1" | curl -s --data-binary @- $URL
else
echo "status 0" | curl -s --data-binary @- $URL
fi
echo "duration $(expr $DRONE_BUILD_FINISHED - $DRONE_BUILD_CREATED)" | curl -s --data-binary @- $URL
depends_on:
- previous-step
when:
branch:
- master
status:
- success
- failure
Note:
- Ensure this step runs as the last step in the pipeline by using
depends_on. - Be deliberate with your choice of job and instance names in the URL.
This technique can be tailored for use in any CI/CD system so long as you are able to programmatically determine the status and duration of a pipeline.
Conclusion
Drone is a powerful CI/CD system out of the box, but the above techniques help you perform complex tasks such as SSH, git clone, terminate long-running processes, and more closely monitor the performance of a pipeline.
Reach out if your team needs help jump-starting their CI/CD journey!
(Image produced by OpenAI DALL·E 2 with prompt “a photo of a robot plumber holding a wrench posing for the camera”.)