This is my second post in a series about the open source split-flap display I’ve been designing in my free time. I’ll hopefully write a bit more about the overall design process in the future, but for now wanted to start with some fairly technical posts about build automation on that project.
Posts in the series:
Scripting KiCad Pcbnew exports
Automated KiCad, OpenSCAD rendering using Travis CI
Using UI automation to export KiCad schematics
In my last post, I discussed how I scripted the export of 2d renderings of the custom PCB. In this post, I’ll cover how I hooked up that script and others to run automatically on every commit using Travis CI, with automated deployments to S3 to keep all the renderings in the README updated, like this one:
The best thing about Travis though is that unlike many build systems (like Jenkins or Buildbot), nearly the entire build system configuration for Travis lives directly inside the repo itself (in a .travis.yml file). This has a few major advantages:
Reproducible (or at least reasonably well defined) build environment
Each Travis build starts off as a clean slate, and you’re responsible for defining and installing any extra dependencies on the machine yourself through code. This way you always end up with clearly documented dependencies, and that documentation can never go stale!
Enables different build/test configurations on each branch
One big problem with keeping your code separate from the build configuration (as is often the case with tools like Jenkins/Buildbot) is that the two need to stay in sync. Typically this is not a huge problem for slow, linear development, since occasional lock-step updates across repo and build system aren’t too painful.
The issues start when you have faster development with frequently changing build configurations or parallel development across branches. Now not only do you have to keep your build configuration in sync with changes in the source repo, but you also have to make it branch-aware and keep each branch’s build config in sync with the branches in the source repo! Travis avoids all of this because the .travis.yml file is naturally versioned alongside the source it’s building, and therefore just works in branches with no extra effort!
Build configuration changes can be tested!
Related to the previous point — since the .travis.yml file is checked in and versioned with the source code, changes to the source code that e.g. require new packages to be installed in the build environment can actually be fully tested as part of a feature branch or pull request before landing in `master`.
Both KiCad (schematic/pcb software) and OpenSCAD (3d cad software) are under fairly active development, and their packages in the Ubuntu 14.04 are woefully out of date, so I use snapshot PPAs to install more modern versions of each (this necessitates the use of `sudo: true` above which allows for running `add-apt-repository` under sudo).
Each of the install scripts referenced above is pretty straightforward and looks roughly like this:
The .travis.yml configuration for actually running the PCB export script and OpenSCAD rendering scripts as the main build steps is likewise pretty simple:
The only interesting part of that is the use of `xvfb-run`. Getting OpenSCAD exports working is slightly trickier than KiCad, since even OpenSCAD’s command-line interface requires a graphical environment to render images. The trick to make this work on a headless build machine is to use X virtual framebuffer (Xvfb), which lets you run a standalone X server detached from an actual display. So in the config above, I use the `xvfb-run` utility, which starts an Xvfb server, sets up the DISPLAY environment, runs the specified command, and then shuts everything down when the command completes; easy! (I’ll discuss the actual `generate_2d.py` and `generate_gif.py` script implementations in a future post)
The configuration is again pretty simple. Here’s what it takes to upload the entire “deploy” directory on the build machine to a publicly-readable directory named “latest” in my “splitflap-travis” S3 bucket:
Since the .travis.yml file is checked into the repo and public, putting your actual S3 credentials inside would be silly! But Travis allows you to encrypt your credentials using a secret that only their build machines know, so everything’s nice and secure despite being public.
This lets me reference the latest 2d laser-cut rendering from the README file by referencing https://s3.amazonaws.com/splitflap-travis/latest/3d_laser_raster.png. Here’s what the current rendering looks like by the way:
One thing you may notice is the black bar at the bottom with the date and commit hash. I added that because Github’s image proxy caches extremely aggressively and I originally didn’t include the `cache_control: no-cache` line in my deployment config, so I needed some way to debug. It was pretty easy to add using ImageMagick, and now I can easily tell that the images in my README are showing the latest designs correctly:
(slight adaptation from the full script: annotate_image.sh)
If you do find yourself stuck with cached images on Github, you can manually evict them from the cache using an http PURGE request to the image url:
`$ curl -X PURGE https://camo.githubusercontent.com/xxxxxxxxxxxxx`
If you want to poke around the actual Travis configuration I’ve discussed above, here are some links to the real files:
/travis.yml
/3d/scripts/dependencies.sh
/electronics/scripts/dependencies.sh
/scripts/annotate_image.sh
In my next post I’ll cover how I used `Xvfb` , `xdotool` , and `recordmydesktop` to automatically export the KiCad schematic and bill of materials, which are only exposed through the GUI!
Posts in the series:
Scripting KiCad Pcbnew exports
Automated KiCad, OpenSCAD rendering using Travis CI
Using UI automation to export KiCad schematics
In my last post, I discussed how I scripted the export of 2d renderings of the custom PCB. In this post, I’ll cover how I hooked up that script and others to run automatically on every commit using Travis CI, with automated deployments to S3 to keep all the renderings in the README updated, like this one:
 |
| I'll talk about this particular animated OpenSCAD rendering in a future blog post |
Why Travis?
Travis CI is a continuous build and test system, with Github integration and a matching free tier for open source projects. If you’ve ever seen one of these badges in a Github README, it’s probably using Travis: |
| That's the current build status, hopefully it's green! |
Reproducible (or at least reasonably well defined) build environment
Each Travis build starts off as a clean slate, and you’re responsible for defining and installing any extra dependencies on the machine yourself through code. This way you always end up with clearly documented dependencies, and that documentation can never go stale!
Enables different build/test configurations on each branch
One big problem with keeping your code separate from the build configuration (as is often the case with tools like Jenkins/Buildbot) is that the two need to stay in sync. Typically this is not a huge problem for slow, linear development, since occasional lock-step updates across repo and build system aren’t too painful.
The issues start when you have faster development with frequently changing build configurations or parallel development across branches. Now not only do you have to keep your build configuration in sync with changes in the source repo, but you also have to make it branch-aware and keep each branch’s build config in sync with the branches in the source repo! Travis avoids all of this because the .travis.yml file is naturally versioned alongside the source it’s building, and therefore just works in branches with no extra effort!
Build configuration changes can be tested!
Related to the previous point — since the .travis.yml file is checked in and versioned with the source code, changes to the source code that e.g. require new packages to be installed in the build environment can actually be fully tested as part of a feature branch or pull request before landing in `master`.
Travis with KiCad and OpenSCAD
The first step to automating my build was to install the right packages. The basic .travis.yml config looks like this:
dist: trusty
sudo: true
language: generic
install:
- ./3d/scripts/dependencies.sh
- ./electronics/scripts/dependencies.sh
Both KiCad (schematic/pcb software) and OpenSCAD (3d cad software) are under fairly active development, and their packages in the Ubuntu 14.04 are woefully out of date, so I use snapshot PPAs to install more modern versions of each (this necessitates the use of `sudo: true` above which allows for running `add-apt-repository` under sudo).
Each of the install scripts referenced above is pretty straightforward and looks roughly like this:
#!/bin/bash
set -ev
sudo add-apt-repository --yes ppa:js-reynaud/kicad-4
sudo apt-get update -qq
sudo DEBIAN_FRONTEND=noninteractive apt-get install -y kicad inkscape imagemagick
The .travis.yml configuration for actually running the PCB export script and OpenSCAD rendering scripts as the main build steps is likewise pretty simple:
# [... other stuff above ...]
script:
- (cd electronics && python -u generate_svg.py)
- (cd 3d && xvfb-run --auto-servernum --server-args "-screen 0 1024x768x24" python -u generate_2d.py)
- (cd 3d && xvfb-run --auto-servernum --server-args "-screen 0 1024x768x24" python -u generate_gif.py)
The only interesting part of that is the use of `xvfb-run`. Getting OpenSCAD exports working is slightly trickier than KiCad, since even OpenSCAD’s command-line interface requires a graphical environment to render images. The trick to make this work on a headless build machine is to use X virtual framebuffer (Xvfb), which lets you run a standalone X server detached from an actual display. So in the config above, I use the `xvfb-run` utility, which starts an Xvfb server, sets up the DISPLAY environment, runs the specified command, and then shuts everything down when the command completes; easy! (I’ll discuss the actual `generate_2d.py` and `generate_gif.py` script implementations in a future post)
From Travis to the README
Now that we’ve got Travis set up installing KiCad and OpenSCAD and exporting images from each on every commit, the next step is to actually get those renderings off the build machine and somewhere useful. To do that, I use Travis’s deploy tool to upload those build artifacts to S3.The configuration is again pretty simple. Here’s what it takes to upload the entire “deploy” directory on the build machine to a publicly-readable directory named “latest” in my “splitflap-travis” S3 bucket:
# [... other stuff above ...]
deploy:
provider: s3
access_key_id: AKIAJY6VAINVQICEC47Q
secret_access_key:
secure: SYHsDA3WZfV6YlZ... [truncated for your viewing pleasure]
bucket: splitflap-travis
local-dir: deploy
upload-dir: latest
skip_cleanup: true
acl: public_read
cache_control: no-cache
on:
repo: scottbez1/splitflap
branch: master
Since the .travis.yml file is checked into the repo and public, putting your actual S3 credentials inside would be silly! But Travis allows you to encrypt your credentials using a secret that only their build machines know, so everything’s nice and secure despite being public.
This lets me reference the latest 2d laser-cut rendering from the README file by referencing https://s3.amazonaws.com/splitflap-travis/latest/3d_laser_raster.png. Here’s what the current rendering looks like by the way:
One thing you may notice is the black bar at the bottom with the date and commit hash. I added that because Github’s image proxy caches extremely aggressively and I originally didn’t include the `cache_control: no-cache` line in my deployment config, so I needed some way to debug. It was pretty easy to add using ImageMagick, and now I can easily tell that the images in my README are showing the latest designs correctly:
#!/bin/bash
set -e
LABEL="`date --rfc-3339=seconds`\n`git rev-parse --short HEAD`"
convert -background black -fill white -pointsize 12 label:"$LABEL" -bordercolor black -border 3 input_image.png +swap -append output_image.png
(slight adaptation from the full script: annotate_image.sh)
If you do find yourself stuck with cached images on Github, you can manually evict them from the cache using an http PURGE request to the image url:
`$ curl -X PURGE https://camo.githubusercontent.com/xxxxxxxxxxxxx`
If you want to poke around the actual Travis configuration I’ve discussed above, here are some links to the real files:
/travis.yml
/3d/scripts/dependencies.sh
/electronics/scripts/dependencies.sh
/scripts/annotate_image.sh
In my next post I’ll cover how I used `Xvfb` , `xdotool` , and `recordmydesktop` to automatically export the KiCad schematic and bill of materials, which are only exposed through the GUI!
Comments
Post a Comment