🚧 Warning, this chapter is being updated, and still mostly the 2e version;
Although it’s not got many code examples in, you may find it’s a little out of date.
As our site grows, it takes longer and longer to run all of our functional tests. If this continues, the danger is that we’re going to stop bothering.
Rather than let that happen, we can automate the running of functional tests by setting up a "Continuous Integration" or CI server. That way, in day-to-day development, we can just run the FT that we’re working on at that time, and rely on the CI server to run all the other tests automatically and let us know if we’ve broken anything accidentally. The unit tests should stay fast enough that we can keep running them every few seconds.
The CI server of choice these days is called Jenkins. It’s a bit Java, a bit crashy, a bit ugly, but it’s what everyone uses, and it has a great plugin ecosystem, so let’s get it up and running.
-
TODO: update to mention github actions, gitlab etc.
There are several hosted-CI services out there that essentially provide you with a Jenkins server, ready to go. I’ve come across Sauce Labs, Travis, Circle-CI, ShiningPanda, and there are probably lots more. But I’m going to assume we’re installing everything on a server we control.
|
Note
|
It’s not a good idea to install Jenkins on the same server as our staging or production servers. Apart from anything else, we may want Jenkins to be able to reboot the staging server! |
We’ll install the latest version from the official Jenkins apt repo, because the Ubuntu default still has a few annoying bugs with locale/unicode support, and it also doesn’t set itself up to listen on the public internet by default:
# add jenkins repo to our apt sources root@server:$ wget -O /usr/share/keyrings/jenkins-keyring.asc \ https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key root@server:$ echo "deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc]" \ https://pkg.jenkins.io/debian-stable binary/ > \ /etc/apt/sources.list.d/jenkins.list # update and install jenkins root@server:$ apt-get update root@server:$ apt-get install jenkins
(Instructions lifted from the Jenkins site.)
While we’re at it, we’ll install a few other dependencies:
# remove the default Firefox installed via snap, # add more reliable "extended support release" version: root@server:$ snap remove firefox root@server:$ add-apt-repository ppa:mozillateam/ppa root@server:$ apt update root@server:$ apt install firefox-esr # install the venv modules for the otherwise-broken default system python: root@server:$ apt install python3-venv
Jenkins is quite memory-hungry, and if you’re running this on a small VM with less than a couple of gigs for RAM, you’ll probably find it gets OOM-killed unless you add some swap:
root@server:$ fallocate -l 4G /swapfile root@server:$ mkswap /swapfile root@server:$ chmod 600 /swapfile root@server:$ swapon /swapfile root@server:$ echo "LABEL=swap /swap.img swap defaults 0 0" >> /etc/fstab
That should be plenty.
You should now be able to visit Jenkins at the URL/IP for your server on port 8080,
and see something like Jenkins unlock screen.
The unlock screen is telling us to read a file from disk to unlock the server for first-time use. I jumped over to a terminal and printed it like this:
root@server$ cat /var/lib/jenkins/secrets/initialAdminPassword
Next we’re offered the choice to choose "suggested" plugins. Suggested ones are fine for now. (As a self-respecting nerd, our instinct is to hit "customize" immediately, but I don’t think it really bought me anything.)
Next we set up a username and password to log in to Jenkins with; see Jenkins admin user config.
And once we log in, we should see a welcome screen (A butler—how quaint).
-
TODO: review and update this section
To finish off securing your Jenkins instance, you’ll want to set up HTTPS, by getting nginx HTTPS to use a self-signed cert, and proxy requests from port 443 to port 8080. Then you can even block port 8080 on the firewall. I won’t go into detail on that now, but here are a few links to instructions which I found useful:
Now we’ve got the basic Jenkins configured, let’s set up our project:
-
Hit the New Item button.
-
Enter 'Superlists' as the name, and then choose "Freestyle project", and hit OK.
-
Add the Git repo, as in Get it from Git.
-
Set it to poll every hour (Poll GitHub for changes; check out the help text here—there are many other options for ways of triggering builds).
-
Run the tests inside a Python 3 virtualenv.
-
Run the unit tests and functional tests separately. See Virtualenv build steps.
Hit "Build Now", then go and take a look at the "Console Output". You should see something like this:
Started by user harry Building in workspace /var/lib/jenkins/jobs/Superlists/workspace Fetching changes from the remote Git repository Fetching upstream changes from https://github.com/hjwp/book-example.git Checking out Revision d515acebf7e173f165ce713b30295a4a6ee17c07 (origin/master) [workspace] $ /bin/sh -xe /tmp/shiningpanda7260707941304155464.sh + pip install -r requirements.txt Requirement already satisfied (use --upgrade to upgrade): Django==1.11 in /var/lib/jenkins/shiningpanda/jobs/ddc1aed1/virtualenvs/d41d8cd9/lib/python3.7/site-packages (from -r requirements.txt (line 1)) Requirement already satisfied (use --upgrade to upgrade): gunicorn==17.5 in /var/lib/jenkins/shiningpanda/jobs/ddc1aed1/virtualenvs/d41d8cd9/lib/python3.7/site-packages (from -r requirements.txt (line 3)) Downloading/unpacking requests==2.0.0 (from -r requirements.txt (line 4)) Running setup.py egg_info for package requests Installing collected packages: requests Running setup.py install for requests Successfully installed requests Cleaning up... + python manage.py test lists accounts ................................................................... --------------------------------------------------------------------- Ran 67 tests in 0.429s OK Creating test database for alias 'default'... Destroying test database for alias 'default'... + python manage.py test functional_tests EEEEEE ====================================================================== ERROR: functional_tests.test_layout_and_styling (unittest.loader._FailedTest) --------------------------------------------------------------------- ImportError: Failed to import test module: functional_tests.test_layout_and_styling [...] ImportError: No module named 'selenium' Ran 6 tests in 0.001s FAILED (errors=6) Build step 'Virtualenv Builder' marked build as failure
Ah. We need Selenium in our virtualenv.
Let’s add a manual installation of Selenium to our build steps:
pip install -r requirements.txt
python manage.py test accounts lists
pip install selenium fabric3
python manage.py test functional_tests
|
Tip
|
Some people like to use a file called 'test-requirements.txt' to specify packages that are needed for the tests, but not the main app. |
And hit "Build Now" again.
Next one of two things will happen. Either you’ll see some error messages like this in your console output:
self.browser = webdriver.Firefox() [...] selenium.common.exceptions.WebDriverException: Message: 'The browser appears to have exited before we could connect. The output was: b"\\n(process:19757): GLib-CRITICAL **: g_slice_set_config: assertion \'sys_page_size == 0\' failed\\nError: no display specified\\n"' [...] selenium.common.exceptions.WebDriverException: Message: connection refused
Or possibly your build will just hang altogether (that happened to me at least once). The reason is that Firefox can’t start, because it doesn’t have a display to run on.
As you can see from the traceback, Firefox is unable to start because the server doesn’t have a display.
There are two ways to deal with this problem. The first is to switch to using a headless browser, like PhantomJS or SlimerJS. Those tools definitely have their place—they’re faster, for one thing—but they also have disadvantages. The first is that they’re not "real" web browsers, so you can’t be sure you’re going to catch all the strange quirks and behaviours of the actual browsers your users use. The second is that they can behave quite differently inside Selenium, and often require some rewriting of FT code.
|
Tip
|
I would look into using headless browsers as a "dev-only" tool, to speed up the running of FTs on the developer’s machine, while the tests on the CI server use actual browsers. |
The alternative is to set up a virtual display: we get the server to pretend it has a screen attached to it, so Firefox runs happily. There are a few tools out there to do this; we’ll use one called "Xvfb" (X Virtual Framebuffer)[1] because it’s easy to install and use, and because it has a convenient Jenkins plugin (now you know why we installed it earlier).
We go back to our project and hit "Configure" again, then find the section called "Build Environment". Using the virtual display is as simple as ticking the box marked "Start Xvfb before the build, and shut it down after", as in Sometimes config is easy.
The build does much better now:
[...] Xvfb starting$ /usr/bin/Xvfb :2 -screen 0 1024x768x24 -fbdir /var/lib/jenkins/2013-11-04_03-27-221510012427739470928xvfb [...] + python manage.py test lists accounts ............................................................... --------------------------------------------------------------------- Ran 63 tests in 0.410s OK Creating test database for alias 'default'... Destroying test database for alias 'default'... + pip install selenium Requirement already satisfied (use --upgrade to upgrade): selenium in /var/lib/jenkins/shiningpanda/jobs/ddc1aed1/virtualenvs/d41d8cd9/lib/python3.7/site-packages Cleaning up...
+ python manage.py test functional_tests
......F.
======================================================================
FAIL: test_can_start_a_list_for_one_user
(functional_tests.test_simple_list_creation.NewVisitorTest)
---------------------------------------------------------------------
Traceback (most recent call last):
File "...goat-book/functional_tests/test_simple_list_creation.py", line
43, in test_can_start_a_list_for_one_user
self.wait_for_row_in_list_table('2: Use peacock feathers to make a fly')
File "...goat-book/functional_tests/base.py", line 51, in
wait_for_row_in_list_table
raise e
File "...goat-book/functional_tests/base.py", line 47, in
wait_for_row_in_list_table
self.assertIn(row_text, [row.text for row in rows])
AssertionError: '2: Use peacock feathers to make a fly' not found in ['1: Buy
peacock feathers']
---------------------------------------------------------------------
Ran 8 tests in 89.275s
FAILED (errors=1)
Creating test database for alias 'default'...
[{'secure': False, 'domain': 'localhost', 'name': 'sessionid', 'expiry':
1920011311, 'path': '/', 'value': 'a8d8bbde33nreq6gihw8a7r1cc8bf02k'}]
Destroying test database for alias 'default'...
Build step 'Virtualenv Builder' marked build as failure
Xvfb stopping
Finished: FAILURE
Pretty close! To debug that failure, we’ll need screenshots though.
|
Note
|
This error was due to the performance of my Jenkins instance—you may see a different error, or none at all. In any case, the following tools for taking screenshots and dealing with race conditions will come in useful. Read on! |
To
be able to debug unexpected failures that happen on a remote PC, it
would be good to see a picture of the screen at the moment of the failure,
and maybe also a dump of the HTML of the page. We can do that using some
custom logic in our FT class tearDown. We have to do a bit of introspection of
unittest internals, a private attribute called ._outcome, but
this will work:
import os
import time
from datetime import datetime
from pathlib import Path
[...]
MAX_WAIT = 5
SCREEN_DUMP_LOCATION = Path(__file__).absolute().parent / "screendumps"
[...]
def tearDown(self):
if self._test_has_failed():
if not SCREEN_DUMP_LOCATION.exists():
SCREEN_DUMP_LOCATION.mkdir(parents=True)
self.take_screenshot()
self.dump_html()
self.browser.quit()
super().tearDown()
def _test_has_failed(self):
# slightly obscure but couldn't find a better way!
return self._outcome.result.failures or self._outcome.result.errorsWe first create a directory for our screenshots if necessary. Then we
iterate through all the open browser tabs and pages, and use some Selenium
methods, get_screenshot_as_file and browser.page_source, for our image and
HTML dumps:
def take_screenshot(self):
path = SCREEN_DUMP_LOCATION / self._get_filename("png")
print("screenshotting to", path)
self.browser.get_screenshot_as_file(str(path))
def dump_html(self):
path = SCREEN_DUMP_LOCATION / self._get_filename("html")
print("dumping page HTML to", path)
path.write_text(self.browser.page_source)And finally here’s a way of generating a unique filename identifier, which includes the name of the test and its class, as well as a timestamp:
def _get_filename(self, extension):
timestamp = datetime.now().isoformat().replace(":", ".")[:19]
return (
f"{self.__class__.__name__}.{self._testMethodName}-{timestamp}.{extension}"
)You can test this first locally by deliberately breaking one of the tests, with
a self.fail() for example, and you’ll see something like this:
[...] .Fscreenshotting to ...goat-book/src/functional_tests/screendumps/MyListsTest.t est_logged_in_users_lists_are_saved_as_my_lists-[...] dumping page HTML to ...goat-book/src/functional_tests/screendumps/MyListsTest. test_logged_in_users_lists_are_saved_as_my_lists-[...]
Revert the self.fail(), then commit and push:
$ git diff # changes in base.py $ echo "src/functional_tests/screendumps" >> .gitignore $ git commit -am "add screenshot on failure to FT runner" $ git push
And when we rerun the build on Jenkins, we see something like this:
screenshotting to /var/lib/jenkins/jobs/Superlists/.../functional_tests/ screendumps/LoginTest.test_login_with_persona-window0-2014-01-22T17.45.12.png dumping page HTML to /var/lib/jenkins/jobs/Superlists/.../functional_tests/ screendumps/LoginTest.test_login_with_persona-window0-2014-01-22T17.45.12.html
We can go and visit these in the "workspace", which is the folder Jenkins uses to store our source code and run the tests in, as in Visiting the project workspace.
And then we look at the screenshot, as shown in Screenshot looking normal.
Hm. No obvious clues there. Well, when in doubt, bump the timeout, as the old adage goes:
MAX_WAIT = 20Then we can rerun the build on Jenkins using "Build Now", and confirm it now works, as in The outlook is brighter.
Jenkins uses blue to indicate passing builds rather than green, which is a bit disappointing, but look at the sun peeking through the clouds: that’s cheery! It’s an indicator of a moving average ratio of passing builds to failing builds. Things are looking up!
There’s a set of tests we almost forgot—the JavaScript tests. Currently our "test runner" is an actual web browser. To get Jenkins to run them, we need a command-line test runner. Here’s a chance to use PhantomJS.
It’s time to stop pretending we’re not in the JavaScript game. We’re doing web development. That means we do JavaScript. That means we’re going to end up with node.js on our computers. It’s just the way it has to be.
Follow the instructions on the node.js homepage. There are installers for Windows and Mac, and repositories for popular Linux distros.[2]
Once we have node, we can install phantom:
$ npm install -g phantomjs-prebuilt # the -g means "system-wide".
Next we pull down a QUnit/PhantomJS test runner. There are several out there (I even wrote a basic one to be able to test the QUnit listings in this book), but the best one to get is probably the one that’s linked from the QUnit plugins page. At the time of writing, its repo was at https://github.com/jonkemp/qunit-phantomjs-runner. The only file you need is 'runner.js'.
You should end up with this:
$ tree src/lists/static/tests/ src/lists/static/tests/ ├── qunit-2.0.1.css ├── qunit-2.0.1.js ├── runner.js └── tests.html 0 directories, 4 files
Let’s try it out:
$ phantomjs src/lists/static/tests/runner.js src/lists/static/tests/tests.html Took 24ms to run 2 tests. 2 passed, 0 failed.
Just to be sure, let’s deliberately break something:
it("sense-check our html fixture", () => {
expect(errorMsg.checkVisibility()).toBe(false);
});Sure enough:
$ phantomjs lists/static/tests/runner.js lists/static/tests/tests.html
Test failed: errors should be hidden on keypress
Failed assertion: expected: false, but was: true
file://...goat-book/lists/static/tests/tests.html:27:15
Took 27ms to run 2 tests. 1 passed, 1 failed.
All right! Let’s unbreak that, commit and push the runner, and then add it to our Jenkins build:
$ git checkout lists/static/list.js $ git add lists/static/tests/runner.js $ git commit -m "Add phantomjs test runner for javascript tests" $ git push
Edit the project configuration again, and add a step for each set of JavaScript tests, as per Add a build step for our JavaScript unit tests.
You’ll also need to install PhantomJS on the server:
root@server:$ add-apt-repository -y ppa:chris-lea/node.js root@server:$ apt update root@server:$ apt install nodejs root@server:$ npm install -g phantomjs-prebuilt
And there we are! A complete CI build featuring all of our tests!
Started by user harry Building in workspace /var/lib/jenkins/jobs/Superlists/workspace Fetching changes from the remote Git repository Fetching upstream changes from https://github.com/hjwp/book-example.git Checking out Revision 936a484038194b289312ff62f10d24e6a054fb29 (origin/chapter_1 Xvfb starting$ /usr/bin/Xvfb :1 -screen 0 1024x768x24 -fbdir /var/lib/jenkins/20 [workspace] $ /bin/sh -xe /tmp/shiningpanda7092102504259037999.sh + pip install -r requirements.txt [...] + python manage.py test lists ................................. --------------------------------------------------------------------- Ran 43 tests in 0.229s OK Creating test database for alias 'default'... Destroying test database for alias 'default'... + python manage.py test accounts .................. --------------------------------------------------------------------- Ran 18 tests in 0.078s OK Creating test database for alias 'default'... Destroying test database for alias 'default'... [workspace] $ /bin/sh -xe /tmp/hudson2967478575201471277.sh + phantomjs lists/static/tests/runner.js lists/static/tests/tests.html Took 32ms to run 2 tests. 2 passed, 0 failed. + phantomjs lists/static/tests/runner.js accounts/static/tests/tests.html Took 47ms to run 11 tests. 11 passed, 0 failed. [workspace] $ /bin/sh -xe /tmp/shiningpanda7526089957247195819.sh + pip install selenium Requirement already satisfied (use --upgrade to upgrade): selenium in /var/lib/ Cleaning up... [workspace] $ /bin/sh -xe /tmp/shiningpanda2420240268202055029.sh + python manage.py test functional_tests ........ --------------------------------------------------------------------- Ran 8 tests in 76.804s OK
Nice to know that, no matter how lazy I get about running the full test suite on my own machine, the CI server will catch me. Another one of the Testing Goat’s agents in cyberspace, watching over us…
I’ve only scratched the surface of what you can do with Jenkins and CI servers. For example, you can make it much smarter about how it monitors your repo for new commits.[3]
Perhaps more interestingly, you can use your CI server to automate your staging tests as well as your normal functional tests. If all the FTs pass, you can add a build step that deploys the code to staging, and then reruns the FTs against that—automating one more step of the process, and ensuring that your staging server is automatically kept up to date with the latest code.
Some people even use a CI server as the way of deploying their production releases!
- Set up CI as soon as possible for your project
-
As soon as your functional tests take more than a few seconds to run, you’ll find yourself avoiding running them all. Give this job to a CI server, to make sure that all your tests are getting run somewhere.
- Set up screenshots and HTML dumps for failures
-
Debugging test failures is easier if you can see what the page looked like when the failure occurred. This is particularly useful for debugging CI failures, but it’s also very useful for tests that you run locally.
- Be prepared to bump your timeouts
-
A CI server may not be as speedy as your laptop, especially if it’s under load, running multiple tests at the same time. Be prepared to be even more generous with your timeouts, in order to minimise the chance of random failures.
- Look into hooking up CI and staging
-
Tests that use
LiveServerTestCaseare all very well for dev boxes, but the true reassurance comes from running your tests against a real server. Look into getting your CI server to deploy to your staging server, and run the functional tests against that instead. It has the side benefit of testing your automated deploy scripts.