Get Started with Visual Regression Testing with BackstopJS

If you’re more into video tutorials:

Let’s say you work on an older, but very much alive, site. You occasionally get tasked with adding features here or enhancing sections there. These are typical tasks from clients. Now in a perfect world, the HTML and CSS would be written in a perfectly extendable way, but in reality, there’s often over-specified CSS targeting imperfect HTML structure. You may be charged with updating all the <h1>s on a page (yes, you read that correctly) or realizing that Lighthouse audits yield some improvements in the way the HTML is structured.

Let’s take a scenario that I’ve seen a million times.

You have a page that has a header with an <h1> for the page title at the top. Further down the page, you get the latest 10 posts each with their own titles. The titles are in <h3> tags. Then further down the page, you have a call to action that has a <h2> heading. Therein lies the problem, your document doesn’t follow a logical flow. Google Lighthouse audits will ping you on this. It can be better.

The solution is to change the markup. So you go in and change the section of the latest 10 blog posts’ titles to <p> tags and restyle them. So far so good. But have you bumped anything else this seemingly simple process? Time to go check every page and ensure everything is still the way it should be. You have to check the About Us page in its entirety to ensure no other headings were bumped, then the Contact Us page, then the store page, on and on. What if you have thousands of blog posts or event dozens of blog templates. Each may or may not contain a similarly styled <h3> or <p> that you adjusted on the home page.

Regression Testing Definition

Checking the other pages is the definition of regression testing. You are visually checking to see if anything inadvertently changed. You are checking to see if something regressed. Checking a few pages is no big deal, but even that can get tedious and it is just as susceptible to human error. Let’s let the computers do it.

Automatic Visual Regression Testing with BackstopJS

BackstopJS is a tool that you can use to test your pages that you work on against a know good page. Let me clarify. Basically, you have a website and you know that the production version is the version that is stable. Everything in production is perfect. So, you install BackstopJS and take screenshots of pages on that perfect website. Then you can perform new work on a  page of the new site, for instance, you want to add a new section with a button to the /about page. You add the section on your local machine and test and refine. Then when you’re almost ready to send it off to QA you run your BackstopJS tests and check that your work on /about didn’t affect anything on /store or /contact, or /blog/a-cool-post. BackstopJS will tell you if there are differences so you can address them in the local environment first before clients or QA even see them.

The power of automating the visual regression tests is that you speed up the tests dramatically. Once you load up the initial configuration, the tests will always be run against the same pages. Each test takes about 5 seconds. That’s 5 seconds to go local page, open it take a screenshot, save the new screenshot, compare it against the old screenshot (the reference), determine if there are differences, then report it all back to you. Just one test from Backstop is far faster than doing it manually. Now it scales because you can test many pages at once! You can also test with greater precision.

Visual Regression Testing at Scale

At the end of the day, the computer will do more work than you will. Computers are perfect machines for performing this repetitive task of comparing two screenshots. Let’s walk through a basic example to see what it takes to get set up with BackstopJS.

Initial Installation of BackstopJS

Install this with NPM. Run the following command.

$ npm install -g backstopjs

Now you have the project installed globally on your machine. You can navigate to your project and initialize BackstopJS there. Once you are in your project’s root directory run the following command.

backstop init

Now you’ll see a new directory and a few files. The file you’re concerned about now is backstop.json

Basic parts of the configuration of BackstopJS

Open up backstop.json and let’s begin to edit the file. But first, let’s look around at a few highlights. Notice at the top of the file there is a field of id. This field is used to name your screenshots. Let’s name it something that is unique to this project. Let’s go with the project name. My project is “Efficiency of Movement”, so I’ll change my id to “eom” for short. That string will prepend all of the screenshots so concise and descriptive. If you have a bunch of projects, you may need something longer, but I doubt that you would need something like www.efficiencyofmovement.com.

Traveling down the file you’ll see a field for viewports. This will be the widths of screenshots. The screenshots that are output ar full screen but at widths specified in the array of viewports. For now, let’s keep the defaults, but adding a desktop, tablet, and mobile makes sense here. I usually set these to my three main sizes set in the configuration of the front end.

The last big section of the configuration is the scenarios. This is where we will specify the pages and the URLs to compare against each other. The scenarios field is an array of objects that represent pages so you can add as many pages as you want. Let’s start with the homepage of our site. In the first object there is a default of the backstop.js website, so let’s change that to our own site. First, change the label. I like to have this structure <page type><page name><page template>. For my home page, the label will be, page_homepage_default-template. This is used in the naming of the screenshots and in my experience it makes it easier to track when there are many pages or post templates.

Next, we change the field of “url” to the production page that we know is perfect. In my case, it’s https://www.efficiencyofmovement.com

Finally, we’ll change the “referenceUrl” to the local version of the page. In my case, the “referenceUrl” will be https://www.efficiencyofmovement.test since that is what my local URL to work on my site is. BackstopJS will take a screenshot of the production page and save it. When I want to test this page locally, that saved screenshot will be the version of the site which the new test screenshot is compared to. Do this for as many pages as you want, but for starters, I’d stick to this one. You can add and change them later if we want to.

Test with Backstop

First, you need to get that set of reference screenshots to compare subsequent tests against. Run the first command.

backstop reference

This will build the set of screenshots in your directory. Remember that these screenshots will be used throughout your tests until you need to replace them with the new version of your site.

Now that you have your references set up you can start testing your local site against your production site. You’ll need one last command:

backstop test

This command will go through your array of scenarios and take screenshots at each of the viewports. For this one page on this one test, you’ll get 2 screenshots. So you can see how this directory can get large and how the cost of running the tests can go up. Regardless, there are more advanced settings to control what gets saved and where and what gets committed to version control to save with your team.

Summary

For now, that’s it. You have installed BackstopJS and set up your first test. Now all you have to do is add more tests, refine tests, and eventually automate this.