The web development world is changing fast. I feel that we are moving towards splitting monolith web applications in two, by extracting backend part into an API. This solution, even though it has many pros, it brings in some difficulties as well. One of which are end to end tests.
It may be really tricky to execute those tests in a proper way, especially when we don't have that much of a control over the process on external hosts, for example during CircleCI build.
I decided to share with you what I've learned - how to set end to end tests on CircleCI. The instructions below can be easily applied to any framework combination other than Rails and React. However, article, in current form, refers to configuration like so:
two separate applications for backend (Rails) and frontend (React - the type doesn't really matter) on separate repos
tests in Rails are executed by RSpec with Capybara + Selenium + ChromeDriver
CircleCI configuration assume incorporating Docker and docker-compose
setup process was conducted on existing project with current CircleCI setup and dockerized app for staging and production
I'd like to fit into the form of a simple article, yet I'll try cover as many "What If's" and "Wait, what's" as possible, that may come up during the setup. Most of the lines in showed files that are essential to this setup are commented and explained.The big picture here is to allow writing integration tests (located in spec/features) on backend, test them locally without Docker and run during frontend app build on CircleCI.
Make sure to add proper gems to
bundle install let's add configuration for Capybara and Selenium in two files:
Since our tests are going to be placed in:
spec/features our test suite on CircleCI is going to fail. We don't want to configure e2e tests on backend's CircleCI. This is why we need to override test command, in
This one is simple - almost none setup needed.
The only thing you have to do is to point requests to specific backend address (in our case it's
http://localhost:5001 as stated in
rails_spec.rb). This is because we want the frontend app to use the same database as is used by rspec during tests - only in this case tests are viable.
package.json file. Usually we started local server by command:
$ yarn start
since, we don't wan't to mess up development environment we can add custom
start-test command in
Our frontend app depends on the env variable
REACT_APP_API_BASE_URL which is used to build reference links to rails API. With the above we can now use:
$ yarn start-test
In your project you must figure out how to set this up. Possibilities:
any other env variable consumed at some point by the app - which sets proper environment in which server is started ex.:
setup .env file with environment variable same as above
some kind of
if clause in API related service / component
If you've set everything correctly at this point you are able to run e2e tests locally. Simple
is it working? test case:
After we started frontend's local server like stated in previous section we are going to run specs manually just by:
$ rspec spec/features
So now let's move to the Crème de la crème - CircleCI setup!
Let's start with identifying the files you will be working on:
Dockerfile - file used by docker which contains every information and command that is needed to build specific container by docker (containers are like little environments with their own dependencies, variables and configurations). Look at your Dockerfile now. It does not have any extension. IMPORTANT check if your current
Dockerfile has two (or more)
FROM instructions. If yes, then this is a problem in terms of CircleCI. See, this is called multi-stage build, which is great (it uses one image, takes something from it, adds something from other image and builds one image from which you set up your container. In single-stage you have to build separate images from the scratch - they take up a lot more space and time), but not for CircleCI. Multi-stage build requires Docker v17.05 and above (standard Docker on Circle 1.0 is much older). We can force that version, but only on CircleCI v2.0. So basically, if you have CircleCI v1.0 and multi-stage build Dockerfile - you have a problem to solve. Dockerfile reference
circle.yml - if you have file named like this then you use CircleCi v1.0. This version is way easier to setup. For v2.0 there is
.circleci/config.yml file and it's structure is different. If you want you can migrate from 1.0 to 2.0 following steps in: Migrating from 1 to 2 but I personally don't recommend unless your really know what you are doing. However, using 2.0 have many pros - newer version of
docker-compose which support commands like
exec which might be useful. Circle.yml reference
docker-compose.yml - file used by, surprisingly,
docker-compose tool. You can think of this file as of list of containers (specified by reference to Dockerfile or by image from Docker Hub) with respective config. This is the most important file in which you have to place every part of setup you need: rails, front, postgres, redis, etc, etc. Docker-compose.yml reference.
In my opinion best approach here is to add specific environment to rails app, tailored specifically for end-to-end tests on CircleCI. I picked name:
e2e. It's just convenient, it can be virtually any name. So to start:
Gemfile to each group specified for
:e2e as well in the group definition.
If you have secrets then you should make a namespace for
config/secrets.yml (copy definitions from development) and add proper
secret_key_base in encrypted keys (in our project we edit secrets by
EDITOR=nano rails secrets:edit)
Remember to have in mind that in places where you use
Rails.env.test? or something similar to setup / check anything you might want to specify how to behave in case of
e2e environment as well
If you use any kind of requests mocking / suppressing / mimicking (like webmock or vcr) you will need to whitelist containers names in virtual network for example
Create new file
Dockerfile.e2e(or duplicate other
Dockerfile you have in project). Our is placed in
Dockerfile may look like the one below. You can also create one from scratch using Dockerfile reference.
Add e2e specific configuration to
rails_helper.rb locally, ENV is 'test', while on CircleCI it's 'e2e'. We don't want to start test server automatically. It will be started in a different way:
It's very good idea to keep all commands you need to start specs within one script. We can create one in
docker/e2e.sh. This bash script will be started in container by a
That's it! We're done with backend.
docker-compose.ci.yml file in project's main directory or just duplicate and rename current one for ex.
docker-compose-staging.yml . You will need to setup few things: proper dependencies for backend, volumes, ports, aliases etc. Everything is explained below. In the end this file should more or less look like this:
If you need another service (in container) just add it to the list in a similar way, watch out for the order.
Now to the frontend
Dockerfile. Same as before - copy current
Dockerfile or create one in project's root. Unfortunately we had multi-stage build
CircleCI v1.0 so I had to convert multi-stage to single-stage. Let me use an example. At first we had this file:
Now it have to be split up in two files:
If you look carefully you see that I deleted
COPY --from=builder /app/build /app/dist - this will be done in a different way.
The above change will force two new steps. In
circle.yml we have to add in
In case your config is:
v1.0- convert multi-stage to single-stage
v2.0- make sure that you force proper Docker version and
Edgebuild of CircleCI
v2.0- do nothing, just use that
v1.0- do nothing, just use that
The last step is to configure
circle.yml and setup precedence of commands and indicate usage of Docker. In the end
circle.yml should look like:
This is it, at this point everything should be safe and sound. If not and your build is failing ssh to CircleCI and play around.
Is it hard to setup?
It may be. Everything depends on CircleCi version you have, which limits interactions with Docker (old versions) and how many dependent containers you need for your backend. Some of them might be really tricky to setup correctly.
Should I consider migrating to CircleCI v2.0?
Definitely! This build takes much time to execute. With newer version of Circle you might speed it up significantly and use new Docker (less bugs, some commands actually work, not just fail).
What if I have CircleCI v2.0 file?
Then you should be happy. Everything you see above is still valid, only that you don't have to worry about multi-staging. Just follow this guide Multi-stage docker build with with CircleCI v2.0 and rewrite our
circle.yml additional lines using 2.0 style CircleCI v2.0 reference
I hope that I've managed to describe everything to enable you to set end to end tests on your own in your project.
Please feel free to ask if anything is unclear. Maybe article is unclear or needs improvements? Point it out as well. I believe the best way we learn is by a mistake, we can both benefit from it.
Later, I'm planning to prepare a follow-up for CircleCI v2.0, so stay tuned!