All Ruby on Rails Node JS Android iOS React Native Frontend Flutter QA

End to End Tests on CircleCI with Docker - Rails, Capybara, Selenium

The web development world is changing fast. I feel that we are moving towards splitting monolith web applications in two, by extracting backend part into an API. This solution, even though it has many pros, it brings in some difficulties as well. One of which are end to end tests.

It may be really tricky to execute those tests in a proper way, especially when we don't have that much of a control over the process on external hosts, for example during CircleCI build.

I decided to share with you what I've learned - how to set end to end tests on CircleCI. The instructions below can be easily applied to any framework combination other than Rails and React. However, article, in current form, refers to configuration like so:

  • two separate applications for backend (Rails) and frontend (React - the type doesn't really matter) on separate repos

  • tests in Rails are executed by RSpec with Capybara + Selenium + ChromeDriver

  • CircleCI configuration assume incorporating Docker and docker-compose

  • setup process was conducted on existing project with current CircleCI setup and dockerized app for staging and production

I'd like to fit into the form of a simple article, yet I'll try cover as many "What If's" and "Wait, what's" as possible, that may come up during the setup. Most of the lines in showed files that are essential to this setup are commented and explained.The big picture here is to allow writing integration tests (located in spec/features) on backend, test them locally without Docker and run during frontend app build on CircleCI.

Local tests - Backend

Environment setup

Make sure to add proper gems to Gemfile:

...
group :test do
	...
	gem "capybara", "~> 2.7", ">= 2.7.1"
	gem "chromedriver-helper", "~> 1.0"
	gem "rspec_junit_formatter"  # Preparing proper output for CircleCI test metadata
	gem "selenium-webdriver", "~> 2.53", ">= 2.53.4"
end

Then after bundle install let's add configuration for Capybara and Selenium in two files:

require "capybara/rails"
require "selenium-webdriver"

...

Capybara.register_driver :chrome do |app|
  Capybara::Selenium::Driver.new(
    app,
    browser: :chrome,
    desired_capabilities: { "chromeOptions" => { "args" => %w[window-size=1024,768] } },
  )
end

Capybara.register_driver :selenium do |app|
  Capybara::Selenium::Driver.new(app, browser: :chrome)
end

Capybara.configure do |config|
  config.default_max_wait_time = 10
  config.default_driver        = :selenium
end

Capybara.app_host = "http://localhost:3000"
Capybara.javascript_driver = :chrome
Capybara.server_port = 5001 # We don't want it to collide with standard rails server on port 5000
Capybara.server_host = "0.0.0.0" # Start server on localhost as meta-address
Capybara.server = :puma, { Silent: true } # Supress puma STDOUT in console

...
require "capybara/rspec"

Since our tests are going to be placed in: spec/features our test suite on CircleCI is going to fail. We don't want to configure e2e tests on backend's CircleCI. This is why we need to override test command, in circle.yml:

machine:
  services:
    - redis
  environment:
    ES_JAVA_OPTS: "-Xms2g -Xmx2g"
    _JAVA_OPTIONS: "-Xms1024m -Xmx2048m"
    CONTINUOUS_INTEGRATION: true
dependencies:
  post:
    - if [[ ! -e elasticsearch-5.5.1 ]]; then wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.5.1.tar.gz && tar -xvf elasticsearch-5.5.1.tar.gz; fi
    - elasticsearch-5.5.1/bin/elasticsearch: { background: true }
test:
  post:
    - bundle exec codeclimate-test-reporter
    - bundle exec bundle-audit check --update
    - bundle exec brakeman -f plain
  override:                                                              # ADD this section
    - bundle exec rspec --exclude-pattern "spec/features/**/*.rb"        # We override Circle's test command here to explicitly exclude features specs
deployment:
  master:
    branch: master
    commands:
      - cd deployment; bundle install; bundle exec cap staging deploy

Local tests - Frontend

Environment setup

This one is simple - almost none setup needed.

The only thing you have to do is to point requests to specific backend address (in our case it's http://localhost:5001 as stated in rails_spec.rb). This is because we want the frontend app to use the same database as is used by rspec during tests - only in this case tests are viable.

This step depends on the technology you use in your project. It's safe to say that most of the javascript projects have package.json file. Usually we started local server by command:

$ yarn start

since, we don't wan't to mess up development environment we can add custom start-test command in package.json:

"scripts": {
    "build": "react-scripts build",
    ...
    "start": "react-scripts start",
    "start-test": "REACT_APP_API_BASE_URL=http://localhost:5001 react-scripts start",
    ...

Our frontend app depends on the env variable REACT_APP_API_BASE_URL which is used to build reference links to rails API. With the above we can now use:

$ yarn start-test

In your project you must figure out how to set this up. Possibilities:

  • any other env variable consumed at some point by the app - which sets proper environment in which server is started ex.: APP_ENV=test or NODE_ENV=test

  • setup .env file with environment variable same as above

  • some kind of if clause in API related service / component

Is it alive?

If you've set everything correctly at this point you are able to run e2e tests locally. Simple is it working? test case:

require 'rails_helper'

feature 'Home page', js: true do
  scenario 'visit home page' do
    visit '/'
    expect(current_path).to eq '/'
    expect(page.first('span').text).to eq("About Us")
  end
end...

After we started frontend's local server like stated in previous section we are going to run specs manually just by:

$ rspec spec/features

So now let's move to the Crème de la crème - CircleCI setup!

CircleCI - Prerequisites

Let's start with identifying the files you will be working on:

Dockerfile - file used by docker which contains every information and command that is needed to build specific container by docker (containers are like little environments with their own dependencies, variables and configurations). Look at your Dockerfile now. It does not have any extension. IMPORTANT check if your current Dockerfile has two (or more) FROM instructions. If yes, then this is a problem in terms of CircleCI. See, this is called multi-stage build, which is great (it uses one image, takes something from it, adds something from other image and builds one image from which you set up your container. In single-stage you have to build separate images from the scratch - they take up a lot more space and time), but not for CircleCI. Multi-stage build requires Docker v17.05 and above (standard Docker on Circle 1.0 is much older). We can force that version, but only on CircleCI v2.0. So basically, if you have CircleCI v1.0 and multi-stage build Dockerfile - you have a problem to solve. Dockerfile reference

circle.yml - if you have file named like this then you use CircleCi v1.0. This version is way easier to setup. For v2.0 there is .circleci/config.yml file and it's structure is different. If you want you can migrate from 1.0 to 2.0 following steps in: Migrating from 1 to 2 but I personally don't recommend unless your really know what you are doing. However, using 2.0 have many pros - newer version of Docker and docker-compose which support commands like exec which might be useful. Circle.yml reference

docker-compose.yml - file used by, surprisingly, docker-compose tool. You can think of this file as of list of containers (specified by reference to Dockerfile or by image from Docker Hub) with respective config. This is the most important file in which you have to place every part of setup you need: rails, front, postgres, redis, etc, etc. Docker-compose.yml reference.

CircleCI - Backend

Setup Environment

In my opinion best approach here is to add specific environment to rails app, tailored specifically for end-to-end tests on CircleCI. I picked name: e2e. It's just convenient, it can be virtually any name. So to start:

  • Duplicate config/environments/test.rb as config/environments/e2e.rb

  • In Gemfile to each group specified for :test add :e2e as well in the group definition.

  • If you have secrets then you should make a namespace for e2e in config/secrets.yml (copy definitions from development) and add proper secret_key_base in encrypted keys (in our project we edit secrets by EDITOR=nano rails secrets:edit)

  • Remember to have in mind that in places where you use Rails.env.test? or something similar to setup / check anything you might want to specify how to behave in case of e2e environment as well

  • If you use any kind of requests mocking / suppressing / mimicking (like webmock or vcr) you will need to whitelist containers names in virtual network for example frontend, elasticsearch, postgres, redis etc.

Create new file config/database.e2e.yml:

e2e:
  adapter:  postgresql
  host:     postgres
  encoding: unicode
  database: your_project_test
  pool:     5
  username: postgres
  password: your_password

Create Dockerfile.e2e(or duplicate other Dockerfile you have in project). Our is placed in docker/Dockerfile.e2e. Your Dockerfile may look like the one below. You can also create one from scratch using Dockerfile reference.

FROM quay.io/netguru/baseimage:0.10.1                                 # Image containing all dependecies like imagemagick etc

ENV RUBY_VERSION 2.4.2                                                # To get proper ruby

## Install Ruby & Dependencies
RUN \                                                                 # Install ruby
  apt-get update -q && \
  apt-get install -q libcurl3 && \
  apt-get install -q -y cron && \
  ruby-install --system --cleanup ruby $RUBY_VERSION -- --disable-install-rdoc && \
  gem install bundler

## Copy Gemfile & bundle
ADD Gemfile* $APP_HOME/                                               # Gemfile installation
RUN bundle install --jobs=8 --retry=3

## Add rest of code
ADD . $APP_HOME/                                                      # Copy to APP_HOME folder

ENV RAILS_ENV e2e                                                     # Set env to e2e
ENV WEB_CONCURRENCY 2                                                 # Puma concurrency
ENV AVAILABLE_MEMORY 1200                                             # How much memory the container can use
ENV RAILS_SERVE_STATIC_FILES true                                     # Serve static assets
ENV REDIS_URL redis://redis:6379/0                                    # Set Redis url and port

ADD ./config/database.e2e.yml /app/config/database.yml                # Copy database config
RUN mkdir /app/rspec_output                                           # Create directory for rspec results

RUN apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*    # Cleaning up

# Elasticsearch
ENV ELASTICSEARCH_URL=http://elasticsearch:9200                       # Our case - setup ElasticSearch url and port

EXPOSE 5001                                                           # Container will listen on this port

Add e2e specific configuration to rails_helper.rb locally, ENV is 'test', while on CircleCI it's 'e2e'. We don't want to start test server automatically. It will be started in a different way:

Capybara.register_driver :chrome do |app|
  Capybara::Selenium::Driver.new(
    app,
    browser: :chrome,
    desired_capabilities: { "chromeOptions" => { "args" => %w[window-size=1024,768] } },
  )
end

if Rails.env.test?
  Capybara.register_driver :selenium do |app|
    Capybara::Selenium::Driver.new(app, browser: :chrome)
  end

  Capybara.configure do |config|
    config.default_max_wait_time = 10
    config.default_driver        = :selenium
  end

  Capybara.app_host = "http://localhost:3000"
  Capybara.javascript_driver = :chrome
  Capybara.server_port = 5001
  Capybara.server_host = "0.0.0.0"
  Capybara.server = :puma, { Silent: true }
else
  args = ["--no-default-browser-check", "--start-maximized"]
  caps = Selenium::WebDriver::Remote::Capabilities.chrome("chromeOptions" => { "args" => args })
  Capybara.register_driver :selenium do |app|
    Capybara::Selenium::Driver.new(
      app,
      browser: :remote,
      url: "http://selenium:4444/wd/hub",
      desired_capabilities: caps,
    )
  end

  Capybara.configure do |config|
    config.default_max_wait_time = 15
    config.default_driver        = :selenium
  end

  Capybara.app_host = "http://frontend:3000"    # Containers communicate by aliases, hence 'frontend'
  Capybara.javascript_driver = :selenium
  Capybara.run_server = false                   # To ensure everything is in sync don't start puma automatically
end                                             

It's very good idea to keep all commands you need to start specs within one script. We can create one in docker/e2e.sh. This bash script will be started in container by a docker-compose.

#!/bin/bash

echo "Setting up and running e2e tests!"

# Create database and seed (since we might want to include some data needed for our app to work properly like oauth clients
rails db:setup

# Start puma server in e2e environment on meta-address host on port 5001 in detached mode
RAILS_ENV=e2e puma -b tcp://0.0.0.0:5001 -d

# Start feature specs and output results to xml file
rspec --format progress --format RspecJunitFormatter --out ./rspec_output/rspec.xml spec/features

That's it! We're done with backend.

CircleCI - Frontend

Setup Environment

Create new docker-compose.ci.yml file in project's main directory or just duplicate and rename current one for ex. docker-compose-staging.yml . You will need to setup few things: proper dependencies for backend, volumes, ports, aliases etc. Everything is explained below. In the end this file should more or less look like this:

version: '2'                                       # Version of composer
services:
  backend:                                         # Backend container
    build:
      context: ~/backend                           # Backend should be pulled by git to this location
      dockerfile: docker/Dockerfile.e2e            # The location of Dockerfile, relative to context above
    depends_on:                                    # The containers below will be resolved and loaded before backed
      - redis
      - postgres
      - frontend
      - selenium
      - elasticsearch
    ports:
      - "5001:5001"                                # Expose host_port:container_port
    volumes:
      - "/rspec_output:/app/rspec_output"          # Mount  /rspec_output (on host) to /app/rspec_output (in container)
    environment:
      - RAILS_ENV=e2e                              # This env var will be available to any container instance - used to run rspec in e2e env
      - RAILS_MASTER_KEY                           # If no value is passed this var is going to be read from machine ENV - very useful. Just set RAILS_MASTER_KEY for secrets in CircleCI build config page
    networks:                                      # Virtual network for containers
      main:                                        # Name of the network
        aliases:
          - backend                                # Alias for reference purposes
    command: bash -c "bash /app/docker/e2e.sh"     # Start container by a bash script (container will be alive as long as this script is running)
  frontend:                                        # Frontend container
    build:
      context: .
      dockerfile: Dockerfile.e2e
    command: nginx
    logging:                                       # Disable spamming irrelevant messages from this container
      driver: none
    networks:
      main:
        aliases:
          - frontend                               # Container name in virtual network that can be addressed
    ports:
      - "3000:3000"
    volumes:                                       # Used to mount ./tmp on machine to /app/dist in container
      - "./tmp:/app/dist"
  selenium:                                        # Selenium container
    image: selenium/standalone-chrome
    ports:
      - "4444:4444"
    logging:                                       # Disable spamming irrelevant messages from this container
      driver: none
    networks:
      main:
        aliases:
          - selenium                               # Container name in virtual network that can be addressed
  elasticsearch:                                   # Elasticsearch container (not required) - we used it in project
    image: docker.elastic.co/elasticsearch/elasticsearch:5.5.1
    volumes:
      - "~/es_data:/usr/share/elasticsearch/data"
    environment:
      - "xpack.security.enabled=false"
      - "transport.host=localhost"                 # This line might be needed to disable errors with circleCI 1.0 container memory amount
      - "bootstrap.system_call_filter=false"       # This line might be needed to disable errors with circleCI 1.0 container memory amount
    ports:
      - "9300:9300"
    logging:                                       # Disable spamming irrelevant messages from this container
      driver: none
    networks:
      main:
        aliases:
          - elasticsearch                          # Container name in virtual network that can be addressed
  postgres:                                        # Postgres container
    image: postgres:9.5                            # Postgres version, installed from image on Docker Hub
    environment:
      - POSTGRES_PASSWORD=password                 # There can't be 'blank' password, use one specified in database.e2e.yml
    logging:                                       # Disable spamming irrelevant messages from this container
      driver: none
    networks:
      main:
        aliases:
          - postgres                               # Container name in virtual network that can be addressed
  redis:                                           # Redis container
    image: redis:4.0.6                             # Redis version, installed from image on Docker Hub
    logging:                                       # Disable spamming irrelevant messages from this container
      driver: none
    networks:
      main:
        aliases:
          - redis                                  # Container name in virtual network that can be addressed

networks:                                          # Set virtual network up
  main:

If you need another service (in container) just add it to the list in a similar way, watch out for the order.

Now to the frontend Dockerfile. Same as before - copy current Dockerfile or create one in project's root. Unfortunately we had multi-stage build Dockerfile and CircleCI v1.0 so I had to convert multi-stage to single-stage. Let me use an example. At first we had this file:

## specify node version
FROM quay.io/netguru/ng-node:6 as builder    # First 'FROM', as builder is used as reference below

## add necessary environments
ENV NODE_ENV staging                         # Set env var

## add code & build app
ADD . $APP_HOME                              # Copy / add external source to image's filesystem
RUN yarn install                             # Install all dependencies for frontend app
RUN yarn build                               # Compile frontend build

## Real app image
FROM nginx:alpine as app                     # Second 'FROM', now it's multi-stage build file

## Copy build/ folder to new image
COPY --from=builder /app/build /app/dist     # Copy file from one image to another by reference - we will have to get rid of it
COPY nginx.conf /etc/nginx/nginx.conf        # Copy nginx config to nginx directory

EXPOSE 3000                                  # Container will listen on this port at runtime

Now it have to be split up in two files:

## Real app image
FROM nginx:alpine                       # Remove 'as app'

COPY nginx.conf /etc/nginx/nginx.conf

EXPOSE 3000
## specify node version
FROM quay.io/netguru/ng-node:6                    # Remove 'as builder'

## add necessary environments
ENV NODE_ENV e2e                                  # Set env
ENV REACT_APP_API_BASE_URL http://backend:5001    # Set env var to point to backend within build. It can be also done by specific command in package.json

## add code & build app
ADD . $APP_HOME
RUN yarn install                                  # Install dependencies
RUN yarn build

If you look carefully you see that I deleted COPY --from=builder /app/build /app/dist - this will be done in a different way.

The above change will force two new steps. In circle.yml we have to add in dependencies/pre after pyenv rehash:


dependecies:
  pre:
    ...
    - pyenv rehash
    - docker build -t frontend_img -f Dockerfile.e2e.build .    # Builds image from Dockerfile.e2e.build (with our front app)
    - docker create --name frontend_pre frontend_img            # Creates container from image
    - docker cp frontend_pre:/app/build ./tmp                   # Copies /app/build directory (compiled app) from a container to ./tmp in CircleCI machine
    ...

In case your config is:

  • multi-stage + v1.0 - convert multi-stage to single-stage
  • multi-stage + v2.0 - make sure that you force proper Docker version and Edge build of CircleCI
  • single-stage + v2.0 - do nothing, just use that Dockerfile
  • single-stage + v1.0 - do nothing, just use that Dockerfile

The last step is to configure circle.yml and setup precedence of commands and indicate usage of Docker. In the end circle.yml should look like:

machine:
  node:
    version: 8.2.1
  pre:
    - mkdir ~/.cache/yarn
    - curl -sSL https://s3.amazonaws.com/circle-downloads/install-circleci-docker.sh | bash -s -- 1.10.0          # Download Docker 1.10.0 (linked to docker-compose v2) for CircleCI
    - >-
      GIT_SSH_COMMAND='ssh -i ~/.ssh/backend-e2e-tests-deploy-key'
      git clone git@github.com:netguru/project.git ~/backend && cd ~/backend                   # Download backend to ~/backend. The SSH for backend had to be placed in CircleCI build setup page
    - set -o allexport; source "${HOME}/${CIRCLE_PROJECT_REPONAME}/.env"; set +o allexport
  environment:
    PATH: "${PATH}:${HOME}/${CIRCLE_PROJECT_REPONAME}/node_modules/.bin"
  services:
    - docker                                                    # Enable Docker for this build

dependencies:
  pre:
    - sudo mkdir /rspec_output && mkdir $CIRCLE_TEST_REPORTS/rspec && mkdir ~/es_data # make directories for mounting volumes and directory for test metadata. es_data is related to elasticsearch container
    - pip install docker-compose                                # Update docker-compose
    - pyenv rehash                                              # Refresh docker-compose link
    - docker build -t frontend_img -f Dockerfile.e2e.build .    # Build first part of former multi-stage Dockerfile
    - docker create --name frontend_pre frontend_img            # Create container from image above
    - docker cp frontend_pre:/app/build ./tmp                   # Copy compiled frontend app build from container to machine ./tmp dir
    - docker-compose -f docker-compose.ci.yml build             # Build all containers in docker-compose.ci.yml
  cache_directories:
    - ~/.cache/yarn
  override:
    - yarn install

test:
  override:
    - docker-compose -f docker-compose.ci.yml up --exit-code-from backend # Start containers from docker-compose. This will run in foreground and output exit code of bash script
    - ? case $CIRCLE_NODE_INDEX in                                        # Standard frontend tests
        0) yarn test --runInBand ;;
        1) yarn eslint && yarn stylelint ;; esac
      : parallel: true
    - cp /rspec_output/rspec.xml $CIRCLE_TEST_REPORTS/rspec/rspec.xml     # Copy rspec results to CircleCI test metadata folder

deployment:
  master:
    branch: master
    commands:
      - cd deployment; bundle install; bundle exec cap staging deploy

This is it, at this point everything should be safe and sound. If not and your build is failing ssh to CircleCI and play around.

Useful commands

$ docker ps                                         # show running containers
$ docker ps -a                                      # show all containers
$ docker ps -n=-1                                   # show n last created containers
$ docker start                                      # start container
$ docker stop                                       # stop container
$ docker cp :                                       # copy file/directory from container to destinated dir in machine
$ docker cp  :                                      # copy file/directory from machine to container (don't get deceived, read 'understanding volumes' article)
$ docker images                                     # list all images
$ docker rm -f                                      # remove container
$ docker rmi                                        # remove image
$ docker-compose run                                # run a command in container specified in docker-compose file. Watch out - this command creates new containers each time. Moreover it doesnt create a network!
$ docker-compose exec                               # execute a command in container that is already running (unavailable in older versions of Docker (CircleCI v1.0))
$ docker-compose up                                 # start containers from docker-compose file by specified commands (creates networks). Containers are running as long as commands which run them

Useful links

docker-compose run reference

docker cp reference

article about multi-stage build

another article about multi-stage build

yet another article about multi-stage build

understanding volumes in containers

copy data between containers

FAQ

Is it hard to setup?

It may be. Everything depends on CircleCi version you have, which limits interactions with Docker (old versions) and how many dependent containers you need for your backend. Some of them might be really tricky to setup correctly.

Should I consider migrating to CircleCI v2.0?

Definitely! This build takes much time to execute. With newer version of Circle you might speed it up significantly and use new Docker (less bugs, some commands actually work, not just fail).

What if I have CircleCI v2.0 file?

Then you should be happy. Everything you see above is still valid, only that you don't have to worry about multi-staging. Just follow this guide Multi-stage docker build with with CircleCI v2.0 and rewrite our circle.yml additional lines using 2.0 style CircleCI v2.0 reference

Summary

I hope that I've managed to describe everything to enable you to set end to end tests on your own in your project.

Please feel free to ask if anything is unclear. Maybe article is unclear or needs improvements? Point it out as well. I believe the best way we learn is by a mistake, we can both benefit from it.

Later, I'm planning to prepare a follow-up for CircleCI v2.0, so stay tuned!

We're building our future. Let's do this right - join us
READ ALSO FROM CircleCi
Read also
Need a successful project?
Estimate project or contact us