Do you sometimes need to make difficult changes in your testing strategy without being entirely sure where it’s going to get you?
If yes, you need to consider their overall impact on the product and how much effort such changes might necessitate.
Changing the end-to-end testing framework from Capybara to Cypress in a commercial project was one of such decisions. We couldn’t have predicted how impactful it would become and how it was going to change our future work. Let’s step back in time to 2019 and check this out.
Ruby on Rails app with Capybara automated tests
The product was an accounting web application for a small business. It was built in Ruby on Rails and, at some point, migrated its frontend development from native Ruby on Rails to React. The application was monorepo with a PostgreSQL database, Rails server, Sidekiq background processes server, and React for the frontend app.
When I joined the project, we used the Capybara testing library written in Ruby to run end-to-end integration tests. Capybara is a well-known tool in the Ruby community. It helps developers to automate users' interaction with the app, creating tests for defined flows and running them inside the browser. It works out of the box with Rails Apps and provides an extensive API for different kinds of interactions.
We used Capybara to automate the most critical test scenarios (having 110+ scenarios altogether). The test suite was fired on the Continuous Integration server each time a new commit was pushed to the branch.
The reasons why we went away from Capybara to Cypress
- Each new frontend developer in the team required one-on-one mentoring sessions about the basics of Capybara and its debugging techniques.
- When a Capybara test failed, frontend developers had to debug the script and fix it. In some cases, that required support from backend developers, which resulted in frustration and a decrease in velocity.
- Developers were not keen to write automated integration tests, since Ruby was not their primary language to code. I would consider this point as a major disadvantage of the previous setup, since we wasted a lot of human potential.
Another issue that we found while working with the Capybara test suite was that the tests were not stable in the Continuous Integration (CI) server. We had many flaky tests with unsolved reasons for their failure. At some point, the test configuration was set to rerun a single Capybara scenario up to six times when it failed, just to make CI build pass. As you might already have guessed, such a test suite was slow. No fewer than 109 scenarios were executed in parallelization, utilizing ten containers, running for about nine minutes.
Our expectations for the new framework
We decided to start doing research to try to find something that could replace Capybara, so we created a list of requirements with arguments why a given point was important to us.
Five major points from the list:
- Tests need to run faster than Capybara – enlarging the test suite should not result in a significant increase in CI server costs.
- Tests should be easy to debug.
- Due to the complexity of the app, we required full access to the application database state. Test data had to be created before test execution and needed to be easily modified. The biggest advantage of Capybara was its integration with the Rails app and the ability to use supported ORM to create test resources directly in the database. Access to resource creation and edition from inside the test code is a very powerful approach, and it allows for the testing of even the most complex scenarios on the integration level. We definitely needed the new framework to have a similar mechanism.
- No randomly failing tests on CI – the aim was to not disturb the team.
The final decision of moving to Cypress tests
Still, we had an important challenge to solve: How to combine the JS code with the Ruby on Rails backend to be able to create Cypress tests with a mechanism used by Capybara. It was essential as we needed to have direct access to the backend code execution so we could create test data based on application models.
Test data management solution
One of our backend developers, Jan Kamyk, suggested a very clever solution. We hid an endpoint inside our API, which only worked in testing environments. This endpoint allowed us to execute code from a selected Ruby class and a selected method placed in a defined location inside our project structure. The aim of this method was to create the necessary data required for a single test.
The endpoint was wrapped in Cypress custom command with two arguments required – class name and method name – that could be executed from inside the JS/TS test file. Almost every Cypress test file had a separate corresponding Ruby class inherited from the base class (including the most common data creation methods) in which you could create code to execute. Editing, deletion, creation, any calculations, returning the ID of resources or any data you wanted – all the things you could do from inside the Ruby code were available to Cypress as well. It seemed like a perfect approach, but was it really so beneficial?
Some disadvantages of Cypress migration
As our project evolved, we had some changes in architecture that required model restructures and data migration. This caused extra maintenance for our test suite. We had to refactor all the Cypress tests related to Ruby classes containing connections to the previous models and needed to rewrite them to the new ones in order to make the tests pass again. That was unplanned, somewhat unexpected, and arduous.
Test automation transition and the overall impact on the project
Despite initial difficulties with writing the first tests, we pushed the initiative further and soon realized that the new setup was starting to have a positive impact on the project's quality. We took an approach where Cypress end-to-end integration tests were written for every new feature that we delivered to the end user. At the same time, we kept our Capybara tests running for critical scenarios (it had been a temporary solution until we rewrote all those scenarios to Cypress).
Frontend developers participated in the development of integration tests and wrote basic scenarios which were further extended by QAs to include additional assertions or new scenarios. Bugs were caught at the early stage of development, since the whole test suite was triggered on every commit pushed to the branch.
The list of manual test cases to be checked before release was not growing, but still there were many tests to write – an automation debt to pay. We discussed the case within the team and decided to include one additional task for automation in the sprint. The task was taken by QA or a developer – whoever had the capacity to deliver it.
Soon, the list of unautomated test scenarios was shorter, and we decided to start to rewrite existing critical Capybara tests to Cypress. It was a good moment to evaluate those tests, think about missing scenarios, and include missing assertions. One year after introducing Cypress, we had 54 test files with 209 test scenarios, 830 different kinds of assertions, and still 62 scenarios from Capybara tests to be rewritten.
Our QA daily work changed significantly. Instead of manual exploration tests performed after the feature was ready in a deployed testing environment, we started writing automation tests locally, which gave us a much bigger understanding of the solution. In the process of writing, we obviously noticed bugs which could also be found during manual tests. Fixes had been made before any code got to a stable testing environment. We worked closely with the frontend developers to decide which scenarios should be covered by unit tests and which assertions should be included in Cypress tests.
We consistently worked on a testing suite to get rid of flaky test scenarios. In the end, we had full confidence in the test suite – once a test failed, it was clear that some problem had come up.
Tests were run on the integration level to test the local frontend, local backend, and the new, fresh database. Both backend and frontend teams had their own unit tests developed. In order to make sure our application worked properly for already existing resources, we added another test layer which ran through the app in the deployed testing environment. Finally, we supported our project with automated visual regression tests. In the end, we built a stable testing system ensuring high quality of the product in two important areas: backend-frontend integration and UI/visual consistency.
With such a setup in place, we changed how we deployed the application to production. Once a feature was developed, the end-to-end integration tests were already there. We had waited for the final confirmation from stakeholders performing acceptance tests and, following that, the code was deployed to production. Production deployment took place every day.
Positive outcomes of Cypress implementation
Our relationship with the investors changed positively. We were able to deliver planned features in a more predictable way. We had fewer regression bugs found at the later stage of development. Once a bug had been found, we wrote proper tests to ensure it wouldn't appear again. There were no longer any manual regression tests before releases other than occasionally scheduled exploratory testing sessions.
One and a half years after introducing Cypress, we finally rewrote all the Capybara specs. Here are some important facts about that:
- Our Cypress integration test suite consisted of over 400 test scenarios.
- We decreased the initial CI build time from nine minutes for 109 Capybara scenarios with the parallelization of ten containers to seven and a half minutes for 419 Cypress scenarios with parallelization of ten containers – almost four times more tests and still visibly faster builds.
- We reduced CI server costs.
- We built confidence in the test suite by removing flaky tests.
- We moved our working flow towards continuous delivery, being able to deploy any fix or feature to production thanks to our powerful test automation solution.
My thoughts after the successful test automation transition
In a period of one and a half years, I made over 1,100 commits to the project's repository. I’ve spent hours on fixing code review requests and learned how to collaborate on the shared code, contribute to others' work. I got a much deeper understanding of the product and learned a lot from the developers I’d worked with.
The most important thing I would like to share is that whenever something is not working as projected, you should start the discussion within your team. Plan your tasks step by step, and do not be discouraged by the amount of work left to achieve your goal. Do not give up when you face an issue: ask for support, talk to people. Think about automation as a strategic investment in both the project and your well being – less stressful deployments will make the job easier.
Every major change needs a driving force – someone who has a sense of ownership, who monitors the progress, who is deeply convinced of the necessity to change, and who pushes the work further. Being the driving force behind such a change was a huge responsibility, but it was definitely one of the most rewarding achievements in my professional career so far.