The Top 6 Python Testing Frameworks

Python's flexibility is great for all kinds of business use cases, but it’s vital to make sure you test your code prior to releasing any products written in Python. In this article, we’ll explain what Python testing means and take you through the top six Python testing frameworks. By the end, you’ll understand which testing framework is right for your specific project.

What is Python Testing?

Python testing is simply the act of testing your code to make sure it works properly. But it’s not always as simple as it sounds. Python is an open source, general-purpose programming language, and its popularity means there are a huge number of testing frameworks to choose from; each offering automated, script-based testing which makes test cases very easy to write. However, with so many options, it can be difficult to decide on the right one for your use case.

Each Python testing framework has its pros and cons, but every project has different requirements, meaning that there almost definitely something out there that suits your business project. So let’s take a closer look at the top 6 Python Testing Frameworks and the pros and cons of each.

Robot Framework

Robot Framework (RF) is an open source test automation framework for acceptance testing, acceptance test-driven development (ATDD), Behavior driven development (BDD) and robotic process automation (RPA). Its core is based in Python, but it also works in Jython (the name given to the Java implementation of Python) and IronPython. This requires Python v.2.7.14 or higher to run.

Pros

• RF is based on keyword-driven testing, meaning it is easy to create test cases using words readable to humans, so no coding experience is required.

• It supports all operating systems (MacOS, Windows, Linux) and all application types, like web apps, mobile apps. etc.

• User friendly reporting data

• Lots of APIs and rich ecosystem making it highly extensible and easy to integrate with third party tools

• Supports If/Else syntax starting from RF v4.0

• Great community support

Cons

• No out-of-box Parallel testing support, but this can be achieved with Selenium Grid via Pabot (a parallel executor for RF)
• Slightly longer learning curve for beginners, who have to learn how to work within RF’s predefined methodology
• Creating generic keywords takes longer than writing coded tests
• Report customization is quite difficult

Pytest

Pytest is an open source testing framework that is one of the most widely used Python testing frameworks available. Pytest supports unit testing, functional testing and API testing. It requires Python version 3.5 or higher to run.

Pros

• Offers compact, simple test suites
• Highly extensible with many plugins available, such as Pytest html plugin, which can be added to your project to print HTML reports with a single command-line option.
• Large community and plenty of support
• Helps to cover all parameter combinations without having to rewrite test cases

Cons

• Pytest’s unique routines impede its compatibility, meaning that although test cases are easy to write in this framework, you won’t be able to use those in other testing frameworks.

TestProject

TestProject is an automation framework which is completely free. It offers cloud and local HTML reports and easy development of test automation. It supports the Pytest and Unittest frameworks and all dependencies required as part of the single cross-platform agent executable. It requires Python v3.6 or higher.

Pros

• Free, automatic reports in HTML/PDF format
• Easy access to execution history through RESTful API
• Always updated with the latest Selenium/Appium driver versions
• Offers a single SDK for Android, iOS, Web, and Generic tests
• Built-in test runner and reporting capabilities
• Cross platform support for all operating systems
• Extensive community and support.

Cons

• Only runs one test at a time, so requires parallel testing support
• Team collaboration tools are limited in offline mode, so when working on-premises, you will need to save tests on a shared network drive/git.

PyUnit (Unittest)

PyUnit (also called Unittest) is a unit testing framework inspired by JUnit. It is the default Python testing framework that comes as standard with the Python package. As such, most developers start their testing with PyUnit.

Pros

• Doesn’t require installation of additional modules as it is part of the standard Python library
• Provides simples, clean and quick test case execution
• Rapid generation of test reports, including XML reports and unittest-sml-reporting

Cons

• Test code supports abstraction, which means intent can sometimes become unclear
• Requires a lot of ‘boilerplate’ code

Testify

Testify is a unit testing framework - also used for integration and system testing - designed to replace the common Unittest and Nose frameworks, adding advanced additional features to and the standard Unittest.

Pros

• Easy to get started with
• Offers extensive plugins for additional functionality
• Enables test discovery
• Simple syntax

Cons

• Difficult to achieve parallel testing
• Lacks extensive documentation, so beginners might need to spend extra time finding relevant resources.
  
  

Doctest

Dockets is a module included in the Python programming language's standard library that allows the easy generation of tests based on output from the standard Python interpreter shell, cut and pasted into docstrings.

It searches for interactive Python sessions to check if they are working properly and uses selective Python capabilities such as docstrings.

Pros

• Comes as standard, so requires no installation and is easy to get started with
• Comprehensive code documentation, with up-to-date interactive examples

Cons

• Only compares printed outputs and any variation will result in test failure
• Static in nature, so cannot be parameterized
• Doesn't support test discovery, test fixtures and test runner features

So which Python testing framework is best for your project?

Which Python testing framework you choose will depend on your - or your team’s - level of experience, as well as the type of testing required. If you want to run unit testing, you’ll need to choose between the likes of Testify or PyUnit. But if you want to achieve more automated functional testing, something like Robot Framework or TestProject could work better. If your team has quite a bit of coding experience, Robot Framework and PyUnit are great tools, whereas Doctest and Testify are very easy to get started with for those with less knowledge.