CodeClimate is a well-developed and very stable solution with a great number of features. It has many advantages over its competitors and many big players recommend it as the best option. However, it lacked some crucial functionalities that we required for the Netguru code review process, so we switched to an alternative solution.
- a great number of supported languages, technologies and frameworks,
- used by the biggest players, including Pivotal, New Relic for enterprise and Rails, jQuery for open-source,
- very stable,
- nice new UI,
- well-maintained test coverage feature gem,
- browser extensions,
- trends charts,
- test coverage out of the box,
- hotspots - a `quick wins` list.
- seemingly an integrated bunch of open-source projects,
- pricing - it seems to be the most expensive tool in this comparison,
- still unpredictable API (in beta),
- no support for Objective-C,
- no distinct types for total issues number,
- no detailed description of the issue, only a header with source code,
- no issue searching/filtering, just a paginated list with all of them,
- no interest from the Code Climate team to extend the tool the way a customer may suggest.
Scrutinizer doesn’t stack up well against other solutions. Actually, it seems to lack basic attributes that are necessary for code review. Analysing the code using different a Ruby version than defined in the repo is actually what disqualifies it. It shows `tests failed` even when the tests passed on CircleCI and locally. You can not reliably test the repository if you’re using a Ruby version other than the ones available through the dashboard. It shows lots of issues and errors with the “parser could not interpret the code” message. Really strange for a tool that you’re supposed to pay for.
- seems to have a very good API,
- it’s actually one of the cheapest solutions if we ignore the performance,
- automatically detects code changes in the dashboard,
- well-written documentation,
- feature of filtering issues by users,
- dedicated site with the current status of the services,
- extended team management, possibility to create organisations.
- available only as a predefined version (e.g. no option of running tests in 2.3.0 version),
- dead test coverage gem (only 2 commits 3 years ago),
- API available only when subscribed to the most expensive package,
- unknown warnings on Regexps (e.g. treating `/` like a division instead of the beginning of the regex when no space after it),
- not a predictable tool.
- performance: you pay per container, and a container is simply one task that can be run at once, fewer containers - less performance and longer waiting time for your code to be analysed,
- it’s possible to configure checker (Settings -> Configurations -> Checks), but there’s no way to restore default configuration or even set a default for an entire organisation,
- it doesn’t read class names properly (it should include namespace - module), it’s harder to find it this way,
- files/issues are paginated in the way that enforces you to click `next` every time you want to move to the next - say - 10 classes,
- there’s no option to see all the rules (checks), there are two groups for enabled or disabled rules, but there’s no “all” option.
Code Review Tools Comparison Summary
Supported Languages and Technologies:
CodeClimate: many existing, open-source tools like Rubocop, Brakeman, CSS/SCSS Lint, ESLint, Flog, etc. full list here;
Codebeat: their own algorithms and implementation written from scratch see how it works;
Codacy: many existing, open-source tools like Rubocop, Brakeman, CSS/SCSS Lint, ESLint, Flog etc. full list here;
Scrutinizer: closed-source codebase with a possibility to use open source tools. OS tools list here.
CodeClimate: $16,67/user/month (when billed yearly - otherwise $20);
Codebeat: $20/user/month, but you can probably negotiate;
Scrutinizer: EUR200/month for unlimited number of users. 2 containers in the plan, where 1 is just 1 task at a time, extra container costs EUR50.
CodeClimate: very good and comprehensive;
Codebeat: still some things missing;
Codacy: not bad, not too much text and some images are not clickable so the readability is limited;
CodeClimate: yes, still in beta version;
Codebeat: yes, simple but usable;
Codacy: yes, not described perfectly in their docs;
Scrutinizer: seems to be really good - read more.
CodeClimate Codebeat Codacy Scrutinizer
Team per project
Github PR integration
total no. of integrations
After a deeper look at all the tools described above, I can disqualify Scrutinizer as it doesn’t support all Ruby versions, has many issues which don’t allow to do the metrics reliably and sometimes makes it actually impossible. On the other hand, Codebeat seems to be a comparable tool to CodeClimate or Codacy. In case of these three solutions, the pros seem to overweigh the cons and a particular feature might clinch the final choice, which would stem from individual needs. Just like it did in case of Netguru.
The article was originally published here.