Your Best Friend For CI/CD - What Is CircleCI?

Photo of Dániel Juhász

Dániel Juhász

Updated Dec 21, 2022 • 13 min read
Netguru-Biuro-2018-6505

In an agile world, we must quickly adapt to changes, but that can be a challenge for engineering teams with fixed deadlines. CircleCI comes to aid here – a top CI/CD tool.


Automation and CircleCI come in here – a top CI/CD tool. Below, we explore what is CircleCI, plus more.

Automation helps speed up work, and a big part of that is CI/CD.

The first part, CI, stands for Continuous Integration. This concept encourages developers to regularly integrate code they’re working on with a shared repository – as frequently as possible. When the code merges, that triggers automated tests and builds.

The second part, CD, is ambiguous. It stands for Continuous Delivery or Continuous Deployment. Both mean further automation of the previous integration process. Continuous Delivery is about the automated creation of images in software applications.

These images are present in some kind of repository and are ready for manual deployment. With Continuous Deployment, that manual step is also covered by the automated process.

What we’ve outlined may seem like a lot of unnecessary automation fuss, but these practices are worthwhile investments. With CI/CD platforms, software teams are notified about bugs faster, helping deliver a more stable product while increasing team confidence and productivity. To give you a better idea, let’s take a look at CircleCI Continuous Integration tool.

What is CircleCI?

CircleCI is a CI/CD solution that helps improve your team’s efficiency. It provides a managed cloud service, and also has a self-hosted option. You can access CircleCI via its website – there’s a free tier, so it’s easy to try out.

Over the years, CircleCI has become one of the most popular CI/CD platforms. Below, we take a look at its features and example pipelines to see why so many companies choose CircleCI to automate their workflows.

Components of CircleCI

In general, most CI tools have a similar way of working. Despite that, it’s important to familiarize yourself with the CircleCI ecosystem, so you understand the example code snippets later in this post.

Steps

Steps are the fundamental building blocks of the workflows. Most of the time, they consist of executable bash commands. Other times, it’s possible to use predefined steps provided by CircleCI, covering common use cases – for example, checking out code from a repository or handling pipeline cache.

Jobs

Jobs have two configuration pieces. First, they need a list of steps. These steps are run as a single unit, in order, when a job is executed. Second, executing a job requires an executor. CircleCI supports many platforms – Docker, Windows, macOS, and virtual machines.


#...
jobs:
 build: # Job name
   docker: # Job executor
     - image: 
       auth:
         username: mydockerhub-user
         password: $DOCKERHUB_PASSWORD  # context / project UI env-var reference
   steps:
     - checkout # Special step to checkout your source code
     - run: # Run step to execute commands, see
         # circleci.com/docs/configuration-reference/#run
         name: Running tests
         command: make test # executable command run in
         # non-login shell with /bin/bash -eo pipefail option
         # by default.
#...

Job and steps example

Workflows

Workflows sit at the top of the hierarchy. They consist of jobs, each of which has a list of steps.

To mitigate long execution times, jobs execute in different ways – sequentially, parallel, or manually. We take a more detailed look at optimization techniques later on.


#...
workflows:
 build_and_test: # name of your workflow
   jobs:
     - build1 # predefined job name from the jobs section
     - build2:
         requires:
           - build1 # wait for build1 job to complete successfully before starting
           # see circleci.com/docs/workflows/ for more examples.
     - build3:
         requires:
           - build1 # wait for build1 job to complete successfully before starting

Workflow example

Orbs

Workflows are implemented from the ground up. To avoid duplicate configurations, orbs come into play. Orbs provide reusable snippets of code, covering common use cases like creating and uploading a Docker image to Docker Hub. CircleCI provides an orbs repository that’s easy to find and use, and anyone can browse.

config elements orbs

source: circleci.com

Context

It’s well-known that pipeline tools provide ways to create configuration options. CircleCI offers this via contexts that hold a set of environment variables. These values are available across projects in your organization.

Let’s say you have a secured API that all services access. You can build a context and configure the API key as an environment variable, and each team references the context in their pipelines. At runtime, CircleCI injects the variables into the workflow jobs.

Pipeline optimization

After a while, it’s common for build application times to increase. With that in mind, CircleCI provides a list of features to make processes faster and more straightforward.

Concurrency

In CircleCI terms, concurrency refers to job execution. If there are many jobs in a workflow and they aren’t dependent on each other, they’ll run concurrently. Note: there’s a soft limit on this.

From the previous section, we know that jobs can have various executor environments. On top of that,you can fine-tune the executors based on CPU and memory needs. It’s possible to influence those factors by providing a resource class like small, medium, or large.

The limitation factor comes into play if you’re using a lot of CPU and memory power. If that happens, CircleCI limits concurrency to make sure the platform stays stable. But fear not, CircleCI’s premium support can raise the limit for you.

As a rule of thumb,don’t define requirements for each job in a workflow if possible, so they run concurrently.

Parallelism

This is interpreted at job level, indicating how many executors are needed to complete the specific job. We use this feature to spread out test executions – it’s less complicated than it sounds.

CircleCI has a command line tool, helping slice test cases into more manageable piles. For splitting logic, use one of three methods: time, name, or file size.

Data persistence

During builds, many files are generated and handled. There are dependencies, test reports, compiled code, and images – to name a few. At some point, you’ll likely need to access and use that data, so you need a place where you can store it. CircleCI offers the following ways:

  1. Cache – store a file or directory with a key. Jobs save to the cache or retrieve data from it via the key. For example, use this to speed up dependency managers.
  2. Workspace – these are workflow-aware storage spaces. Use them to pass generated version numbers between jobs or compiled code that you need later on.
  3. Artifacts – a storage class that persists data after a workflow completes. These are useful for storing jar files and test reports.

Docker Layer Caching

Docker is regularly used in pipelines. For example, we use Docker executors for jobs or have steps that generate some kind of Docker image. This CircleCI feature caches common Docker layers. They persist through workflow runs and speed up the build process of the images.

Extra features of CircleCI

To round up our introduction of CircleCI, we have a few more features to mention.

  1. Branch-specific configuration – it’s possible to exclude or include jobs based on the current branch the pipeline is currently executing.
  2. Conditional execution of steps - you can write conditions, and if they test true, the step is executed. You can see examples if you look at the orb repository, where you can view the source code.
  3. Commands – these define a sequence of steps and are a useful feature to make codes reusable.
  4. Parameters – defined under jobs, commands, and executors, parameters are best used with conditions to define dynamic workflows.

Examples of CircleCI for Java

So far, we’ve covered theory. Now, let’s outline code examples.

To create a CircleCI workflow, you need to define the following:

  1. List of orbs you’d like to use
  2. List of defined jobs, if something isn’t supported by an orb
  3. List of workflows that will execute the jobs

Put this information in the yml file .circleci/config.yml of your project and CircleCI will pick it up.

Gradle simple workflow


version: 2.1

orbs:
  gradle: circleci/gradle@3.0.0
  aws-ecr: circleci/aws-ecr@8.1.2
  
workflows:
  build-and-push:
    jobs:
      - gradle/run:                         # <--------------- 1.
          executor:
            name: gradle/default
            tag: 17.0.4
          post-steps:                       # <--------------- 2.
            - persist_to_workspace:
                root: build
                paths:
                  - libs
      - aws-ecr/build-and-push-image:       # <--------------- 3.
          repo: ng-java-circle-test-backend
          attach-workspace: true
          workspace-root: ./build
          tag: ${CIRCLE_BRANCH}.${CIRCLE_SHA1}
          requires:
            - gradle/run

Key highlights:

  1. The above is a job from the official Gradle orb. By default, it’ll run the build command. It also uses a cache to store the dependencies. We made a small change to the default settings, because the base job is currently configured to use Java 13. Using the tag parameter, we changed it to a newer version that’s present in the CircleCI image repo.
  2. We defined pre- and post-steps for the job to add new functionality. Here, we wanted to carry the jar file to the next job so we could push it to the Elastic Container Registry (ECR). To make this happen, we also used the workspace feature.
  3. This is a job from the AWS-ECR orb. We only needed simple parameters to ensure the jar file was available from the previous job. For tagging, we used CircleCI’s built-in environment variables. Beneath the surface, the orb uses extra configurations – a list of environment variables – that we need to communicate with AWS:

  1. AWS_ACCESS_KEY_ID
  2. AWS_SECRET_ACCESS_KEY
  3. AWS_ECR_REGISTRY_ID
  4. AWS_REGION

Gradle parallel workflow


version: 2.1

jobs:
  build-with-caches-and-parallel:
    parallelism: 2
    environment:
      GRADLE_OPTS: "-Dorg.gradle.daemon=false -Dorg.gradle.workers.max=2"
    docker:
      - image: cimg/openjdk:17.0.4
    steps:
      - checkout
      - restore_cache:
          key: v1-gradle-wrapper-
      - restore_cache:
          key: v1-gradle-cache-
      - run:
          name: Run tests in parallel
          command: |
            cd src/test/java
            # Get list of classnames of tests that should run on this node
            CLASSNAMES=$(circleci tests glob "**/*.java" \
              | cut -c 1- | sed 's@/@.@g' \
              | sed 's/.\{5\}$//' \
              | circleci tests split --split-by=timings --timings-type=classname)
            cd ../../..
            # Format the arguments to "./gradlew test"
            GRADLE_ARGS=$(echo $CLASSNAMES | awk '{for (i=1; i<=NF; i++) print "--tests",$i}')
            echo "Prepared arguments for Gradle: $GRADLE_ARGS"
            ./gradlew test $GRADLE_ARGS
      - save_cache:
          paths:
            - ~/.gradle/wrapper
          key: v1-gradle-wrapper-
      - save_cache:
          paths:
            - ~/.gradle/caches
          key: v1-gradle-cache-
      - run:
          name: Assemble JAR
          command: |
            if [ "$CIRCLE_NODE_INDEX" == 0 ]; then
              ./gradlew bootJar
            fi
      - run:
          name: Create build folder on all nodes
          command: |
            mkdir -p ./build/libs
      - persist_to_workspace:
          root: build
          paths:
            - libs

workflows:
  build-and-push-with-caches-and-parallel:
    jobs:
      - build-with-caches-and-parallel
      - aws-ecr/build-and-push-image:
          repo: ng-java-circle-test-backend
          attach-workspace: true
          workspace-root: ./build/libs
          tag: ${CIRCLE_BRANCH}.${CIRCLE_SHA1}
          executor:                                   # <------- Extra Note
            name: aws-ecr/default
            use-docker-layer-caching: true
          requires:
            - build-with-caches-and-parallel

We copied this example from the official Java guide, and it has most of the steps described.

We have one extra note: The orbs that include jobs have an executor assignment. As shown in the example, it’s possible to change that if you require something else. In this case, we configured it to use the Docker Layer Caching feature.

Terraform simple workflow


version: 2.1

orbs:
  terraform: circleci/terraform@3.1.0

common:                                                 # <------------ 1.
  terraform-common: &terraform-common
    path: terraform
    tag: 1.2.8

workflows:
  deploy_infrastructure:
    jobs:
      - terraform/validate:
          <<: *terraform-common
          checkout: true
      - terraform/plan:                                 # <------------ 2.
          <<: *terraform-common
          var_file: vars/test.tfvars
          persist-workspace: true
          requires:
            - terraform/validate
      - wait-for-apply-approval:
          type: approval
          filters:                                      # <------------ 3.
            branches:
              only: main
          requires:
            - terraform/plan
      - terraform/apply:
          <<: *terraform-common
          attach-workspace: true
          filters:
            branches:
              only: main
          requires:
            - wait-for-apply-approval

We should automate infrastructure-related changes as well. Using Terraform, we can manage that via code. Here’s an example pipeline:

Key highlights:

  1. Example for YAML code reuse. The section is anchored, which can be referenced later.
  2. These are jobs from the Terraform orb. We added common steps to the config file, and extended them where needed. Here, Terraform uses the provided var file to fill up the defined variables. Once it has a plan, it persists to the workspace, ensuring the apply step executes what we approved.
  3. Branch filtering in action. We only deploy the infrastructure from the main branch; for every other branch, the last two steps aren’t visible.

Netguru best practices for CircleCI

CircleCI is the CI/CD solution we encourage developers to use at Netguru. Many engineering teams use it for internal and client projects.

We try to adhere to the following guidelines:

  • Use a greater number of smaller jobs
  • Identify where parallelism could help
  • Make code reusable as much as possible
  • Use orbs if appropriate
  • Integrate Rollbar to track deployment status
  • Integrate CodeClimate for test reports

CircleCI: the way forward

This guide will help you get started more easily with Java projects using the CircleCI Continuous Integration tool. We answered the question “what is CircleCI?” and outlined how it helps development teams create an efficient workflow for their daily tasks. If you’d like more information, feel free to get in touch.

Photo of Dániel Juhász

More posts by this author

Dániel Juhász

Java Developer at Netguru
Software development services  Design, build and scale software solutions Discover how!

Read more on our Blog

Check out the knowledge base collected and distilled by experienced professionals.

We're Netguru

At Netguru we specialize in designing, building, shipping and scaling beautiful, usable products with blazing-fast efficiency.

Let's talk business