An introduction to Highway – Our New Build System

Not exactly. We’ve prepared a new tool for iOS developers to make the CI/CD integration process smooth. Why have we done that? After all, there is Fastlane, which does a really awesome job, and we also have tools for CI/CD like Jenkins, Bitrise, TeamCity, and many others. However, we have found that CI/CD can still be greatly simplified and improved. This article is about a solution that we developed internally, have been testing in our projects for quite a while, and want to share with the world.

Let’s start from the beginning...

At Netguru we use Bitrise as our CI/CD. It’s really great and easy to configure. All you need to do is to create a YAML file with configuration or set up a workflow from the web panel. Sometimes we also use Fastlane to automate development flows. While both of these tools are great, we’ve found out that there are some pros and cons.

First of all, Bitrise uses YAML files for configuration, where we can set up trigger maps, workflows, variables, and basically everything else that happens during CI. This file is super-readable and easy to use even for beginners. We can also run Bitrise locally by using CLI, which is doing a great job. You can store a YAML file in a repository or use their GUI config; there is even a free plan. However, there are also some cons. What if something happens and we will need to migrate from Bitrise? We will have to configure a new tool from scratch. Bitrise was also built around Git - it was not designed for arbitrary scripting. Instead, it was designed as a system for CI and made trade-offs to achieve that. There are also too many ways to do the same thing, which is both a blessing and a curse.

The second thing - Fastlane. There’s probably no need to explain how it works. Let’s just focus on the things that it makes easier and those that are hard. Fast support, hundreds of steps and plugins, weekly releases, and a large team of Google employees working full-time to ease the pain of hundreds of thousands of iOS and macOS developers - it’s really awesome. We can also run Fastlane locally and on any CI server - all we need is a Mac with Ruby installed. Continuous delivery with this service is possible - you may not have known, but Bitrise and Microsoft App Center use it under the hood.

So, are there any cons? Unfortunately, yes.

While Fastlane is a powerful tool, it consists of imperative instructions. Easy to use at first, but not so maintainable after a while. It requires advanced Ruby skills and this can be a huge blocker for teams like ours. While knowledge of additional technologies is always appreciated, it shouldn’t be a requirement. It lazily defers errors. Failures after 30 mins because of a typo? Standard. It carries a huge compatibility burden. Fastfile is just an eval'd Ruby file. This means that Fastlane just can’t break backwards compatibility. That is clearly visible if you dive into its codebase.

That’s why we decided to combine what’s best about Bitrise and Fastlane into one tool.

Solution

Highway is nothing but a build system built on top of Fastlane.

• It prefers declarative configuration over convenience and takes advantage of the library of steps provided by Fastlane.
• Reduces feedback loops - provides information faster and in more detail so that you don't waste time, especially when integrated with Danger.
• Uses a very simple YAML file for configuration.
• Can be used for scripting; run it locally and on Continuous Integration.
• Allows centralization of configuration. Provides first-party support to set default values and behaviors across projects. In the future, adding e.g. Carthage-Rome to our projects will be a matter of editing a central “default” configuration file in some repository.

Highway can do literally anything.

Infrastructure

As mentioned above, Highway was built on top of Fastlane and we benefit from it, but it still has its own infrastructure. Highway consists of two fundamental elements - compiler and runtime.

Compiler

The first step is to make a syntactic analysis of our Highwayfile, check whether it contains valid keys and values, and then produce an abstract syntax tree. With this output we’re going to the semantic analysis phase. Here we are going to do a few things starting from checking stage names and variable references to resolving variables, steps, and parameters. This phase outputs an abstract semantic tree. The last phase of the compiler’s operation is manifest generation. We take the output from semantic analysis and produce a manifest which will include a resolved preset with steps to be executed by Highway. As you can see, the last two phases strongly rely on the steps library.

Runtime

Runner is a class that is responsible for evaluating the invocation parameters, then validating and running the step invocations. We also have Context, which is a class responsible for maintaining the runtime context between step invocations and allowing steps to access runtimes of both Fastlane and Highway.

Configuration

As mentioned, Highway uses a YAML file for configuration. That means you don’t need to know any language like Ruby to configure it. It's a super easy and readable form of configuration which is friendly even for beginners.

Each Highwayfile can be configured in different ways depending on the version used. Thanks to that we will be able to add new features in the future without any problems. Version is marked as a single integer number - it’s not semantic - so it’ll be much easier to upgrade without thinking about if it should be numbered 1.1.3 or maybe 1.2.0. Each version can contain different features and stages, but let’s focus on the current one.

The structure looks like this:

variables:
    <preset>:
        <variable>:

bootstrap:
    <preset>:
        <step>:

test:
    <preset>:
        <step>:

deploy:
    <preset>:
        <step>:

report:
    <preset>:
        <step>:

First of all, we need to declare the version of the Highwayfile. Next, we can add our variables divided into presets. A preset represents a specific workflow (unit tests build, production build, etc.) which you've probably used before in tools like Bitrise or Github Actions - you can use `default` if you would like to have variables that will be available for all steps in your Highwayfile.

This version also supports 4 different stages - bootstrap, test, deploy, and report. You can add any step here, but Highway will run it in this order starting from bootstrap.

This is what a complete configuration looks like:

# Highway configuration file version.
version: 1

# Variables available for presets.
variables:
    default:
        XCODEBUILD_SCHEME: Development
        XCODEBUILD_PROJECT: ./Project.xcworkspace
    staging:
        XCODEBUILD_SCHEME: Staging
    release:
        XCODEBUILD_SCHEME: Production

# Bootstrap stage.
bootstrap:
    default:
        - carthage:
            command: "bootstrap"
            platforms:
                - ios
        - sh:
            command: "cp .env.sample .env"
        - cocoapods:
            command: "install"

# Test stage.
test:
    default:
        - xcode_test:
            project: $(XCODEBUILD_PROJECT)
            scheme: $(XCODEBUILD_SCHEME)

# Deploy stage.
deploy:
    staging:
        - xcode_archive:
            project: $(XCODEBUILD_PROJECT)
            scheme: $(XCODEBUILD_SCHEME)
            method: "enterprise"
        - appcenter:
            api_token: $(ENV:APPCENTER_API_TOKEN)
            owner_name: $(ENV:APPCENTER_ORG_NAME)
            app_name: $(ENV:APPCENTER_APP_NAME)
            distribution_group: $(ENV:APPCENTER_DISTRIBUTION_GROUP)
            notify: false

# Report stage.
report:
  default:
    - copy_artifacts:
        path: $(ENV:BITRISE_DEPLOY_DIR)
    - slack:
        webhook: $(ENV:SLACK_WEBHOOK_URL)
        channel: "#notify-project-xd"

What next? How to run Highway? You only need the simple command `fastlane highway` and that’s it - Highway will take care of it.

Real-life case

It’s easier to learn new things on an example project, right? We’ve been running Highway in our projects for some time and fortunately one of these projects is open-source! Have you heard about our BabyGuard application? You can check the code here! Highway has been running there for a few months and now it’s even configured for GitHub Actions and CircleCI. You can check how the Highwayfile looks like when configured for Bitrise, GH Actions and CircleCI at the same time. You can also refer to the Highway documentation if you would like to check how the configuration file looks for CI.

According to Murphy's law, there will be a time when your CI will go down during a Friday deploy (this happened in one of our projects in the past). If your configuration is tightly coupled with the selected CI architecture, you will end up trying to configure all the certificate provisioning profiles locally, working under a huge pressure, in conditions very prone to error. If you are lucky you will get it right in a few tries (we all saw weird Xcode errors). Imagine you have Highway integrated in your project. With the approach we implemented in BabyGuard, all you need to do is set a proper build number and run fastlane Highway. In case of any problem, you will get feedback much sooner.

You can find Highway here.

Photo by Markus Spiske on Unsplash