Automated tests for a Swift CLI tool with Cucumber

As you might already know, I’m devoting part of my free time to build Tuist, a command line tool that helps Swift developers maintain their Xcode projects regardless of their scale. Since we added the first Swift file to the project, having a good test suite has been one of our key design principles to ensure that features do what they are supposed to do, and that new versions are backward compatible (unless it’s impossible to achieve so). If companies and developers start using Tuist in a daily basis, the last thing that we want is disturbing their work as a consequence of a buggy or breaking update.

There’s nothing more annoying than not being able to do your work because the tool that you are using doesn’t work as expected.

The project initially contained a target with decent list of unit tests. This allowed us to test each piece of code in isolation but didn’t bring enough confidence for us to release new versions of Tuist. If unit tests were not enough, what else could we do? We could have adopted an analogic approach. Before releasing a new version, we could have asked users to try the next version before going out into wild. Beta testing is a tedious process, requires an effort on the users side, and slows down the release due to all the back and forth that it entails.

The approach that we decided to take, and about which I’d like to talk, is based on the Ruby BDD testing framework Cucumber. Before I jump into details about why we made that choice, I want to show you an example of a test run output:

As you can see in the example above, the steps of the scenario that we are testing can be read as if you were reading a story. At the end of the day, they are user stories. We describe the scenarios as a set of steps that are in fact sentences, and Cucumber maps those steps into Ruby code that gets executed. Cucumber offers the expressiveness that XCTest doesn’t. The latter is, in my opinion, more suitable for unit tests where having a more verbose API and output makes more sense.

Before introducing Cucumber to the project, I was a big skeptical about adding Ruby code to the mix. I’m comfortable writing code Ruby but what about contributors to the project? It turns out that people can quickly understand how Cucumber works. Whenever we see a use case with potential to be tested with include a fixture project that we use to run the tests on. We keep track of all the fixtures on this README where each fixture includes a description of what the project is like.

Here are some examples of fixtures that we run automated tests on:

• ios_app_with_static_libraries: This application provides a top level application with two static library dependencies. The first static library dependency has another static library dependency so that we are able to test how tuist handles the transitiveness of the static libraries in the linked frameworks of the main app.
• app_with_frameworks: Slightly more complicated project consists of an iOS app and few frameworks.

The automated tests see the Swift Package Manager as a tool that builds the object under testing, the Tuist binary. The only interactions that they can have with Tuist is through the CLI, the standard output and error, and the generated output artifacts. Something like this can also be achieved defining a tests target with the Swift Package Manager, but I find it odd that the test runners, SwiftPM or Xcode, run tests that depend on themselves. When we describe the tests, we put ourselves in someone elses shoes:

I’m a user that have just installed Tuist and I’d like to initialize a project.

I’d expect to be able initialize the project with Tuist and get an Xcode project with a target that I can build. We’d describe that scenario in Cucumber like this:

Feature: Initialize a new project using Tuist

  Scenario: The project is a compilable macOS application
    Given that tuist is available
    And I have a working directory
    When I initialize a macos application named Test
    Then tuist generates the project
    Then I should be able to build the scheme Test

Imagine that we introduce a change in the project that doesn’t break the project generation, for which we already have unit tests, but for some reason the generated Xcode project cannot be built. Do you think it’d be a good experience for the user? I doubt so. Luckily our automated test would fail immediately and raise a flag.

I believe being able to release new software versions with confidence is crucial to move fast without breaking things. When developers use your software, they trust the software and the people behind it. They use it because it brings value to them and they’d like to continue using it if it continues to work reliably. If we are not able to meet that expectation and release with confidence, we are putting ourself at the risk of breaking the trust between our users and us.

This is just an example of how we are bringing that confidence to Tuist, and it’s certainly not the only one. The next time you merge a PR or release a new version ask yourself if you feel confident enough. If you don’t, you’d better adjust things in your project.