iOS testing

Unit testing

• Use vanilla XCTest testing framework for Unit testing. We found that third party BDD frameworks does not integrate well with Xcode and lacks stability after a new Xcode version is released.
• Match tests folder structure to app structure. For example, App/Data/Models/Client.swift tests are located in AppTests/Data/Models/ClientTests.swift.

UI Testing

• Use Xcode UI Testing framework for user interface testing.
• Record tests with Xcode and clean up the generated code.
• Run UI tests on both iPhone and iPad simulators.
• Do not stub network requests. We run our tests against dedicated testing environment. This ensures that our API works correctly.
• All test cases inherit from BaseUITests class which handles initialization and defines helpers.
• Use MagicalRecord’s CoreDataStackWithInMemoryStore to clean up the database before running a test.
• Clear NSUserDefaults after the app starts.
• Define helper methods (waitForElementToAppear, goBack, isIPad, etc) in BaseUITests.
• Create methods for steps common to many tests (login, signOut, etc) in BaseUITests.
• Create methods for common steps to one test case (selectClient, etc) in the same test case where they are used.
• Use waitForElementToAppear helper when expecting UI to change after a network request (defaults to 60 seconds).
• Use accessibility identifiers in UI elements to find them instead of using their position in the element tree.

Disabling features in UI tests

• Disable features which are not being tested in a test case. It saves time when running specs and sometimes makes it easier to write a test case.
• We have AppSettings struct defined in tests to allow disabling features. All features are enabled by default.

struct AppSettings {
    var loginEnabled = true
    var welcomeEnabled = true

    func toLaunchEnvironment() -> [String: String] {
        return [
            "loginEnabled" = loginEnabled.stringValue,
            "welcomeEnabled" = welcomeEnabled.stringValue
        ]
    }
}

• Call launchApp with AppSettings before each test.

  override func setUp() {
    super.setUp()
  
    let settings = AppSettings(loginEnabled: false, welcomeEnabled: true)
    launchApp(settings: settings)
  }
  

• Before the app is launched when running a test, app settings are converted to launch arguments and passed to app runtime.

    func launchApp(settings: AppSettings) {
        app = XCUIApplication()
        XCUIDevice.sharedDevice().orientation = .Portrait
        app.launchEnvironment = settings.toLaunchEnvironment()
        app.launch()
        waitForAppLaunch()
    }

• In the app delegate, launch environment values are parsed and features are disabled by settings variables on a shared instance of Settings.

for (key, value) in NSProcessInfo.processInfo().environment {
    switch key {
    case "loginEnabled":
        Settings.sharedInstance.surveyEnabled = NSString(string: value).boolValue
    case "welcomeEnabled":
        Settings.sharedInstance.welcomeEnabled = NSString(string: value).boolValue
    default:
        break
    }
}

Running tests

• We have a dedicated Jenkins slave running macOS to run iOS tests.
• Use fastlane scan tool to run tests on both iPhone and iPad simulators.

desc "Runs all the tests"
lane :test do
  scan(scheme: 'PhysiApp', devices: ["iPhone 7", "iPad Air 2"])
end

• Use ios-deploy tool to install the newest version of the app (from testing branch) on a dedicated iOS device for manual testing.

desc "Install on local device"
lane :install do
  gym(scheme: 'PhysiApp', output_directory: 'build', output_name: 'PhysiApp.ipa')
  install_on_device(device_id: 'abc123', ipa: './build/PhysiApp.ipa')
end

Build Configuration

• We have 3 build configurations: Debug, Testing, and Release. Having different configurations allows us to pass Swift compiler flags (DEBUG, TESTING), which we convert to ProjectConfiguration enum. Then we can write conditional code based on ProjectConfiguration.currentConfiguration.

enum ProjectConfiguration {
    case Debug
    case Testing
    case Release
}

static var currentConfiguration: ProjectConfiguration {
    # if DEBUG
        return .Debug
    # elseif TESTING
        return .Testing
    # else
        return .Release
    # endif
}

static var APIURLString: String {
    switch currentConfiguration {
    case .Debug:
        return "https://staging.example.com"
    case .Testing:
        return "https://test.example.com"
    case .Release:
        return "https://example.com"
    }
}