Blazingly Fast Flutter Driver Tests

Photo by Djim Loic on Unsplash

TL;DR: Ever wanted to speed up your Flutter Driver Tests by 750%?

Here is how we did it.

Flutter has two approaches to testing user interfaces — Widget Testing and Flutter Driver Testing.

Widget Testing has more in common with Unit Testing — it does not need to display UI to test it, whereas Flutter Driver is closer to UI (similar to Appium) as it interacts with rendered components.

In your projects, you will be using Widget Testing much more as it’s more robust and faster. Using Flutter Driver tests has benefits:

• You see how your screen looks like — you can verify if everything renders properly on the different screen sizes and in different languages
• You can create png screenshots of the tests and use them for regression testing
• You can test end-to-end — backend calls work as in a normal application. In Widget Test Fake Futures are used and that prevents you from testing your UI together with the backend

There is still a couple of drawbacks of Flutter Driver tests:

They require a real device to run on

In comparison to Widget Tests, which basically run like Unit Tests (headlessly on a development machine), Flutter Driver Tests require a device that can render UI. As mobile developers, we know that maintaining an emulator or a real device is quite a pain.

They are much slower than Widget Tests

Driver Tests build the whole application to run the tests. After the application is launched, the tests need to connect to Dart VM to be able to interact with the application — that takes around 10 seconds for each test file.

It’s hard to inject different setup from test file into the application file

The application file and test file are separated. For every test file (main_text.dart) you need one application file (main.dart).

It’s hard from the test to change the setup of the application.

This separation is even more visible when trying to use any class from dart:ui package: in the application file, you can obviously use Widgets.

In _test.dart file, you cannot use anything from dart:ui(that includes even Locale class) because in runtime the test will instantly fail.

Each test in _test.dart file depends on the previous test

Because you cannot easily change the setup of the application file, you also cannot ‘reset’ the state after each test.

If the first test navigates to a certain page, the next test will start on that page.

Running one test file at a time

In flutter drive command there is no option to specify multiple files to run — only one at a time.

Hard to debug failing test

When a test fails it will only say it did not find an element. If you are lucky you can see on the screen what’s wrong but the majority of issues won’t so obvious.

With so many challenges I am not surprised that people do not use Flutter Driver Test so often.

I really believe that if they were resolved, many more people would start using those

…so let’s fix them!

Fixing: Hard to debug failing test

You cannot debug an application when the test is running, but you can repeat the same steps of the test when you are debugging the main.dart file.

If a test in main_test.dart fails (eg HTTP request is not properly mocked), then just start the main.dart as a normal application in the debug mode.

First, you need to add -t (or — target) parameter in the IntelliJ settings:

Then press the debug button — at this point just reproduce your test’s steps.

IMPORTANT: If you need to enter manually text into TextFields, remember to comment out enableFlutterDriverExtension from main.dart.

Fixing: They require a real device to run on

Now Flutter Desktop gets handy — you can build your application for Windows/macOS/Linux so why not running UI tests against desktop as well.

You won’t be able to run every (testing device-specific components like WebView Widget), but from my experience, those tests are around 5% of all the Flutter Driver Tests.

Run main.dart file as a normal desktop application:

flutter run -d windows

After the application launches you will see the message:

An Observatory debugger and profiler on Windows is available at: http://127.0.0.1:65521/vBWe3pt2_0k=/

The URL is here important as the tests require it to connect to the application

Run the main_test_dart and pass that link to it:

dart main_test.dart http://127.0.0.1:65521/vBWe3pt2_0k=/

dart command can run any file that has main() function.

Within the main_test.dart file you need to extract that link form args that you receive in main(List<String> args) and pass it as dartVmServiceUrl to FlutterDriver.connect .

And done — the test is running on your desktop! Or even docker!

Fixing: Running one test file at a time

From the previous fix we’ve learned that we need to do three things to run a single file:

1. Run main.dart with flutter run command
2. Wait for the app to start and copy the Dart VM URL
3. Run main_test.dart with given URL

To run multiple files you just need a script that:

• Looks up all the files ending with _test.dart in a given folder that are the tests files
• Looks up files that have the same name as the tests but without _test.dart — those are the application files
• Runs flutter run command for each application file sequentially and observe the console output for is available at: (http://.*/) match and copy it
• Runs dart <test file> <dart vm url> on each file

Now you have automated running multiple test files.

Fixing: They are much slower than Widget Tests

At this point, the test from a single file runs quite quickly but when we start each test file we need to recompile the application and reconnect to the Dart VM which can take an additional 20 seconds per file.

To improve this we simply should leverage Flutter hot restart.

This includes two things:

Communicate from main_test.dart that main.dart should perform a hot restart

This can be achieved with FlutterDriver::requestData method that sends a message to main.dart. Inside main.dart you will receive this message in enableFlutterDriverExtension handler:

enableFlutterDriverExtension(
  handler: (request) async {
    // Here will be the message
  },
);

Perform hot restart on main.dart side

To do that, you need to wrap your main application in a widget that will reinsert your application’s widget into the widget tree, I use streams for the communication:

Done. Now we need to run all our tests from a single _test.dart file:

import ‘main1_test.dart’ as main_file_1;
import ‘main2_test.dart’ as main_file_2;
import ‘main3_test.dart’ as main_file_3;
void main(List<String> args) {
  main_file_1.main(args);
  main_file_2.main(args);
  main_file_3.main(args);
}

Fixing: It’s hard to inject different setup from test file into the application file

Now when we can communicate with the app we need to send serialized configuration using FlutterDriver::requestData method.

I would recommend having a configuration class that can be passed to your MyApplication widget.

For me, this config contains the initial route, page’s arguments or even it specifies if the application should fake or real HTTP requests.

Fixed: Each test in _test file depends on the previous test

With the ability to send a restart request with a specific configuration, we can restart the application between each test:

setUp(() async {
  await restart(
    driver,
    config: const Configuration(
      route: Routes.curves,
      repeatAnimations: false,
    ),
  );
});
test(‘shows curves’, () async {
  await driver.waitFor(find.byType(‘CurvesPage’));
});
test(‘scroll’, () async {
  await driver.scroll(find.byType(‘CurvesPage’), 0, -400,
  const Duration(milliseconds: 100));
  await driver.waitFor(find.byType(‘CurvesSection’));
}, retry: 1);

With this change, every test has a fresh start.

If your tests are flaky, you can now add retry property to rerun tests if it fails.

Final Result

Here you can find an example application and run Flutter Driver Tests.

Command-line Tool

We’ve created a package + command-line tool so that you do not need to write all this boilerplate code yourself.

Final Thoughts

Before the injection of configuration and hot restart, our Flutter Driver Tests took 15 minutes for around 150 tests. After the switch, they take around 30 seconds on a local machine and 2 minutes on dockerized CI.

Personally, I think that it’s amazing to have access to the technology that allows you to run hundreds of UI tests on every commit you push.

Flutter Community