The New Flutter Integration Tests — Are They any Good?

The current Flutter Driver tests will be deprecated and it’s recommended to move to the new Integration Tests.

In this post let’s talk about what are the new tests good at and what are they still lacking.

The Positives

+ Same API

The API does not only look but is the same that is used by widget tests — only the deep internals are different.

Thanks to this, on my first try, when trying to run 20 random Widget tests, 15 run out of the box without modifications.

+ Runs on device

With widget tests to verify why some tests are not working you just print with debugDumpApp the widget tree— if you are a beginner that widget tree is terrifying due to its size.

With the new tests, you can see what is an issue and if you are new to tests in Flutter that might be a great help.

+ Referencing Flutter classes

In Flutter Driver tests, you could not reference anything from Flutter, even if you try to import package:flutter/widget.dart file, your tests will fail and if you do not know why it is happening, good luck resolving that.

Thankfully now you can reference any widget in your tests and you do not need to find widgets by string, eg “SizedBox” but by actual type — so many times I made a typo and was wondering why the test is failing.

+ No need for connection to desktop

In the Flutter Driver, your tests were actually running on your desktop and communicating with the application on the tested device.

That has some good sides but the downside was that running your tests on device farms was really hard to do.

Now the tests and the application is packaged together and only that ipa/apk is enough to run those tests on device farms like Firebase Test Lab.

+ Independent Tests

In Flutter Driver all the tests started where the previous finished — if you navigated to a new page in one test, in the next one you would still be on that page.
In Integration Tests, tests are independent and that makes writing them much easier.

The Negatives

- No Virtual Time:

The virtual time you can manipulate at will — Widget tests support virtual time by default and if you have a delay in your code eg Future<void>.delayed(const Duration(seconds: 10) or long-running animation, it will instantly finish.

New Integration Tests do not use virtual time and they will have to wait for the time to elapse.

Here is a test that in Widget Tests will finish within milliseconds and in Integration Tests will take at least 10 seconds:

One of the ways to make an Integration Test that uses Future<void>.delayedfaster, is to mock the time — instead of using Future<void>.delayed directly, use injection to inject your own time provider that in real application uses Future<void>.delayed but in tests, you would inject mocked version of it that returns a future that finishes instantly

To speed up animations just use timeDilationproperty from package:flutter/scheduler.dart in your test.

It will speed up all your animations — just remember if you overdo it, your tests might be unstable.

- No Screenshots

Before the tests were run on the desktop but the app was running on the devices (now everything is run on the device).

Making screenshots was easy that way — the screenshot was made on the device and then transferred to the desktop.

Now, making screenshots is not yet supported (the Flutter team is working on it).

If you are testing desktop application, you can still make screenshots:

- End to End testing

By End To End (e2e) testing, I mean that you do not mock your http requests and rely on a hopefully stable staging environment. The mocked tests should be the majority of your tests but having some e2e test is also valuable.

In Flutter Driver implementation, the tests will wait for an element or will timeout.

With Integration Tests, if you do not have any running animation during the async operation, the tests will instantly fail.

My suggestion would be to add loading animation during async call — then using pumpAndSettle will wait for the animation to finish.

Don’t want to add animation? You need to check if the element is already there by polling:

This code will wait 10 times one second and stop waiting earlier if the Done text appears.

- Need to be run separately

The command for Integration Tests does not allow us to run all test files from a folder — the way we are used to in Unit and Widget tests. For now, you can run one file at a time:

flutter drive \
  --driver=test_driver/integration_test.dart \
  --target=integration_test/main_test.dart 

The solution to it is to put all the calls to your tests inside one test file:

You can do it manually, you can write a script that scans a directory for tests and updates this file or you can use my package does exactly that.

Conclusion

Should you migrate all your tests now?

Not necessarily — the new tests are still not as stable as the previous ones (at least for us) but they are the future of UI testing in Flutter.

My recommendation is to start writing the new Integration Tests instead of the old ones but keep your old tests until you are comfortable migrating them.

As goes for Widget tests, do not replace them with Integration Tests as they are still faster and more reliable due to virtual time.