Flutter integration tests stop running after certain time on AWS Device Farm iOS device

Question

I have a Flutter mobile app (iOS and Android) for which I'm writing integration tests and running them on Device Farm. I run the Android tests via Instrumentation and iOS' via XCTest UI. Android tests run fine but the iOS ones stop every time around the 6 minute mark. I have the job timeout set to the default 150 minutes and I have even tried to set -maximum-test-execution-time-allowance to 2400 seconds on the xcodebuild command on the test spec. Any idea what's causing the tests to stop? Below is the test spec output. I have around 12 tests but you can see only up to 9 are run.

Test Spec Output

```
[DEVICEFARM] Setting up your device. This usually takes 2-3 minutes.
[DEVICEFARM] ########### Start executing testspec ###########

[DEVICEFARM] ########### Entering phase install ###########

[DEVICEFARM] ########### Entering phase pre_test ###########

[DEVICEFARM] ########### Entering phase test ###########
 
[DeviceFarm] cd $DEVICEFARM_XCTEST_BUILD_DIRECTORY
[DeviceFarm] echo "Starting the XCTest UI test suite using xcodebuild"
Starting the XCTest UI test suite using xcodebuild
[DeviceFarm] echo "dir = "$DEVICEFARM_XCTEST_BUILD_DIRECTORY
dir = /tmp/devicefarm-workspace/execution-s2_9laj4/Productsv1q0_3qt
[DeviceFarm] ls
Debug-iphoneos
RunnerTests.xctestrun
application.xctestrun
build
[DeviceFarm] export DEVICEFARM_XCUITESTRUN_FILE="RunnerTests.xctestrun"
[DeviceFarm] xcodebuild test-without-building \
  -destination id=$DEVICEFARM_DEVICE_UDID \
  -xctestrun $DEVICEFARM_XCUITESTRUN_FILE \
  -derivedDataPath $DEVICEFARM_DERIVED_DATA_PATH \
  -maximum-test-execution-time-allowance 2400
Command line invocation:
    /Applications/Xcode_16.app/Contents/Developer/usr/bin/xcodebuild test-without-building -destination id=00008120-001E7D8E0E87C01E -xctestrun RunnerTests.xctestrun -derivedDataPath /tmp/derived_datadpr1v0ai -maximum-test-execution-time-allowance 2400

User defaults from command line:
    IDEDerivedDataPathOverride = /tmp/derived_datadpr1v0ai
    IDEPackageSupportUseBuiltinSCM = YES

2025-09-25 08:18:03.336 xcodebuild[910:6719]  DVTDevice: Error locating DeviceSupport directory using Optional("arm64e") or Optional("arm64e"): nilError
2025-09-25 18:18:04.683250+0300 Runner[801:110736] [Scene] CLIENT OF UIKIT REQUIRES UPDATE: This process does not adopt UIScene lifecycle. This will become an assert in a future version.
2025-09-25 18:18:04.709843+0300 Runner[801:110736] [UIFocus] FlutterView implements focusItemsInRect: - caching for linear focus movement is limited as long as this view is on screen.
2025-09-25 18:18:04.726761+0300 Runner[801:110736] fopen failed for data file: errno = 2 (No such file or directory)
2025-09-25 18:18:04.726784+0300 Runner[801:110736] Errors found! Invalidating cache...
2025-09-25 18:18:04.746191+0300 Runner[801:110736] fopen failed for data file: errno = 2 (No such file or directory)
2025-09-25 18:18:04.746220+0300 Runner[801:110736] Errors found! Invalidating cache...
2025-09-25 18:18:04.758178+0300 Runner[801:110736] Warning: Unable to create restoration in progress marker file
2025-09-25 18:18:04.771631+0300 Runner[801:111028] [[FirebaseCore]] 11.15.0 - [FirebaseCore][I-COR000005] No app has been configured yet.
2025-09-25 18:18:04.788562+0300 Runner[801:110736] [Firebase/Crashlytics] Version 11.15.0
2025-09-25 18:18:04.789775+0300 Runner[801:111030] [[FirebaseMessaging]] 11.15.0 - [FirebaseMessaging][I-FCM001000] FIRMessaging Remote Notifications proxy enabled, will swizzle remote notification receiver handlers. If you'd prefer to manually integrate Firebase Messaging, add "FirebaseAppDelegateProxyEnabled" to your Info.plist, and set it to NO. Follow the instructions at:
https://firebase.google.com/docs/cloud-messaging/ios/client#method_swizzling_in_firebase_messaging
to ensure proper integration.
2025-09-25 18:18:04.894448+0300 Runner[801:111030] [[FirebaseAnalytics]] 11.15.0 - [FirebaseAnalytics][I-ACS023007] Analytics v.11.15.0 started
2025-09-25 18:18:04.894728+0300 Runner[801:111030] [[FirebaseAnalytics]] 11.15.0 - [FirebaseAnalytics][I-ACS023008] To enable debug logging set the following application argument: -FIRAnalyticsDebugEnabled (see http://goo.gl/RfcP7r)
2025-09-25 18:18:04.894788+0300 Runner[801:111030] [[FirebaseAnalytics]] 11.15.0 - [FirebaseAnalytics][I-ACS044002] The AdSupport Framework is not currently linked. Some features will not function properly. Learn more at http://goo.gl/9vSsPb
2025-09-25 18:18:04.906305+0300 Runner[801:110736] no valid “aps-environment” entitlement string found for application
Twilio-iOS didFailToRegisterForRemoteNotificationsWithError => onFail
2025-09-25 18:18:04.906418+0300 Runner[801:110736] no valid “aps-environment” entitlement string found for application
2025-09-25 18:18:04.912358+0300 Runner[801:110736] flutter: 00:00 +0: Integration tests - Associate profile - Share associate contact card via copy link
2025-09-25 18:18:04.938342+0300 Runner[801:111067] [[FirebaseMessaging]] 11.15.0 - [FirebaseMessaging][I-FCM012002] Error in application:didFailToRegisterForRemoteNotificationsWithError: no valid “aps-environment” entitlement string found for application
2025-09-25 18:18:05.011614+0300 Runner[801:111026] [[FirebaseAnalytics]] 11.15.0 - [FirebaseAnalytics][I-ACS800023] No pending snapshot to activate. SDK name: app_measurement
2025-09-25 18:18:05.059848+0300 Runner[801:111065] [[FirebaseAnalytics]] 11.15.0 - [FirebaseAnalytics][I-ACS023012] Analytics collection enabled
2025-09-25 18:18:05.060375+0300 Runner[801:111065] [[FirebaseAnalytics]] 11.15.0 - [FirebaseAnalytics][I-ACS023220] Analytics screen reporting is enabled. Call Analytics.logEvent(AnalyticsEventScreenView, parameters: [...]) to log a screen view event. To disable automatic screen reporting, set the flag FirebaseAutomaticScreenReportingEnabled to NO (boolean) in the Info.plist
2025-09-25 18:18:06.847207+0300 Runner[801:111029] [[FirebaseCrashlytics]] 11.15.0 - [FirebaseCrashlytics][I-CLS000000] Warning: std::get_terminate is '_ZL20_XCTTerminateHandlerv' in '/System/Developer/Library/PrivateFrameworks/XCTestCore.framework/XCTestCore'
Twilio-iOS PluginHandler.handle => received create
Twilio-iOS TwilioConversationsPlugin.create => called
Twilio-iOS ChatListener.onConnectionStateChange => state is DISCONNECTED
Twilio-iOS ChatListener.onConnectionStateChange => state is CONNECTING
Twilio-iOS ChatListener.onClientSynchronization => state is STARTED
Twilio-iOS ChatListener.onConnectionStateChange => state is CONNECTED
Twilio-iOS ChatListener.onClientSynchronization => state is CHANNELS_COMPLETED
Twilio-iOS ChatListener.onClientSynchronization => state is COMPLETED
Twilio-iOS ChatListener.onUserSubscribed => user 'Optional("sales_associate_5891")'
Twilio-iOS TwilioConversationsPlugin.create => ChatClient.create onSuccess: myIdentity is 'sales_associate_5891'
Twilio-iOS ChatStreamHandler.onListen => Chat eventChannel attached
Twilio-iOS NotificationStreamHandler.onListen => Notification eventChannel attached
2025-09-25 18:18:47.303634+0300 Runner[801:110736] flutter: 00:42 +1: Integration tests - Associate profile - Reorder home screen lists
2025-09-25 18:19:18.927032+0300 Runner[801:110736] flutter: 01:14 +2: Integration tests - Client search - Search clients by name
2025-09-25 18:19:53.550283+0300 Runner[801:110736] flutter: 01:48 +3: Integration tests - Client search - Search clients by email
2025-09-25 18:20:33.863113+0300 Runner[801:110736] flutter: 02:28 +4: Integration tests - Client search - Search clients by phone number
2025-09-25 18:21:09.543002+0300 Runner[801:110736] flutter: 03:04 +5: Integration tests - Home > Plus menu - Create client via app
2025-09-25 18:21:41.857613+0300 Runner[801:110736] flutter: 03:36 +6: Integration tests - Home > Plus menu - Create manual list
2025-09-25 18:22:14.766964+0300 Runner[801:110736] flutter: 04:09 +7: Integration tests - Home > Plus menu - Create smart list
2025-09-25 18:22:48.778891+0300 Runner[801:110736] flutter: 04:43 +8: Integration tests - Home > Plus menu - Create Appointment
2025-09-25 18:23:32.312241+0300 Runner[801:110736] flutter: 05:27 +9: Integration tests - Home > Plus menu - Create Reminder

*** If you believe this error represents a bug, please attach the result bundle at /tmp/derived_datadpr1v0ai/Logs/Test/Test-RunnerTests-2025.09.25_08-17-42--0700.xcresult

2025-09-25 08:23:51.226 xcodebuild[910:6661] [MT] IDETestOperationsObserverDebug: 368.778 elapsed -- Testing started completed.
2025-09-25 08:23:51.226 xcodebuild[910:6661] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start
2025-09-25 08:23:51.226 xcodebuild[910:6661] [MT] IDETestOperationsObserverDebug: 368.778 sec, +368.778 sec -- end

Test session results, code coverage, and logs:
	/tmp/derived_datadpr1v0ai/Logs/Test/Test-RunnerTests-2025.09.25_08-17-42--0700.xcresult

Testing failed:
	Runner (801) encountered an error (Test runner never began executing tests after launching)

** TEST EXECUTE FAILED **

Testing started
 
[DEVICEFARM] ########### Entering phase post_test ###########

[DEVICEFARM] ########### Finish executing testspec ###########
 
[DEVICEFARM] ########### Setting upload permissions ###########

[DEVICEFARM] Tearing down your device. Your tests report will come shortly.
```

re:Post Agent · Answer

The issue you're experiencing with your Flutter integration tests stopping around the 6-minute mark on AWS Device Farm iOS devices appears to be related to a test execution timeout or a WebDriverAgent session issue.

Looking at your test output, I notice that the tests are running successfully for about 5 minutes and 27 seconds (your last logged test starts at "05:27") before stopping abruptly. The error message "Test runner never began executing tests after launching" suggests that the test runner lost connection with the device or timed out.

There are a few potential causes and solutions to consider:

1. **WebDriverAgent Session Issues**: Device Farm uses WebDriverAgent for iOS testing, and sometimes it can fail to maintain a session for longer test runs. This is a known issue that has affected other users.

2. **Device-Specific Problems**: Try changing the iOS device in your device pool. Some users have reported that switching from one iOS device to another (for example, from an iPhone with iOS 16.0.2 to one with iOS 16.6) resolved similar issues.

3. **Test Structure**: Consider breaking your test suite into smaller chunks. Instead of running all 12 tests in one go, you could split them into multiple test runs with fewer tests each.

4. **Appium Configuration**: If you're using Appium under the hood, you might need to adjust your configuration. Some users have had success with specific versions of Appium and the XCUITest driver.

5. **Test Execution Timeout**: While you've set the `-maximum-test-execution-time-allowance` parameter to 2400 seconds, there might be another timeout at play. The execution timeout for Device Farm should be set to the maximum duration of your test run plus some buffer (e.g., if your tests take 20 minutes, set a timeout of 30 minutes).

Try these approaches to resolve the issue, with changing the iOS device being potentially the quickest solution to test.  
**Sources**  
[Setting the execution timeout for test runs in AWS Device Farm - AWS Device Farm](https://docs.aws.amazon.com/devicefarm/latest/developerguide/how-to-set-default-timeout-for-test-runs.html)  
[Device farm (iOS) run failure - Unable to start WebDriverAgent session because of xcodebuild failure | AWS re:Post](https://repost.aws/questions/QUIoP8rRhzTCyYJx2ro9qEOg/device-farm-ios-run-failure-unable-to-start-webdriveragent-session-because-of-xcodebuild-failure)

Flutter integration tests stop running after certain time on AWS Device Farm iOS device

Add your answer

Relevant content