Run test on a device in multiple orientations in a single run. (#167)

* Added initial support for multiple orientations * Removed unused tests * Added orientation prefix to image name to prevent overwriting * Added comments * Added documentation on using multiple orientations per device. Also some cleanup * Specified parameter type * Fixed and enhanced test * Call run tests when orientation not specified * Enable test to run on CI * Fixed test * Find cirrus artifacts * Fixed test * Fixed test for CI * Fixed test for CI
mmcc007 · Nov 27, 2019 · b24c16a · b24c16a
1 parent eea5171
commit b24c16a
Show file tree

Hide file tree

Showing 16 changed files with 306 additions and 209 deletions.
diff --git a/.cirrus.yml b/.cirrus.yml
@@ -29,5 +29,6 @@ integration_task:
     - export PATH="$HOME/.pub-cache/bin:$PATH" # needed to find screenshots
     - cd example
     - screenshots -c screenshots_ios.yaml -v
+  find_artifacts: find ios/fastlane/screenshots
   screenshot_artifacts:
-    path: ios/fastlane/screenshots/*
+    path: ios/fastlane/screenshots
diff --git a/README.md b/README.md
@@ -24,9 +24,9 @@ For introduction to _Screenshots_ see https://medium.com/@nocnoc/automated-scree
 
 _Screenshots_ is a standalone command line utility and package for capturing screenshot images for Flutter.   
 
-_Screenshots_ will start the required android emulators and iOS simulators (or find attached devices), run your screen capture tests on each emulator/simulator (or device), process the images, and drop them off to Fastlane for delivery to both stores.
+_Screenshots_ will start the required android emulators and iOS simulators (or find attached devices), run tests, process the captured screenshots, and drop them off to Fastlane for delivery to both stores.
 
-It is inspired by three tools from Fastlane:  
+_Screenshots_ is inspired by three tools from Fastlane:  
 1. [Snapshots](https://docs.fastlane.tools/getting-started/ios/screenshots/)  
 This is used to capture screenshots on iOS using iOS UI Tests.
 1. [Screengrab](https://docs.fastlane.tools/actions/screengrab/)  
@@ -40,15 +40,15 @@ Since all three of these Fastlane tools do not work with Flutter, _Screenshots_
 Since Flutter integration testing is designed to work transparently across iOS and Android, capturing images using _Screenshots_ is easy.
 
 Features include: 
-1. Works with your existing tests  
+1. Works with existing tests  
 Add a single line for each screenshot.
-1. Run your tests on any device  
-Select the devices you want to run on, using a convenient config file. _Screenshots_ will find the devices (real or emulated) and run your tests.
+1. Run tests on any device  
+Select the devices to run on using a convenient config file. _Screenshots_ will find the devices (real or emulated) and run tests.
 1. One run for both platforms  
-_Screenshots_ runs your tests on both iOS and Android in one run.  
+_Screenshots_ runs tests on both iOS and Android in one run.  
 (as opposed to making separate Snapshots and Screengrab runs)
 1. One run for multiple locales  
-If your app supports multiple locales, _Screenshots_ will optionally set the locales listed in the config file before running each test.
+If app supports multiple locales, _Screenshots_ will optionally set the locales listed in the config file before running each test.
 1. One run for frames  
 Optionally places images in device frames in same run.  
 (as opposed to making separate FrameIt runs... which supports iOS only)
@@ -141,17 +141,18 @@ sample usage: screenshots
 -h, --help                                          Display this help information.
 ```
 
-# Modifying your tests for _Screenshots_
-A special function is provided in the _Screenshots_ package that is called by the test each time you want to capture a screenshot. 
+# Modifying tests for _Screenshots_
+A special function is provided in the _Screenshots_ package that is called by the test to capture a screenshot. 
 _Screenshots_ will then process the images appropriately during a _Screenshots_ run.
 
-To capture screenshots in your tests:
-1. Include the _Screenshots_ package in your pubspec.yaml's dev_dependencies section  
+To capture screenshots in tests:
+1. Add _Screenshots_ package in app's pubspec.yaml's dev_dependencies section  
    ````yaml
+   dev_dependencies:
      screenshots: ^<current version>
    ````
-2. In your tests
-    1. Import the dependencies  
+2. In tests
+    1. Import the dependency  
        ````dart
        import 'package:screenshots/screenshots.dart';
        ````
@@ -164,15 +165,15 @@ To capture screenshots in your tests:
            await screenshot(driver, config, 'myscreenshot1');
        ````
 
-Note: make sure your screenshot names are unique across all your tests.
+Note: make sure screenshot names are unique across all tests.
 
-Note: to turn off the debug banner on your screens, in your integration test's main(), call:
+Note: to turn off the debug banner, in the integration test's main(), call:
 ````dart
   WidgetsApp.debugAllowBannerOverride = false; // remove debug banner for screenshots
 ````
 
 ## Modifying tests based on screenshots environment
-In some cases it is useful to know what device, device type, screen size, screen orientation and locale you are currently testing with. To obtain this information in your test use:
+In some cases it is useful to know what device, device type, screen size, screen orientation and locale the test is currently running with. To obtain this information use:
 ```
 final screenshotsEnv = config.screenshotsEnv;
 ```
@@ -224,9 +225,17 @@ _frame_ parameter notes:
 - set to true for devices unknown to _screenshots_.
 
 _orientation_ parameter notes:
+- multiple orientations can be specified. For example:
+    ```yaml
+        iPhone XS Max:
+          orientation:
+            - Portrait
+            - LandscapeRight
+    ```
+- landscape orientation disables framing  
+  This is because status/navigation bars in landscape mode are currently not implemented.
 - orientation on iOS simulators is implemented using an AppleScript script which requires granting permission on first use.
 
-
 ## Test Options
 In addition to using the default flutter driver mode, tests can also be specified using flutter driver parameters. For example:
 ```
@@ -239,11 +248,11 @@ tests:
 _Screenshots_ can be used to monitor any unexpected changes to the UI by comparing the new screenshots to previously recorded screenshots. Any differences will be highlighted in a 'diff' image for review.  
 
 To use this feature:
-1. Add the location of your recording directory to a `screenshots.yaml`
+1. Add a recording directory to `screenshots.yaml`
     ```yaml
     recording: /tmp/screenshots_record
     ```
-1. Run a recording to capture your screenshots:
+1. Run a recording to capture expected screenshots:
     ```
     screenshots -m recording
     ```
@@ -257,7 +266,7 @@ To use this feature:
 To generate screenshots for local use, such as generating reports of changes to UI over time, etc... use 'archive' mode. 
 
 To enable this mode:
-1. Add the location of your archive directory to screenshots.yaml:
+1. Add an archive directory to screenshots.yaml:
     ```yaml
     archive: /tmp/screenshots_archive
     ```
@@ -267,7 +276,7 @@ To enable this mode:
     ````
 
 # Integration with Fastlane
-Since _Screenshots_ is intended to be used with Fastlane, after _Screenshots_ completes, the images can be found in your project at:
+Since _Screenshots_ is intended to be used with Fastlane, after _Screenshots_ completes, the images can be found in the project at:
 ````
 android/fastlane/metadata/android
 ios/fastlane/screenshots
@@ -282,40 +291,32 @@ Tip: One way to use _Screenshots_ with Fastlane is to call _Screenshots_ before
 ## Fastlane FrameIt
 iOS images generated by _Screenshots_ can also be further processed using FrameIt's [text and background](https://docs.fastlane.tools/actions/frameit/#text-and-background) feature.
 
-# Changing devices
+# Adding a device
 
-To change the devices to run your tests on, just change the list of devices in screenshots.yaml.
-
-Make sure each device you select has a supported screen and a
-corresponding attached device or installed emulator/simulator. To bypass
-the supported screen requirement use `frame: false` for each related device in your
-screenshots.yaml.
-
-For each selected device:
 1. Confirm device is present in [screens.yaml](https://github.com/mmcc007/screenshots/blob/master/lib/resources/screens.yaml).  
 2. Add device to the list of devices in screenshots.yaml.  
 3. Confirm a real device is attached, or install an emulator/simulator for device.   
 
-If your device is not found in screens.yaml but matches a screen size in screens.yaml, please create an issue or PR to add the device to screens.yaml.
+If screen is not available, disable framing by setting `frame: false` for the device.
 
 ## Config validation
-_Screenshots_ will check your configuration before running for any errors and provide a guide on how to resolve.
+_Screenshots_ will check configuration before running for errors and provide a guide on how to resolve.
 
-# Adding new screens
+# Adding a screen
 
-If your device does not have a screen in screens.yaml please create an issue to request a new screen.
+If device does not have a screen in screens.yaml please create an issue to request a new screen.
 
-If you want to submit a new screen please see related [README](https://github.com/mmcc007/screenshots/blob/master/test/resources/README.md).
+To submit a new screen please see related [README](https://github.com/mmcc007/screenshots/blob/master/test/resources/README.md).
 
 # Upgrading
 To upgrade, simply re-issue the install command
 ````bash
 $ pub global activate screenshots
 ````
-Note: the _Screenshots_ version should be the same for both the command line and in your `pubspec.yaml`.   
+Note: the _Screenshots_ version should be the same for both the command line and in the `pubspec.yaml`.   
 1. If upgrading the command line version of _Screenshots_, also upgrade
- the version of _Screenshots_ in your pubspec.yaml.    
-2. If upgrading the version of _Screenshots_ in your pubspec.yaml, also upgrade the command line version.
+ the version of _Screenshots_ in the pubspec.yaml.    
+2. If upgrading the version of _Screenshots_ in pubspec.yaml, also upgrade the command line version.
 
 To check the version of _Screenshots_ currently installed:
 ```
@@ -332,8 +333,8 @@ https://github.com/mmcc007/screenshots/releases/
 To view a similar run on windows see:  
 https://ci.appveyor.com/project/mmcc007/screenshots
 
-* Running _Screenshots_ in the cloud is  useful for automating the generation of your screenshots in a CI/CD environment.  
-* Running _Screenshots_ on macOS in the cloud can be used to generate your screenshots when developing on Linux and/or Windows (if not using locally attached iOS devices).
+* Running _Screenshots_ in the cloud is  useful for automating the generation of screenshots in a CI/CD environment.  
+* Running _Screenshots_ on macOS in the cloud can be used to generate screenshots when developing on Linux and/or Windows.
 
 # Related projects
 To run integration tests on real devices in cloud:  
@@ -344,6 +345,4 @@ https://github.com/mmcc007/fledge
 
 # Issues and Pull Requests
 [Issues](https://github.com/mmcc007/screenshots/issues) and 
-[pull requests](https://github.com/mmcc007/screenshots/pulls) are welcome.
-
-Your feedback is welcome and is used to guide where development effort is focused. So feel free to create as many issues and pull requests as you want. You should expect a timely and considered response.
+[pull requests](https://github.com/mmcc007/screenshots/pulls) are welcome.
diff --git a/example/screenshots.yaml b/example/screenshots.yaml
@@ -20,7 +20,9 @@ devices:
       orientation: LandscapeRight
   android:
     Nexus 6P:
-      orientation: LandscapeRight
+      orientation:
+        - LandscapeRight
+        - Portrait
 
 # Frame screenshots
 frame: true

diff --git a/lib/src/config.dart b/lib/src/config.dart
@@ -2,6 +2,7 @@ import 'dart:async';
 import 'dart:convert';
 import 'dart:io' as io;
 
+import 'package:collection/collection.dart';
 import 'package:meta/meta.dart';
 import 'package:screenshots/src/orientation.dart';
 
@@ -13,6 +14,7 @@ const kEnvConfigPath = 'SCREENSHOTS_YAML';
 
 /// Config info used to manage screenshots for android and ios.
 // Note: should not have context dependencies as is also used in driver.
+// todo: yaml validation
 class Config {
   Config({this.configPath = kConfigFileName, String configStr}) {
     if (configStr != null) {
@@ -48,6 +50,7 @@ class Config {
 
   Map _configInfo;
   Map _screenshotsEnv; // current screenshots env
+  List<ConfigDevice> _devices;
 
   // Getters
   List<String> get tests => _processList(_configInfo['tests']);
@@ -56,7 +59,7 @@ class Config {
 
   List<String> get locales => _processList(_configInfo['locales']);
 
-  List<ConfigDevice> get devices =>
+  List<ConfigDevice> get devices => _devices ??=
       _processDevices(_configInfo['devices'], isFrameEnabled);
 
   List<ConfigDevice> get iosDevices =>
@@ -88,11 +91,12 @@ class Config {
   }
 
   /// Check if frame is required for [deviceName].
-  bool isFrameRequired(String deviceName) {
+  bool isFrameRequired(String deviceName, Orientation orientation) {
     final device = devices.firstWhere((device) => device.name == deviceName,
         orElse: () => throw 'Error: device \'$deviceName\' not found');
-    return (device.orientation == Orientation.LandscapeLeft ||
-            device.orientation == Orientation.LandscapeRight)
+    if (orientation == null) return device.isFramed;
+    return (orientation == Orientation.LandscapeLeft ||
+            orientation == Orientation.LandscapeRight)
         ? false
         : device.isFramed;
   }
@@ -142,11 +146,34 @@ class Config {
     }).toList();
   }
 
-  List<ConfigDevice> _processDevices(Map devices, bool globalFraming) {
+  List<ConfigDevice> _processDevices(
+      Map<String, dynamic> devices, bool globalFraming) {
+    Orientation _getValidOrientation(String orientation, deviceName) {
+      bool _isValidOrientation(String orientation) {
+        return Orientation.values.firstWhere(
+                (o) => utils.getStringFromEnum(o) == orientation,
+                orElse: () => null) !=
+            null;
+      }
+
+      if (!_isValidOrientation(orientation)) {
+        print(
+            'Invalid value for \'orientation\' for device \'$deviceName\': $orientation}');
+        print('Valid values:');
+        for (final _orientation in Orientation.values) {
+          print('  ${utils.getStringFromEnum(_orientation)}');
+        }
+        io.exit(1); // todo: add tool exception and throw
+      }
+      return utils.getEnumFromString(Orientation.values, orientation);
+    }
+
     List<ConfigDevice> configDevices = [];
 
     devices.forEach((deviceType, device) {
       device?.forEach((deviceName, deviceProps) {
+        final orientationVal =
+            deviceProps == null ? null : deviceProps['orientation'];
         configDevices.add(ConfigDevice(
           deviceName,
           utils.getEnumFromString(DeviceType.values, deviceType),
@@ -156,12 +183,13 @@ class Config {
                   globalFraming, // device frame overrides global frame
           deviceProps == null
               ? null
-              : deviceProps['orientation'] == null
+              : orientationVal == null
                   ? null
-                  : utils.getEnumFromString(
-                      Orientation.values, deviceProps['orientation'],
-                      allowNull: true),
-          deviceProps == null ? null : deviceProps['orientation'],
+                  : orientationVal is String
+                      ? [_getValidOrientation(orientationVal, deviceName)]
+                      : List<Orientation>.from(orientationVal.map((o) {
+                          return _getValidOrientation(o, deviceName);
+                        })),
           deviceProps == null ? true : deviceProps['build'] ?? true,
         ));
       });
@@ -171,21 +199,21 @@ class Config {
   }
 }
 
+Function eq = const ListEquality().equals;
+
 /// Describe a config device
 class ConfigDevice {
   final String name;
   final DeviceType deviceType;
   final bool isFramed;
-  final Orientation orientation;
-  final String orientationStr; // for validation
+  final List<Orientation> orientations;
   final bool isBuild;
 
   ConfigDevice(
     this.name,
     this.deviceType,
     this.isFramed,
-    this.orientation,
-    this.orientationStr,
+    this.orientations,
     this.isBuild,
   )   : assert(name != null),
         assert(deviceType != null),
@@ -197,12 +225,12 @@ class ConfigDevice {
     return other is ConfigDevice &&
         other.name == name &&
         other.isFramed == isFramed &&
-        other.orientation == orientation &&
+        eq(other.orientations, orientations) &&
         other.deviceType == deviceType &&
         other.isBuild == isBuild;
   }
 
   @override
   String toString() =>
-      'name: $name, deviceType: ${utils.getStringFromEnum(deviceType)}, isFramed: $isFramed, orientation: ${utils.getStringFromEnum(orientation)}, isBuild: $isBuild';
+      'name: $name, deviceType: ${utils.getStringFromEnum(deviceType)}, isFramed: $isFramed, orientations: $orientations, isBuild: $isBuild';
 }
diff --git a/lib/src/daemon_client.dart b/lib/src/daemon_client.dart
@@ -295,7 +295,7 @@ class DaemonDevice extends BaseDevice {
   }
 }
 
-DaemonEmulator loadDaemonEmulator(emulator) {
+DaemonEmulator loadDaemonEmulator(Map<String, dynamic> emulator) {
   return DaemonEmulator(
     emulator['id'],
     emulator['name'],
@@ -304,7 +304,7 @@ DaemonEmulator loadDaemonEmulator(emulator) {
   );
 }
 
-DaemonDevice loadDaemonDevice(device) {
+DaemonDevice loadDaemonDevice(Map<String, dynamic> device) {
   return DaemonDevice(
       device['id'],
       device['name'],