Menu Sheet Overlay
Search Sheet

      Mobify Test Framework


      The Mobify Test Framework is a package of integrated testing tools used for End-to-end (E2E) and Performance testing PWAs.

      The framework uses:

      1. Lighthouse
      2. Nightwatch


      mobify-test-framework [command]
          lighthouse [options] <urls...>  Runs lighthouse tests on a list of URLs
          nightwatch [options]            Runs nightwatch end-to-end tests, using Mobify's recommended settings.

      Performance Testing #

      Lighthouse #

      Lighthouse is a tool we use to assess an app against Google's PWA Checklist.

      It outputs a lighthouse report (in html and json) and our tool will assert a PASS or FAIL dependent on values and whether they meet configurable thresholds. We focus on two metrics: the Progressive Web App score and Time to First Interactive.

      The Progressive Web App score is a percentage of passed PWA audits from various requirements. Aim for a score of 90 or better before project launch.

      Time to First Interactive measures the first point in which necessary scripts have loaded and the CPU is idle enough to handle most user input. This threshold is given in milliseconds (ms). Our default threshold is under 10,000 ms and can be configured to be either higher or lower depending on the project.

      # How to install:
      npm i @mobify/test-framework
      # How to use:
      mobify-test-framework lighthouse
      # Basic Settings:
      --MaxTTI - set your the threshold of Time to Interactive in ms before reporting a failure (default: 10000 ms)
      --minPwaScore - Minimum PWA score before reporting a failure (default: 90)
      --minSEOScore - Minimum SEO score before reporting a failure (default: 100)
      --minAccessibilityScore - Minimum accessibility score before reporting a failure (default: 100)
      --checkConsoleErrors - Assert that browser errors are not logged to the console (default: false)
      --mobifyPreview - Run tests using Mobify preview (default: false)
      --outputDir - Output directory for reports (default: 'tests/lighthouse-reports')
      # We can also run Lighthouse against multiple URLs:
      mobify-test-framework lighthouse
      # To run against local files with Mobify Preview, for when the PWA is not live:
      # First run `npm start` to start development server of the PWA in another tab
      mobify-test-framework lighthouse --mobifyPreview

      End to End Testing #

      Nightwatch #

      Nightwatch.js is a tool that automates user interaction for browser end-to-end tests. The Mobify Test Framework manages the installation of Selenium and Chromedriver and the latest versions are automatically downloaded when the project is installed.

      # The following are optional environment variables of which you can set before running:
      Environment variables:
          NIGHTWATCH_SAUCE_USERNAME      Saucelabs username
          NIGHTWATCH_SAUCE_ACCESS_KEY    Saucelabs password
          NIGHTWATCH_SRC_FOLDERS         Space-separated list of folders containing tests (default ['./tests/e2e'])
          NIGHTWATCH_OUTPUT_FOLDER       Output folder for test reports (default './tests/reports')
          NIGHTWATCH_SCREENSHOTS_PATH    Output folder for test screenshots (default './tests/screenshots')
      # Run all end-to-end tests
      mobify-test-framework nightwatch

      Tests by default will use the default environment which is a Chrome instance that emulates a Nexus 5 device with browser push notifications enabled. You can view the configuration file at node_modules/@mobify/test-framework/src/nightwatch-config.js.

      To specify custom paths to Selenium or Chromedriver, or to create new test configurations, you can create your own nightwatch-config and override it entirely. Additionally, you can pass any regular nightwatch argument to the Mobify Test Framwork.

      # To run nightwatch while passing a custom nightwatch-config
      mobify-test-framework nightwatch -config <file_path>
      # Use '--' to pass extra args directly to nightwatch.
      # eg. mobify-test-framework.js nightwatch -- "--verbose --env chrome_incognito"
      # Run all end-to-end tests for a given tag
      mobify-test-framework nightwatch -- "--tag checkout"
      # Run all tests within a subdirectory
      mobify-test-framework nightwatch -- "--group workflows/smoke-test"
      # Run all end-to-end tests under a specific test environment
      mobify-test-framework nightwatch -- "-e safari"
      # Run only one test
      mobify-test-framework nightwatch -- tests/e2e/workflows/merlins/home-example.js
      # Run end-to-end tests on the production environment
      env SKIP_PREVIEW=1 mobify-test-framework nightwatch
      # If a test has failed and you would like to debug a single test without the browser closing in the end
      env DEBUG=1 mobify-test-framework nightwatch --test test/e2e/workflows/guest-checkout.js

      A typical project will have the tests fall under the web/tests/e2e/workflows/ folder while page-objects (template-specific selectors and functions) are located in the web/tests/e2e/page-objects/ folder.

      We recommend creating End-to-end tests go through the core flow of checkout for both registered and guest to ensure the core flow does not break throughout the development cycle.


      Was this page helpful?