Menu Sheet Overlay
Search Sheet

      The Mobify Test Framework

      The Mobify Test Framework is a package of testing tools used for performance and end-to-end (E2E) testing of your PWAs.


      mobify-test-framework [command]
          lighthouse [options] <urls...>  Runs lighthouse tests on a list of URLs
          nightwatch [options]            Runs nightwatch end-to-end tests, using Mobify's recommended settings.

      Performance Testing with Lighthouse

      Lighthouse is a tool we use to assess an app against Google's PWA Checklist.

      Lighthouse outputs a report (in HTML and JSON) and our tool will assert a PASS or FAIL dependent on whether values meet configurable thresholds.

      See the list of configurable thresholds and recommended values below:

      # How to use:
      mobify-test-framework lighthouse
      # Basic Settings:
      --maxTTI - Maximum time to interactive before reporting a failure (default: 10000 ms)
      --minPWAScore - Minimum PWA score before reporting a failure (default: 90)
      --minSEOScore - Minimum SEO score before reporting a failure (default: 100)
      --minAccessibilityScore - Minimum accessibility score before reporting a failure (default: 100)
      --checkConsoleErrors - Assert that browser errors are not logged to the console (default: false)
      --mobifyPreview - set Mobify Preview parameters to the url (default: false)
      --device - Form factor for tests (choices: 'mobile', 'desktop') (default: 'mobile')
      --outputDir - set the Output directory for reports (default: 'tests/lighthouse-reports')
      # We can also run Lighthouse against multiple URLs:
      mobify-test-framework lighthouse
      # To run against local files with Mobify Preview, for when the PWA is not live:
      # First run `npm start` in another tab
      mobify-test-framework lighthouse --mobifyPreview

      End to End Testing with Nightwatch

      Nightwatch.js is a tool that automates user interaction for browser end-to-end tests. The Mobify Test Framework manages the installation of Chromedriver and the latest version is automatically downloaded when the project is installed.

      # The following are optional environment variables of which you can set before running:
      Environment variables:
          NIGHTWATCH_SAUCE_USERNAME      Saucelabs username
          NIGHTWATCH_SAUCE_ACCESS_KEY    Saucelabs password
          NIGHTWATCH_SRC_FOLDERS         Space-separated list of folders containing tests (default ['./tests/e2e'])
          NIGHTWATCH_OUTPUT_FOLDER       Output folder for test reports (default './tests/reports')
          NIGHTWATCH_SCREENSHOTS_PATH    Output folder for test screenshots (default './tests/screenshots')
      # Run all end-to-end tests
      mobify-test-framework nightwatch

      The default test environment is a Chrome instance that emulates a Pixel 2 device. You can view the configuration file at node_modules/@mobify/test-framework/src/nightwatch-config.js.

      To specify a custom path to Chromedriver, or to create new test configurations, you can create your own nightwatch-config and override it entirely. Additionally, you can pass any regular nightwatch argument to the Mobify Test Framwork.

      # To run nightwatch while passing a custom nightwatch-config 
      mobify-test-framework nightwatch -config <file_path>
      # Use '--' to pass extra args directly to nightwatch.
      # eg. mobify-test-framework.js nightwatch -- "--verbose --env chrome_incognito"
      # Run all end-to-end tests for a given tag
      mobify-test-framework nightwatch -- "--tag checkout"
      # Run all tests within a subdirectory
      mobify-test-framework nightwatch -- "--group workflows/smoke-test"
      # Run all end-to-end tests under a specific test environment
      mobify-test-framework nightwatch -- "-e safari"
      # Run only one test
      mobify-test-framework nightwatch -- tests/e2e/workflows/merlins/home-example.js
      # Run end-to-end tests on the production environment
      env SKIP_PREVIEW=1 mobify-test-framework nightwatch
      # If a test has failed and you would like to debug a single test without the browser closing in the end
      env DEBUG=1 mobify-test-framework nightwatch --test test/e2e/workflows/guest-checkout.js

      A typical project will have the tests fall under the web/tests/e2e/workflows/ folder while page-objects (template-specific selectors and functions) are located in the web/tests/e2e/page-objects/ folder.

      We recommend creating end-to-end tests go through the core flow of checkout for both registered and guest to ensure the core flow does not break throughout the development cycle.



      Was this page helpful?