🎉 Check out DevCenter, our new documentation site for v2.0! (Or keep using this site for earlier versions.)

Switch to DevCenter

Menu Sheet Overlay

The Mobify Test Framework

The Mobify Test Framework is a package of testing tools used for performance and end-to-end (E2E) testing of your PWAs.


mobify-test-framework [command]
    lighthouse [options] <urls...>  Runs lighthouse tests on a list of URLs
    nightwatch [options]            Runs nightwatch end-to-end tests, using Mobify's recommended settings.

Performance Testing with Lighthouse

Lighthouse is a tool we use to assess an app against Google's PWA Checklist.

Lighthouse outputs a report (in HTML and JSON) and our tool will assert a PASS or FAIL dependent on whether values meet configurable thresholds.

See the list of configurable thresholds and recommended values below:

# How to use:
mobify-test-framework lighthouse https://www.merlinspotions.com

# Basic Settings:
--maxTTI - Maximum time to interactive before reporting a failure (default: 10000 ms)
--minPWAScore - Minimum PWA score before reporting a failure (default: 90)
--minSEOScore - Minimum SEO score before reporting a failure (default: 100)
--minAccessibilityScore - Minimum accessibility score before reporting a failure (default: 100)
--checkConsoleErrors - Assert that browser errors are not logged to the console (default: false)
--mobifyPreview - set Mobify Preview parameters to the url (default: false)
--device - Form factor for tests (choices: 'mobile', 'desktop') (default: 'mobile')
--outputDir - set the Output directory for reports (default: 'tests/lighthouse-reports')

# We can also run Lighthouse against multiple URLs:
mobify-test-framework lighthouse https://www.merlinspotions.com https://www.merlinspotions.com/potions

# To run against local files with Mobify Preview, for when the PWA is not live:
# First run `npm start` in another tab
mobify-test-framework lighthouse https://www.merlinspotions.com --mobifyPreview

End to End Testing with Nightwatch

Nightwatch.js is a tool that automates user interaction for browser end-to-end tests. The Mobify Test Framework manages the installation of Chromedriver and the latest version is automatically downloaded when the project is installed.

# The following are optional environment variables of which you can set before running:
Environment variables:
    NIGHTWATCH_SAUCE_USERNAME      Saucelabs username
    NIGHTWATCH_SAUCE_ACCESS_KEY    Saucelabs password
    NIGHTWATCH_SRC_FOLDERS         Space-separated list of folders containing tests (default ['./tests/e2e'])
    NIGHTWATCH_OUTPUT_FOLDER       Output folder for test reports (default './tests/reports')
    NIGHTWATCH_SCREENSHOTS_PATH    Output folder for test screenshots (default './tests/screenshots')

# Run all end-to-end tests
mobify-test-framework nightwatch

The default test environment is a Chrome instance that emulates a Pixel 2 device. You can view the configuration file at node_modules/@mobify/test-framework/src/nightwatch-config.js.

To specify a custom path to Chromedriver, or to create new test configurations, you can create your own nightwatch-config and override it entirely. Additionally, you can pass any regular nightwatch argument to the Mobify Test Framwork.

# To run nightwatch while passing a custom nightwatch-config 
mobify-test-framework nightwatch -config <file_path>

# Use '--' to pass extra args directly to nightwatch.
# eg. mobify-test-framework.js nightwatch -- "--verbose --env chrome_incognito"

# Run all end-to-end tests for a given tag
mobify-test-framework nightwatch -- "--tag checkout"

# Run all tests within a subdirectory
mobify-test-framework nightwatch -- "--group workflows/smoke-test"

# Run all end-to-end tests under a specific test environment
mobify-test-framework nightwatch -- "-e safari"

# Run only one test
mobify-test-framework nightwatch -- tests/e2e/workflows/merlins/home-example.js

# Run end-to-end tests on the production environment
env SKIP_PREVIEW=1 mobify-test-framework nightwatch

# If a test has failed and you would like to debug a single test without the browser closing in the end
env DEBUG=1 mobify-test-framework nightwatch --test test/e2e/workflows/guest-checkout.js

A typical project will have the tests fall under the web/tests/e2e/workflows/ folder while page-objects (template-specific selectors and functions) are located in the web/tests/e2e/page-objects/ folder.

We recommend creating end-to-end tests go through the core flow of checkout for both registered and guest to ensure the core flow does not break throughout the development cycle.



Was this page helpful?