The Mobify Test Framework is a package of testing tools used for performance and end-to-end (E2E) testing of your PWAs.
mobify-test-framework [command] Commands: lighthouse [options] <urls...> Runs lighthouse tests on a list of URLs nightwatch [options] Runs nightwatch end-to-end tests, using Mobify's recommended settings.
Performance Testing with Lighthouse
Lighthouse outputs a report (in HTML and JSON) and our tool will assert a PASS or FAIL dependent on whether values meet configurable thresholds.
See the list of configurable thresholds and recommended values below:
# How to use: mobify-test-framework lighthouse https://www.merlinspotions.com # Basic Settings: --maxTTI - Maximum time to interactive before reporting a failure (default: 10000 ms) --minPWAScore - Minimum PWA score before reporting a failure (default: 90) --minSEOScore - Minimum SEO score before reporting a failure (default: 100) --minAccessibilityScore - Minimum accessibility score before reporting a failure (default: 100) --checkConsoleErrors - Assert that browser errors are not logged to the console (default: false) --mobifyPreview - set Mobify Preview parameters to the url (default: false) --device - Form factor for tests (choices: 'mobile', 'desktop') (default: 'mobile') --outputDir - set the Output directory for reports (default: 'tests/lighthouse-reports') # We can also run Lighthouse against multiple URLs: mobify-test-framework lighthouse https://www.merlinspotions.com https://www.merlinspotions.com/potions # To run against local files with Mobify Preview, for when the PWA is not live: # First run `npm start` in another tab mobify-test-framework lighthouse https://www.merlinspotions.com --mobifyPreview
End to End Testing with Nightwatch
Nightwatch.js is a tool that automates user
interaction for browser end-to-end tests. The Mobify Test Framework manages the
installation of Chromedriver and the latest version is
automatically downloaded when the project is installed.
# The following are optional environment variables of which you can set before running: Environment variables: NIGHTWATCH_SAUCE_USERNAME Saucelabs username NIGHTWATCH_SAUCE_ACCESS_KEY Saucelabs password NIGHTWATCH_SRC_FOLDERS Space-separated list of folders containing tests (default ['./tests/e2e']) NIGHTWATCH_OUTPUT_FOLDER Output folder for test reports (default './tests/reports') NIGHTWATCH_SCREENSHOTS_PATH Output folder for test screenshots (default './tests/screenshots') # Run all end-to-end tests mobify-test-framework nightwatch
default test environment is a Chrome instance that emulates a Pixel 2 device. You can
view the configuration file at
To specify a custom path to Chromedriver, or to create new test configurations, you can create your own nightwatch-config and override it entirely. Additionally, you can pass any regular nightwatch argument to the Mobify Test Framwork.
# To run nightwatch while passing a custom nightwatch-config mobify-test-framework nightwatch -config <file_path> # Use '--' to pass extra args directly to nightwatch. # eg. mobify-test-framework.js nightwatch -- "--verbose --env chrome_incognito" # Run all end-to-end tests for a given tag mobify-test-framework nightwatch -- "--tag checkout" # Run all tests within a subdirectory mobify-test-framework nightwatch -- "--group workflows/smoke-test" # Run all end-to-end tests under a specific test environment mobify-test-framework nightwatch -- "-e safari" # Run only one test mobify-test-framework nightwatch -- tests/e2e/workflows/merlins/home-example.js # Run end-to-end tests on the production environment env SKIP_PREVIEW=1 mobify-test-framework nightwatch # If a test has failed and you would like to debug a single test without the browser closing in the end env DEBUG=1 mobify-test-framework nightwatch --test test/e2e/workflows/guest-checkout.js
A typical project will have the tests fall under the
folder while page-objects (template-specific selectors and functions) are
located in the
We recommend creating end-to-end tests go through the core flow of checkout for both registered and guest to ensure the core flow does not break throughout the development cycle.