All multivariate testing (MVT) solutions provide capabilities for marketing teams to change the functionality of a shopper experience with minimal IT involvement by providing:
- Management of experiences for segments of customers
- Management of MVT or A/B experiments to help marketing teams find the optimal converting experience (at either a page level or a flow level)
- Tracking of shopper behavior via analytics or tracking pixels so that shoppers can be segmented for experiences or experiments
Table of contents
- Project phases
- Common limitations
- Soft navigations
- Leveraging the Analytics Manager
- Multiple environments
- SDK components
Project phases #
A typical MVT integration project will involve the following phases:
- MVT setup - creating sites or instances and integrating the MVT SDK into the Mobify PWA
- Analytics integration - the sending of behavioral data about the shopper to the MVT solution for tracking
- Experience integration - the integration and testing of all supported MVT actions, such as inserting HTML, changing CSS, or hiding HTML elements. Some of the actions supported by the MVT solution may require extra work to integrate them with your PWA
- MVT Tools integration - the integration and testing of MVT tools, such as preview and debugging tools. Some customers may decide not to use some of the MVT tools such as WYSIWYG editors due to workflow constraints or integration limitations.
- Performance validation - the testing and validation of the MVT solution. It should have minimal impact on the overall Time to Interactive or Time to First Contentful Paint for a given shopper. A trade-off may be required where the flexibility of integrating the MVT solution will be at the cost of overall perceived performance across all shoppers.
Common limitations #
Most MVT solutions are built for traditional web experiences that leverage a standard request and response cycle. However, given that Mobify’s Progressive Web Apps are built on a single-page app architecture, the standard request and response cycle is not available for integration. Due to this difference, the degree of difficulty for integrating with a given MVT solution will vary based on its support for single-page apps. Support for single-page apps in the majority of MVT solutions is still relatively immature, but you can expect this to improve with future updates.
Based on the current level of support for single-page apps, the following limitations may apply to your MVT integration.
Some MVT solution will provide a WYSIWYG interface for previewing and editing experiences so that non-technical users can easily edit HTML and CSS. Getting the WYSIWYG tool to work with Mobify Preview will typically take extra project time, and we recommend working with your marketing team to see if WYSIWYG tools are required.
Not all experiences can be set up with just a WYSIWYG editor, and your team should be prepared to create experiences by adding code to the PWA in certain scenarios. We recommend using a WYSIWYG tool only for setting up the simplest experiences.
Links in MVT experiences
Moving and swapping DOM elements
Using the MVT solution to move or swap DOM elements in the PWA is not recommended due to potential race conditions between the MVT and PWA lifecycles. Movement of a PWA component that hasn’t been fully initialized could result in unexpected behavior. We recommending setting up variations of components in the PWA that can be shown or hidden instead of moved/swapped.
Changing DOM elements
Changing DOM elements by adding CSS or changing HTML needs to be done carefully. When selecting the element to target (typically via a CSS selector), the changes applied by the MVT solution may persist over the entire shopper session. For example, changing the button on a specific product details page may possibly apply this change to all product details pages if the CSS selector is not set up correctly. Typically, the CSS selector suggested by the MVT WYSIWYG tool will not be the correct one to use; therefore, training the users of the MVT solution to craft the correct selector is critical.
Resetting UI state across soft navigations
Separating mobile and desktop experiences
Typically, the MVT solution will use a single tag and site that covers desktop and mobile experiences. This is optimal for the user of the MVT solution because they don’t have to switch between sites when creating experiments. However, Desktop and Mobile experiments will need to be set up differently. Any experiences that affect both desktop and mobile will need to be set up and managed for each device type. We recommend tagging or naming experiments so that this is clear to teams. Also, experiences need to be set up to make sure the experiments run only for the intended device type. For example, a new action may need to be added to experiments to allow or disallow the running of an experiment on mobile.
Whenever possible, we recommend using an asynchronous tag when loading MVT solutions. We have found that using synchronous tags for loading the MVT will introduce a major impact to all Time to Interactive measurements. Given the strong impact of performance to conversion and revenue, we recommend prioritizing mobile performance over MVT.
Asynchronous tags will possibly result in a brief flash of content as the PWA will render first and then the MVT experiences will execute to add/update content. We recommend adding placeholders in areas where MVT experience will modify content, so that any reflow of layout is minimized.
Soft navigations #
MVT engines typically rely on hard navigations (visiting a page or refreshing a full page) to execute their lifecycles. When integrating into a Single Page App, these hard navigations only occur on initial page load, and all further interactions are soft navigations that rely on the single-page app to load data and update any components on the page as necessary.
To integrate correctly with soft navigations correctly into the MVT lifecycle, there are two typical approaches: Leverage the MVT solution support for React integrations. Some MVT solutions support React by hooking into React Router to simulate hard navigations based on routes Custom integrate the React Router lifecycle into the MVT. MVT solutions will typically provide an API via their SDK that will trigger the equivalent of a hard navigation. However, this solution may be subject to race conditions if the MVT solution doesn’t provide a way of knowing when the MVT engine is loaded and ready to go, given that asynchronous loading is recommended.
Leveraging the Analytics Manager #
MVT solutions will typically require the sending of shopper behavior data, such as which pages have been visited, which products have been viewed and added to cart, and which searches have been executed in order to leverage the full set of capabilities of audience segmentation.
We recommend leveraging the Analytics Manager as a framework to send any required data, as most of the data required is already available and hooked into the PWA architecture. Analytics Manager also follows our best practices for minimizing performance impact and provides tools for debugging any analytics calls made.
A Monetate integration via Analytics Manager is available for projects to start with.
Multiple environments #
We recommend both the MVT and Mobify PWA solution have multiple development environments setup for Development, Staging and Production so that marketers and IT teams can test and validate the setup of different experiments against upcoming PWA code deployments in a safe environment prior to launching either solution into Production.
SDK Components #
Here are the SDK components that will help make multivariate testing easier: