Menu
Menu Sheet Overlay
Search
Search Sheet

      Using the Request Processor to Optimize CDN Caching

      Note: This article discusses performance-optimizing techniques you can implement with the request processor, which are only applicable to server-side rendered PWAs. To get the most out of this guide, you’ll want to have a good understanding of networking and caching.

      Introduction

      In this article, we’ll introduce you to a new part of the Mobify Platform called the request processor, which handles requests as soon as they're received by the Mobify Platform. (Before the CDN looks for cached responses.) The request processor is a powerful tool that can modify parts of the request to affect caching, or mark requests for special processing.

      The fastest responses are when we fulfill from cache in the CDN:

      Cached Response

      The cached response.

      The request processor allows us to change how things are looked up in the cache:

      Request Processor

      The request processor.

      The request processor also lets us change the request, this changes what gets sent through:

      Request Processor Request Handling

      The request processor can modify the request.

      Below, we’ll cover three ways you can leverage the request processor to improve your Progressive Web App's (PWA’s) performance. You can use it for query parameter filtering, to cache optimally for different types of requests, or to efficiently handle unsupported browsers.

      Boosting CDN cache performance with query parameter filtering

      Your PWA may require the use of query parameters, which can affect how well your PWA’s content delivery network (CDN) caching works. Below, we'll introduce the problem they pose for cache-based site performance, and how you can use Mobify’s request processor to achieve query parameter filtering.

      The problem for performance

      One reason that Mobify’s server-side rendered PWAs are extremely fast is that content can be cached in the CDN. When a specific URL is requested, the CDN compares the URL (also known as the cache key), with those that are already in the cache. The comparison includes the origin, path and query parameters of the URL.

      Suppose the URL www.example.com/shoes has already been cached, and that there is a second variation of the URL with a unique query parameter appended to the end, something like www.example.com/shoes?id=10307&id=7473261. The added query parameter would mean that the cache key does not match, therefore the page would be treated like a new URL. In the example of the email campaign, every link using a query parameter is unique, meaning that each shopper would need to have their page re-loaded rather than using the cached version, costing precious load time.

      The solution

      With the request processor, you can implement query parameter filtering. This allows you to use query parameters, but with none of the associated impacts to cache-based site performance. When a URL is requested from the CDN, the request processor can edit the set of query parameters, removing any query parameters that would affect caching beyond those required by the server. The CDN can then better cache responses because URLs are more likely to match. With this implementation, the PWA running in the browser will still see the original query parameters (and can report them to analytics), but they will be removed from the request sent to the server.

      Let’s revisit the example of the email marketing campaign. Now, the page from the email marketing campaign can be served from the cache for most browsers, and the unique tracking links would still be intact.

      Example: query parameter filtering

      The following example shows a possible processRequest implementation, but keep in mind that for your own implementation, you should replace it with code appropriate to your project's specific requirements. To view the most up-to-date version of this example in the code, it’s accessible at packages/pwa/app/request-processor.js.example.

      /**
      * This example uses only the querystring parameter to processRequest. There are
      * other parameters that may be needed for other uses of the request-processor,
      * and these are documented in the Scaffold's example request-processor.
      *
      * @returns {{path: *, querystring: *}}
      */
      export const processRequest = ({ path, querystring }) => {
      
         // This example will remove any of the parameters whose keys appear
         // in the 'exclusions' array.
         const exclusions = [
             'gclid',
             'utm_campaign',
             'utm_content',
             'utm_medium',
             'utm_source'
         ]
      
         // Build a first QueryParameters object from the given querystring
         const incomingParameters = new QueryParameters(querystring)
      
         // Build a second QueryParameters from the first, with all excluded
         // parameters removed
         const filteredParameters = QueryParameters.from(
             incomingParameters.parameters.filter(
                 // parameter.key is always lower-case
                 (parameter) => !exclusions.includes(parameter.key)
             )
         )
      
         // Re-generate the querystring
         querystring = filteredParameters.toString()
      
         // Return the path unchanged, and the updated query string
         return {
             path,
             querystring
         }
      }
      
      

      Improving SEO by caching optimally for different requests

      Another functionality with the request processor is that you can alter and optimize the caching behavior for different types of requests. Consider requests from web crawlers, such as Googlebot. Caching is paramount for these requests, because the time to first byte determines whether search engine bots index the page in their search results. Ideally, you would set a longer CDN cache lifetime for responses to bot requests, to maximize the chances that those responses are cached. (Page staleness is less important for bots, as they are only concerned with indexing.) For example, you may decide to set your cache for only one hour for users, and set it to one day for bots. You may also want to load images differently depending on the type of request. You’re likely lazy-loading images to enhance the perceived speed of your PWA, but you may want to turn off lazy-loading for requests from bots, so that Googlebot gets all the images. The request processor allows us to do this by recognizing a particular type of request, in this case Googlebot requests, and putting them into a different class that’s cached separately from the user requests. The request class is also passed to the server-side rendering server, so that it can handle requests differently.

      Example: caching optimally for different requests

      In the example below, we extend the existing request processor to detect bot requests and set a longer cache lifetime:

      First, install the isbot npm package as a dependency of your project. (Run npm i --save isbot to install the isbot package.)

      Next, you’ll need to create the request-processor.js file:

      • For projects generated before March 2019: create the file in web/app/request-processor.js
      • For projects generated between March and June 2019: create the file in packages/pwa/app/request-processor.js
      • For projects generated on or after July 2019: rename packages/pwa/app/request-processor.js.example by deleting .example

      Within your new request-processor.js file, paste in the following code:

      const isbot = require('isbot')
      
      export const processRequest = ({headers, setRequestClass}) => {
          // Identify bots and set the request class
          if (headers && setRequestClass && isbot(headers.getHeader('user-agent'))) {
              setRequestClass('bot')
          }
      }
      

      Next, add the following code to the responseHook, which is located within pwa/app/ssr.js.

      // If this is a bot response, set the cache-control headers to
      // extra-long. We do this even on a local development server
      // so that the behaviour can be tested.
      if (options.requestClass === 'bot') {
          // Allow caching in the CDN for one week
          response.set({
              'Cache-Control': 'max-age=0, s-maxage=604800'
          })
      } else {
          // This 'else' block should set the cache-control header
          // appropriately for non-bot responses.
      }

      That’s it! You will now have a longer cache lifetime for bots.

      Handling unsupported browsers efficiently

      You can also use the request processor as a simple way to distinguish requests by type and version of browser such as Chrome, Firefox, and Internet Explorer. If you have a request from a browser that’s unsupported, you can set a different requestClass for that request, and configure the server side-rendering server to return a page that asks the user to update their browser. Even though the URL is exactly the same, the requestClass allows you to detect requests from browsers that aren’t supported, and modify the page that’s returned.

      Example: handling unsupported browsers

      In the example below, we extend the existing request processor to detect unsupported browsers and set the requestClass to unsupported. We'll make the following changes to the file request-processor.js:

      // First, we define a function that detects the browser type from the user agent
      const isSupported = (ua) => {
          // insert your browser detection logic here!
          return true
      }
      
      // Next, we identify whether the user's browser is supported, and set
      // a requestClass for any unsupported browsers inside request-processor.js
      if (headers && setRequestClass && isSupported(headers.getHeader('user-agent'))) {
          setRequestClass('unsupported')
      }
      

      Using the requestHook, we can set the response page to return a special type of page for the unsupported browser case. Enter the following code in pwa/app/ssr.js:

      if (options.requestClass === 'unsupported') {
          return response.send(`
              <html>
                  <head></head>
                  <body>
                       <h1>You are using an unsupported browser</h1>
                       <p>Please upgrade your browser and try again
                  </body>
              </html>
          `)
      }
      

      Using the request processor

      Follow these steps based on when your project was generated. If your project was generated:

      Important: Ensure you have the lastest Mobify SDK release.

      1. A prerequisite for these instructions include adding new webpack configurations to your project. In your project’s /web/webpack directory, add a file called base.request-processor.js and paste this code:
      // In your project's /web/webpack directory
      
      /* eslint-env node */
      /* eslint-disable import/no-commonjs */
      
      const path = require('path')
      
      module.exports = {
          entry: './app/request-processor.js',
          target: 'node',
          output: {
              path: path.resolve(process.cwd(), 'build'),
              filename: 'request-processor.js',
              // Output a CommonJS module for use in Node
              libraryTarget: 'commonjs2'
          },
          module: {
              rules: [
      
                      test: /\.js$/,
                      exclude: /node_modules/,
                      use: {
                          loader: 'babel-loader',
                          options: {
                              cacheDirectory: path.join(__dirname, 'tmp')
                          }
                      }
                  }
              ]
          }  
      }
      
      1. Then, in the same directory, open the dev.js file and paste this code (follow the instructions in the comments):
      // In your project's /web/webpack directory
      
      // In the dependencies at the top, add...
      const requestProcessorConfig = require('./base.request-processor')
      
      // ...
      
      // Where the configs are added together in an array, add
      // `requestProcessorConfig` to it.
      let configs = [mainConfig, loaderConfig, workerConfig, requestProcessorConfig]
      
      1. Next, open production.js and paste this code (also follow the instructions in the comments):
      // In your project's /web/webpack directory
      
      // In the dependencies at the top, add...
      const requestProcessorConfig = require('./base.request-processor')
      
      // ...
      
      // Where the configs are added together in an array, add
      // `requestProcessorConfig` to it.
      let configs = [
          productionMainConfig,
          baseLoaderConfig,
          workerConfig,
          requestProcessorConfig
      ]
      
      1. Create a request processor: It must be called request-processor.js, and stored in the same directory as your main.jsx file. This is located in the your-project/web/app sub-directory.

      2. Update the processRequest function: within request-processor.js, you must update the function called processRequest so that it sets the requestClass. You will need to update the functions getRequestClass and setRequestClass. For details on parameters and return values, visit packages/pwa/app/request-processor.js.example.

      3. Add code within packages/pwa/app/ssr.js that responds conditionally based on the requestClass you set. For example, you can set a longer cache lifetime for bot requests.

      4. Test your code: the request-processor.js file is also run by the local development server, so it can be tested and debugged locally without having to be deployed.

      Important: Ensure you have the lastest Mobify SDK release.

      1. Your project may need to update its webpack configuration in order to continue. Check your webpack.config.js file and verify whether there is a requestProcessor object.

      2. If there is no requestProcessor object, then paste this code into the bottom of the file (follow the instructions in the comments):

      // Near the bottom of the file, before the `module.exports`,
      // paste in the following code:
      const requestProcessor = Object.assign(
          {},
          {
              entry: './app/request-processor.js',
              target: 'node',
              output: {
                  path: path.resolve(process.cwd(), 'build'),
                  filename: 'request-processor.js',
                  // Output a CommonJS module for use in Node
                  libraryTarget: 'commonjs2'
              },
              module: {
                  rules: [
                      {
                          test: /\.js$/,
                          exclude: /node_modules/,
                          use: {
                              loader: 'babel-loader',
                              options: {
                                  cacheDirectory: path.join(__dirname, 'tmp')
                              }
                          }
                      }
                  ]
              }
          }
      )
      
      // Modify the module.export to include the new `requestProcessor` object
      module.exports = [main, others, ssrServerConfig, requestProcessor]
      
      1. Create a request processor: It must be called request-processor.js, and stored in the same directory as your main.jsx file. This is located at your-project/packages/pwa/app/.

      2. Update the processRequest function: within request-processor.js, you must update the function called processRequest so that it sets the requestClass. You will need to update the functions getRequestClass and setRequestClass. For details on parameters and return values, visit packages/pwa/app/request-processor.js.example.

      3. Add code within packages/pwa/app/ssr.js in the server side-rendering server that responds conditionally based on the requestClass you set. For example, you can set a longer cache lifetime for bot requests.

      4. Test your code: the request-processor.js file is also run by the local development server, so it can be tested and debugged locally without having to be deployed.

      IN THIS ARTICLE:

      Feedback

      Was this page helpful?