Introduction

The marketing team at one of our clients wanted to undertake A/B tests for their landing pages without needing any intervention from the tech team. As they started the A/B tests, we observed that the landing pages under experiment experienced a speed degradation. We then set on course to achieve a setup where:

• A/B tests would not affect the page speed.
• The marketing team would be able to undertake many AB tests without tech team's intervention.

This post details our experience & findings to achieve such a setup.

About Our Site & Setup

To set the context, below are the details of the site setup:

• The marketing team wanted to perform A/B tests for a specific set of landing pages.
• The landing pages are part of the website code. The website front-end is written in React JS (with Server Side Rendering).
• The marketing team already controlled the landing page content via CMS.
• The A/B tests were performed via Google Optimize.
• The A/B tests JavaScript code provided by Google Optimize was integrated into the site via Google Tag Manager.
• An anti-flicker snippet provided by Google Optimize was directly added to our landing page code.
• The A/B tests were focussed on trying different variants of the copy text and display layouts.

Example Landing Page

For the purpose of this blog post, we have created a demo landing page (to avoid disclosing client specific information). We have also set up Google Optimize A/B test experiments for this page. You can see the demo landing page here (without any A/B tests configured for it). Below snapshot shows how this page loads without any A/B tests configured for it:

Why was an Anti-flicker Snippet Required?

When the A/B tests were originally configured, no anti-flicker snippet was added. Google Optimize's JavaScript (that controlled rendering of A/B test variants) was added to the site via Google Tag Manager. However, this resulted in a "flicker effect". The loading snapshot below shows this via our demo landing page:

This flicker effect raised two issues:

• An undesirable page loading experience.
• Questions over sanctity of the A/B test experiment & it's results.

To overcome these two issues, an anti-flicker snippet provided by Google Optimize (here) was added directly to the landing page code.

How the Anti-flicker Snippet Changed Page Loading Behavior

The anti-flicker snippet resulted in the page content remaining invisible untill the Optimize JavaScript came into action. The loading snapshot below shows this change of behavior via our demo landing page:

As seen above, the anti-flicker snippet addressed the two issues highlighted earlier. The undesirable flicker during loading was avoided and the page content was only displayed as controlled by the A/B test tool. This maintained the sanctity of the A/B test experiment.

Speed Impact of Anti-flicker Snippet

A side-effect of adding the anti-flicker snippet was that the page content started to render a lot later. This translated into PageSpeed score degradation of ~16 points for our landing page:

A similar speed degradation can be seen via our demo landing page. As seen in the loading snapshots above, the start render happens at ~3 seconds with the anti-flicker snippet as compared to ~1.5 seconds without it. The below table captures these numbers:

How We Addressed the Speed Impact of the Anti-flicker Snippet

Anti-flicker snippet hides the entire page content till the A/B test tool's JavaScript kicks into action. An alternative to this could be to selectively hide only those parts of the page that are controlled by the A/B test. With this in mind, we created a modified version of the anti-flicker snippet. This resulted in a page-loading sequence as following:

As seen in the loading snapshots above, with the modified anti-flicker snippet:

• The page render starts as early as that without anti-flicker snippet.
• The undesirable flicker is avoided by concealing part of the page under experiment.

With the modified anti-flicker snippet, the start render time got back to 1.5 seconds from the earlier observed 3.0 seconds:

Our Modified Anti-flicker Snippet

Below is the modified anti-flicker snippet we added to our landing pages code directly:

<script>
function hideOptimizeElements()
{
        for (e in googleOptimizeEleList)
                document.querySelector(googleOptimizeEleList[e]).style.opacity=0;

}
function showOptimizeElements()
{
        for (e in googleOptimizeEleList)
                document.querySelector(googleOptimizeEleList[e]).style.opacity=1;
}
hideOptimizeElements();
setTimeout(showOptimizeElements, 4000);
</script>

We requested the marketing team to modify the the code snippet to load Google Optimize as following (notice the onload attribute):

<script async src="https://www.googleoptimize.com/optimize.js?id=<GTM-XYZXYZ>" 
onload="showOptimizeElements();"></script>

As can be seen above, rather than controlling the opacity of the entire document, the above snippet controls opacity of elements within a list googleOptimizeEleList. googleOptimizeEleList is expected to be a list of query selectors for container-element that is part of the A/B test experiment. We leveraged our CMS to populate the value for googleOptimizeEleList.

How We Incorporated Modified Anti-flicker in our Workflow

In order to incorporate the modified anti-flicker snippet, we required a way by which the marketing team could provide the query selector for in-experiment element(s). To achieve this, we requested the marketing team to:

• Extract the query-selector for the container-element from Google Optimize like following:
  
  
  Extracting query selector for an under-experiment element from Google Optimize
• Copy the "Element Selector" from Google Optimize into our CMS.

Our front-end code was setup to fetch the selector from the CMS and add it to the googleOptimizeEleList list. In this way, whenever the marketing configured an A/B test, they performed an additional step of letting the site know the elements to hide via the CMS. The site page concealed only part of the page instead of the complete page.

Limitations of Our Approach

While the solution suggested above is the best-fit for our setup, following are it's limitations:

• While the start render occurs as early as possible with our approach, the Speed Index still happens a lot later. This is because we still need to wait for A/B test tool JavaScript to load. The only way to overcome this would be if the A/B test can be driven entirely on the server-side.
• Our modified anti-flicker snippet requires to be fed a list of query-selectors for the in-experiment elements. This may not be feasible without tech team's intervention in absence of a CMS.
• Addition of the in-experiment elements' query-selector is an additional step for the marketing teams to perform.
• Extracting the in-experiment elements' query-selector may not be simple with all A/B testing tools. We could not verify if A/B test tools other than Google Optimize readily provide this.

Conclusion

A/B testing tools need to evolve to facilitate server-side handling of A/B tests. Untill that happens, an early start render can be achieved via our modified anti-flicker snippet (albiet some restrictions as listed above).

---

About the Author

Punit Sethi has been working with large e-Commerce & B2C websites on improving their site speed, scalability and frontend architecture. He tweets on these topics here.

---