Earlier this week, I hosted a webinar on Google Optimize 360, along with Google’s own Jonathan Mesh, Lead Product Manager for Optimize and Optimize 360, and discussed ways to accelerate testing and optimization strategies. We stepped through the power of personalization, shared a real-time demo of Optimize 360, and explored experiments and tactics for delivering more compelling user experiences.
The webinar Q&A session quickly showed the need for practical tips and actionable advice for getting started with testing. Jonathan and I answered as many questions as time permitted, and summarized them below. If you have additional questions, please submit them to marketing@cardinalpath.com and we’ll be sure to respond directly, and add them to this FAQ.
Q: If I am interested in creating a randomly selected control group of users that persists over the period of a test, is that something I would do in Optimize or GA 360?
A: Optimize will allow you to select the proportion of users you’d like to be included in your test. For users included in a test, Optimize will then distribute traffic evenly across the variations, but this is customizable. Once someone has been included in an experiment, they will remain in the experiment until it concludes.
Q: What’s the critical point for Optimize 360 versus a CMS?
A: That depends on your CMS. Some CMS platforms offer some ability to run A/B tests and some level of personalization, though these are often basic functions.
Optimize is not a content platform, like a CMS, but is a platform to test different types of content to understand what works best for each audience (or overall). Once you understand what works, you can then deploy this content through your CMS.
Typically, if you want to start running a testing and personalization program, you want to use a tool that is developed specifically for that, and that gives you the flexibility to run the types of tests you want, and integrates well with the other platforms that you use.
Q: Optimize 360 looks very good for layout, images, front-end changes. What if I want to test pretty sophisticated javascript logic? What about for single-page web apps or AJAX based apps?
Optimize still gives you plenty of power – the visual editor allows you to modify HTML, inject javascript, and use javascript variables and functions as targeting conditions, as well. Our Activation Event functionality makes it easy for Optimize to work well with single-page apps or other dynamic features.
Q: How much does Optimize 360 cost?
Pricing is dependent on the amount of traffic to the Google Analytics Web Properties that Optimize 360 is linked to and can also vary based on the support services you may require. If you are interested in understanding what pricing would look like for your organization, please reach out to Cardinal Path and we’d be happy to give you an outline of costs.
Q: Are there any plans for Google to make this tool and the rest of the 360 suite HIPAA compliant so that customers in the Health Insurance and Healthcare spaces can use these tools more effectively? BigQuery is, but none of the other production in the suite are, to my knowledge. (Eric Berry)
HIPAA applies to protected health information held by health care providers and healthcare clearinghouses. Because Google Optimize and Google Analytics are not such entities, and because both products prohibit customers from sending any Personally Identifiable Information, whether it is protected health information or not, it remains our customers’ obligation to comply with HIPAA where applicable.
Q: Can we test variants of display versions?
If you want to run tests on your AdWords campaigns, you can run AdWords Campaign Experiments directly within that platform.
Q: Does Optimize integrate with third party tools, like CrazyEgg?
There is currently no out of the box integration with Crazy Egg, but Google Analytics integrates with many third party tools and because Optimize is also integrated with GA you can combine these together for analysis within that platform.
Q: Is there any limitation for website with CMS frame work?
This is dependent on the CMS framework. If you can deploy javascript in your <head>, then you should be able to deploy and run Optimize
Q: Is it possible to run tests on AdWords?
You can run tests in AdWords on your various creatives in the AdWords interface using AdWords Campaign Experiments.
Optimize is designed to test content on your website. The new AdWords integration with Optimize will allow you to customize your tests to users that are coming to your site from specific AdWords accounts, campaigns, and keywords.
Q: Who is the best person on a team to run Optimize 360?
Depending on the complexity of your tests and the experience of your team, you may need multiple people.
- Implementation: You need someone who can implement Optimize on your website.
- Creative: You need someone who can create the different variations. This may just be text and very easy, or this may be graphics/images and you may need assistance.
- Analysis & Follow Up: You need someone who will launch and monitor the test, and then guide the required follow up to action on any of the learnings from your tests.
This is a simplified view but gives you an idea of the types of skills you may need.
In some cases this can be one person, in other cases, you may require multiple people depending on the skillsets available within your organization.
If you are the only person, then pick the types of tests that fit with your skillset so that you are able to execute on these efficiently.
Q: Do you recommend starting w/Optimize free and moving to premium from there?
That depends on a few factors, but generally, if you have never had an active testing / personalization program in your organization before and you are just getting started, Optimize Free is a fantastic way to begin. As you mature and want to take advantage of the great features in Optimize 360, then you can look to move to the paid product.
Q: What kind of setup is involved with getting this tool up and running? Can we do this ourselves?
Optimize and Optimize 360 are quite easy to set up and can be deployed through Tag Manager. If you have Google Tag Manager or Tag Manager 360, enabling and activating it can be quite simple. Once you have it deployed across your site, you can now go into the interface and start running tests–just like that. So it’s extremely easy to deploy.
If you don’t have Google Tag Manager or Tag Manager 360, some other tag managers will work, though they do need to support tagging tools. Otherwise, you can do on-page tagging as well.
The other aspect is the need to link this up to Google Analytics. You’ll need administrative access to both accounts to ensure that you can link those together. That’s a pretty simple operation that Google has made relatively easy. There are some performance tradeoffs that you have to consider and we can help you evaluate those.
If you want to deploy within Google Tag Manager vs. putting something more directly on your page, it’s all about what that tradeoff is like and what your site is like and what the right option is for you.
Q: How do I access Optimize reporting in Google Analytics? If I want to drill down and do some customized analysis, what are some of the options available?
If any of you are familiar with the ability to use dimensions in your reports or create custom reports, Optimize data comes in as additional dimensions. If you look in your Dimensions Explorer, you’ll find Experiment ID, Experiment Name, and Variant. And those allow you to really slice and dice almost any report by those experiment dimensions. What you can do is create a Segment that refers to a particular experience, one variant, and then make another one with another variant. You can then look at any given report, segmented by those dimensions.
In addition to using Segments, you can also create Custom Reports with a mix of GA and Optimize dimensions and metrics.
The Optimize dimensions are also exported as part of the Analytics 360 BigQuery export. We have many clients who’ve been doing even deeper dives and performing some of their own analysis, to see which other metrics are being impacted by the tests. Some of those more advanced clients have their own data scientists teams or leverage our team to really dig in, and that’s been really useful for them.
Q: As per the example you showed about the free shipping service, will it only be applicable to web device? Will it be available to apps?
Right now, Optimize works on websites, including desktop websites and mobile websites. Google also support some degree of arbitrary hit sending from other devices but as for apps, they don’t yet support apps. But stay tuned there. There’s more to come. There’s a lot more work that Google is putting into Optimize.
Q: Is there any limitation for websites with a CMS framework?
From my perspective, going back to the questions we had about deploying Optimize, most CMS platforms today support javascript tagging, whether it’s through a tag manager or there’s a way of deploying tags through the CMS platform. The key is just where those get deployed on the page as that can have an impact on how well your variations display. But most CMS frameworks fully support some type of testing platform that you can deploy. Review the implementation guidelines to ensure you can deploy the Optimize tags in the recommended spots on your pages.
Q: What types of items can we target against with the upcoming AdWords integration?
Once AdWords is linked to Analytics, there’s one more checkbox in the AdWords interface to complete the integration. In Optimize, what will then appear are your Accounts, Campaigns, AdGroups and Keywords–and that’s just the start. Google may very well build in more in the future. But that will let you choose the level of granularity that you need, depending on how you structure your campaigns ads. What I’m really excited about, for is the simplicity, in that we do really fast lookups to ensure we’re getting that information from AdWords campaigns and serving up the right experience to those users. You don’t have to do any manual tagging. In fact it works best with auto tagging and you won’t need to do any more set up with AdWords. All of that information will show up in Optimize and it’s very easy to setup and deploy a test to those specific AdWords Accounts, Campaigns, Ad Groups or Keywords.
Landing pages are tied so closely to the tests that you’re running and it’s great to be able to really tailor a single page where you can tailor experiences across a variety of your media and advertising through AdWords.
Q: Any suggestions for minimizing performance impact when implementing optimize?
Google uses what’s known as an asynchronous approach to this. What that means is they’re able to load this in parallel and we’re able to timeout that request. For example, if a user is on a really slow connection, what we generally recommend to the degree possible: there’s one snippet that goes directly on your page which prevents the site from flickering. So it hides the page really briefly until you hit that timeout or we get a response back from Optimize.
Google really wanted to help customers get the best performance on their page. What they’ve generally found–and much research has been done on this–is that performance with Optimize is often faster than the site itself. Often times there’s no need to hide the page at all. And then for implementation what we suggest there, to the degree that you can, is to put some of that information directly on the page. You’re usually able to get a little bit better performance from that. But if you do need to put that in Tag Manager we’ll still see some really good cases of performances, and we’re continuing to find ways to make this as quick as possible and as easy as possible to set up.
Ultimately, it all depends on your individual situation, there are also those folks who may not be running a test on that very first page. Or might be using something like a dynamic web app that perhaps has a set of chained loading, that they’re more interested in getting that first render time rather than having the ability to change the very first page. Or perhaps they’re changing something further down the page and they’re not as worried about things flickering. But for the best performance, we often find that people who do more of that direct implementation see better success with performance.
Check out the other Google Analytics 360 webinars in this series or reach out to ramp up on your own testing and personalization strategy.
