Experimentation / CRO

Experimentation and Testing Programs acknowledge that the future is uncertain. These programs focus on getting better data to product and marketing teams to make better decisions.

Research & Strategy

We believe that research is an integral part of experimentation. Our research projects aim to identify optimization opportunities by uncovering what really matters to your website users and customers.

Data and Analytics

90% of the analytics setups we’ve seen are critically flawed. Our data analytics audit services give you the confidence to make better decisions with data you can trust.

How Users Read the NYTimes Website – 2004 vs. 2016 [Original Research]

We found an interesting, and rather old, eye-tracking study from 2004 and decided to try to replicate a part of it to see how it works today.

This study, conducted through CXL Institute, involved eye-tracking a couple homepages of the New York Times, one from this year, 2016, and one from 2004. Our primary goal wasn’t the comparison to the old study, rather it was to see what were the ‘priority viewing areas’ for how people process a news site and to see if ‘today’s users’ process the contemporary design differently than one from more than a decade ago.

Results summary

  • The large banner ads in the 2016 variation made users jump around the page, causing way more variability in how people read compared to the 2004 version.
  • Study participants started staring at the ad in the 2016 version, but quickly went elsewhere, although where they went was much more variable compared to the 'seen path' of the 2004 design.
  • The priority viewing areas did not differ much at all between the two version of the New York Times website, and also agreed with the 2004 study.
  • Given the designs tested, the regions that have more immediate prominence were the center and upper left of the page.

How do I apply this research?

This study was primarily exploratory, as we have follow-up plans looking at reading patterns, however, it does provide us an example of viewer perception path variability. Website owners usually want visitors to quickly process a homepage, get the point, and move further down the sales funnel....whatever that might look like. So if one design over another causes viewers to process things in a more variable way, this can have consequences on visitor flow through a site. So not only do these results show banner blindness, but that banner blindness has consequences on collective user focus and perception.

Background

So the original study that this little test was inspired by was performed by a seemingly out of business research branch of Poynter.org. This study doesn’t necessarily reproduce any part of the old study explicitly, rather, we take their approach and see how it works today.

Study Report

Data Collection Methods and Operations:

Here is the ‘priority viewing area’ aggregate map of 5 news sites analyzed in the 2004 eyetracking study by Eyetrack III. This study only considered user viewing patterns of the news site http://www.nytimes.com/.

Priority viewing areas of news websites, from a 2004 Eyetrack III study
Priority viewing areas of news websites, from a 2004 Eyetrack III study

Our ‘priority viewing area’ grid had to adjust according to the precision of our eye-tracking tool. We used a grid of 3X5 rather than 4X4.

The 2004 version of the New York Times homepage was obtained through the internet archive ‘way back machine’.

Participants

The eye-tracking survey was completed by 200 participants, though only 132 resulted in data with enough accuracy to be useful, 68 for the 2016 version and 64 for the 2004 version.

Findings

Heatmaps of the New York Times online homepages.
Heatmap for the two versions. 2016 on left and 2004 on right.
seen order comparison
Priority viewing areas and seen order for the New York Times homepage in 2016 (left) and 2004 (right).

Conclusion

The large banner ads in the 2016 variation made users jump around the page, causing way more variability in what people read compared to the 2004 version. Study participants started staring at the ad in the 2016 version, but quickly went elsewhere, although where they went was much more variable compared to the ‘seen path’ of the 2004 design.

The priority viewing areas did not differ much at all between the two version of the New York Times website, and also generally agreed with viewing patterns from the 2004 study.  Given the designs tested, the regions that have more immediate prominence were the center and upper left of the page. The variation in the 2016 design was seemingly a result of the banner advertisement from IBM, which caused viewers to essentially skip around it to find and fixate on text content.

Related Posts

Who's currently reading The Experimental Revolution?