As conversion optimization continues to mature and become adopted by more organizations, it’s always interesting to see how companies are approaching growth and optimization. Especially, for me, in the tech startup space, as these companies often live and die by data, and tend to build their organizations around experimentation.
LawnStarter is one such company, so we sat down with their CTO, Jonas Weigert, to learn about how they experiment across their product and communication and how they deal with optimization as a company.
Introducing Jonas Weigert and LawnStarter
LawnStarter makes it easy to book lawn care. They are a consumer market place where you order a lawn care subscription with just a few clicks. They have a ratings system and build distribution algorithms to best distribute lawn care providers to customers.
In short, they need to approach data and experimentation from multiple fronts, so it gets a bit complicated in how they actually do this, as well as how they balance experimentation with their typical product development process.
Jonas Weigert is the driving force being the majority of these decisions. As LawnStarter’s CTO, he heads their technical efforts, which includes experimentation across their product and communications.
Q: How did you get involved with LawnStarter?
A/B Testing Beyond the Landing Page
Since LawnStarter involves more than the traditional add-to-cart and then complete purchase conversion metrics, they utilize their customer and web data in various channels. They’re a subscription business, so ongoing communications are important. They also experiment frequently with their own product’s functionality.
Q: What does experimentation look like at LawnStarter?
The industry seems to moving further towards universal optimization, or as Optimizely’s tagline puts it, “experiment everywhere.” Instead of surface level changes, organizations are asking how they can use all that data they’re collecting to experiment across their stack and communication channels, making data-driven decisions where previously gut-decisions prevailed.
Whether you use a tool like Optimizely or Conductrics to do that, or build your own platform that is uniquely tailored to your needs, it’s becoming increasingly important to have that level of flexibility and freedom. Add to the fact that Intuit just released an open source testing tool that has these capabilities, and I think we’re going to see a lot more of this approach to optimization in the future.
Optimizing Communications and the Customer Experience
What we usually talk about when we talk about CRO is running experiments on a web interface – most of the time with ecommerce or lead generation, sometimes with SaaS optimization.
LawnStarter, being a marketplace, represents a different set of challenges. But they also push the boundaries in terms of experimentation because they move past the web interface to test their product communications as well. They wrote about it on their engineering blog pretty extensively. Here’s a quick diagram they published that visualizes their engagement tracking for email communications:
Since they’re testing their web interface, app, product communications, and product features, what technology do they use to accomplish all of this?
Q: What do you guys use to run experiments in different parts of your product and communications?
Q: Wow, so you built your own system to do all that – why did you choose to do it this way? Why not use an out of box platform?
So there it is. When testing more prototypical interface and user experience elements, they keep things simple with Optimizely and off-the-shelf reporting. When testing product communications or internal features, things that require a more customized approach, they built their own system to handle it.
The Challenges with Single Page Apps
Single Page Apps (SPAs) are becoming more popular for a variety of reasons, allowing for more dynamic apps and a more seamless user experience. But because the client loads just once, they present technology problems for A/B testing. I asked Jonas about how they tackle these challenges.
Q: Funny that you mention single page apps. I understand they’re becoming more popular. Could you explain why they are so hard to test with? What advice do you have for people dealing with them?
Talking to Jonas, I was most interested in how they approach experimenting with their product features and especially their distribution algorithm, which is how they match customers to lawn care providers.
It’s not as simple as a randomized bucket of web visitors via paid, organic, social, etc. These are live experiments on your current customers, and therefore, there are a whole new set of challenges, including randomization, bucketing, customer experience complexity issues, and a more challenging analysis.
Q: You mentioned testing your distribution algorithm, could you go into a little more depth on what that means? How do you determine if one distribution algorithm ‘won’?
How to Prioritize Experimentation with Little Resources
LawnStarter doesn’t have the resources that American Express has. But they still take experimentation very seriously (as many startups do). However, they’re constantly engaged in a balancing act: how many resources can they feasibly expend with A/B testing as opposed to feature development and other opportunity areas?
This balancing act is fascinating with startups, because they’re constantly required to be scrappy and creative.
Q: Being an early stage company, you must be very resource constrained. How do you justify investing so many engineering resources to A/B testing?
Of course, resource challenges don’t go away when a company has more general resources, talent, and cashflow. In fact, I’ve more frequently seen startups of sparse resources devoting more time and attention to experimentation than the juggernauts of enterprise, who are often slow to act and treat product development much differently.
So the problems of iteration, customer research, and reporting don’t go away with resources. The challenge of maintaining a culture of experimentation are always there.
Wrapping up the conversation with Jonas, I asked him to sum up his advice to fellow tech startups looking to get serious about optimization.
Q: What’s one tip you would give to a company in your stage?
As Jonas mentioned, data is their most valuable asset. But it’s not just the collection, it’s about putting it into action.
Though they have unique challenges according to their industry and business model, there are some more general takeaways here, too.
First, think critically about how you want to prioritize testing and optimization in your company. Do you want to bring experimentation beyond the landing page and really use it to fuel growth? As you scale, how will you empower teams to actually do this? There’s not a one-size-all answer here, but it helps to start the conversation.
Second, try to make things as simple as you feasibly can. Of course, make sure your technology and approach are effective, but if no one on your team can access your data or run tests, then there is an inherent bottleneck. LawnStarter favors the simple solution, as it’s conducive to speed and democratizes testing and data.
That’s what a “data-driven” organization is at heart, right?