At Product Camp 2024, Brazil’s largest product event, Minders made its mark with a content aquarium filled with ideas and inspiration. Among the many presentations, one of the most anticipated was led by Leonard Karic Klovrza, Solutions Architect at Minders. The topic: “Maximizing Results with Experiments: How Tests Are the Key to Personalization” highlighted how experimentation can drive engagement and significantly personalize experiences.
In this article, we explore the key insights from this talk, bringing concepts, practices, and examples that will help professionals in product, martech, growth, and data apply experimentation in their own contexts.
What Is Experimentation and Why Does It Matter?
Leonard began his talk by highlighting the difference between feature flags and A/B tests:
- Feature Flags: Tools that allow the creation of dynamic products without relying on a deploy or heavy development interventions. They are useful for gradually releasing features and mitigating risks.
- A/B Tests: Used to validate hypotheses through controlled comparisons between different variants, based on quantitative and qualitative data.
Leonard emphasized that experimentation is much more than “testing ideas.” It’s a structured process that allows companies to be more dynamic and data-driven.
Step-by-Step to Create Experiments
The speaker shared a practical model for conducting experiments, from hypothesis formulation to result analysis:
- Define a Hypothesis: “What do we believe will happen and why?” The hypothesis should be clear, specific, and outcome-oriented. An example was the test of showing the lowest price for tickets in the Buser app, which led to a significant increase in conversion.
- Establish a Target Audience: Based on precise segmentations, select the right group of users for the experiment. For example, in the case of Gol, the website was personalized to cater to the preferences of Argentine customers.
- Design the Test:
- Use no-code or low-code tools, like Amplitude, to create quick and dynamic experiences.
- Design clear variables for comparison (e.g., control and treatment).
- Use no-code or low-code tools, like Amplitude, to create quick and dynamic experiences.
- Measure Results: Use appropriate metrics like conversion, session time, and bounce rates to assess the impact of the experiment. Remember to aim for statistical significance to ensure result reliability.
- Iterate and Scale: Each experiment provides insights that can be used to create new tests. This continuous cycle mindset is essential for sustainable growth.
Experimentation in Practice: Inspiring Use Cases
Leonard shared real-world cases that demonstrate the power of experimentation:
- iFood: A simple modal notifying users that a restaurant was about to close reduced errors in the order journey by 25%, in addition to increasing conversion rates by 84%.
- Reforge: Turning off the largest acquisition channel on TikTok to test the impact on conversion revealed that results remained stable, significantly reducing paid media costs.
- Gol: Personalizing the website for Argentine customers based on their shopping habits brought greater relevance and efficiency to the user experience.
These examples show how small tests can have a big impact and guide strategic decisions.
Benefits of an Experimentation Culture
Leonard emphasized that experimentation is not just a technique, but a culture that needs to be adopted across the organization. Some key benefits include:
- Data-Driven Decision Making: Avoids assumptions and promotes a more objective approach.
- Documentation and Legacy: Each experiment generates learnings that can be replicated and expanded in the future.
- Greater User Knowledge: Constant testing helps to better understand audience behaviors and preferences.
