Contents · 9
- What an A/B test is and when to run one - and when NOT to
- Statistical significance, sample size and the p-value in plain language
- A proper hypothesis - the structure "Change X will raise conversion by at least Y%"
- What to test in what order - the priority hierarchy
- A/B testing tools in 2026 - after Google Optimize
- The most common A/B testing pitfalls - 7 deadly sins
- Multivariate testing - when to use it instead of A/B
- Real case studies of Wrocław companies - 3 tests from the Dekada72H portfolio
- Practical conclusions - how Dekada72H runs A/B testing for Wrocław clients
A/B testing sounds simple - you show two variants of a page to different users, measure which converts better, and deploy the winner. In practice 80% of A/B tests run by Polish companies give false results, and another 15% test things that don't matter. Only 5% are tests that are correctly designed statistically, run with discipline and deployed on the basis of real data. The rest is optimisation theatre that burns budget and gives the illusion of work.
The problem isn't the tools - VWO, Optimizely, Convert.com or a custom solution on Next.js are all technically great. The problem is methodology. Most marketing teams stop tests after 3 days, when "a winner is visible". Most A/B tests have a sample size of 200 visits per variant instead of 2000. Most reports of "variant B wins by 12%" are a pure statistical lottery. And the conclusions from those tests then drive decisions about months of work on the landing page - decisions based on false data.
This guide will show you how to run A/B tests that are statistically correct, with the limited traffic typical of Wrocław B2B, B2C and e-commerce companies. I'll explain what the p-value, sample size and statistical significance are in plain language - no maths degree required. I'll show the 2026 tools after Google Optimize was shut down, the hierarchy of test priorities (headline > CTA > social proof > form > micro), the 7 most common pitfalls (peeking, multiple testing, sample ratio mismatch, no segmentation) and when to use multivariate testing instead of A/B. Plus real case studies from Dekada72H clients in Wrocław.
A/B testing is the foundation of optimising a Google Ads landing page - without it, conversion optimisation is reading tea leaves. With it, it is precision engineering where every change is measurable, every decision documented and every uplift repeatable.
40%
30%
14 days
What an A/B test is and when to run one - and when NOT to
An A/B test is a controlled experiment in which you randomly split traffic into two groups: variant A (the control) and variant B (the change). Both groups see the same page with one controlled difference - a different headline, a different CTA, a different form layout. After collecting a sufficient sample size you compare the conversion rate in both groups and statistically verify whether the difference is real or statistical noise.
The key word is controlled. An A/B test differs from "yesterday we changed the headline and the conversion rate went up" in the same way that a clinical RCT differs from "I ate more vitamin C and my cold went away". External variables - the day of the week, a marketing campaign, seasonality, weather - affect the conversion rate far more strongly than most changes on a landing page. Without a parallel comparison (A vs B at the same time, on the same traffic) you don't know whether the headline change helped, or whether Monday simply converts better than Friday.
An A/B test makes sense when you meet five conditions at once. First - you have enough traffic (at least 200-500 visits a week). Second - you have a defined conversion metric with event tracking (GA4, Mixpanel, custom). Third - you have a data-based hypothesis (heatmaps show a problem, a survey points to an objection, analytics reveals a bottleneck). Fourth - the patience to wait 14-28 days. Fifth - the discipline not to stop the test early.
When does an A/B test NOT make sense? The first case - low traffic (below 200 visits a week). The test would then take 6-12 months, and in that time the landing page ages, the offer changes, the market context evolves. It is better to do qualitative research - heatmaps, session recordings, interviews with 5-10 customers, post-conversion surveys. These methods give insights after 5-15 observations, not after 5000.
The second case - testing things with an obvious result. "Will adding social proof above the fold raise the conversion rate?" - yes, we know that from 1000 industry tests. "Will changing from a 12-field form to a 4-field one raise the conversion rate?" - yes, we know. You deploy those changes without a test, because testing costs time and traffic and the result is certain. Test uncertain things - whether headline X beats headline Y, whether social proof as logos or as numbers works better, whether the CTA "Book a consultation" or "Check availability" converts higher.
The third case - testing under time pressure. The client says "we need to raise conversion by the end of the month". In 4 weeks you won't manage a rigorous A/B test with 5000 visits per variant. Instead do a holistic redesign based on best practices and heatmap data - deploy 5-7 changes at once, accept that you don't know which one worked, but the conversion rate will rise. A/B testing is a long-term strategy, not a quick fix.
The fourth case - testing different offers (price, guarantee, deliverable). It is technically possible, but it introduces legal and ethical complications. A customer who sees one variant with one price and another seeing a different price feels cheated if they find out. Run a price test on separate campaigns (different UTMs, different landing pages), not as a split test on the same URL.
In most cases the first 6-8 weeks of a new landing page should not be A/B tested - they should be optimised based on best practices, heatmaps and analytics. You start A/B testing when the landing page is "good enough" (a conversion rate above 4-5%) and every 1% is real money. For Google Ads for small businesses with a limited budget this is often the 2nd-3rd month after launching the campaign.
Statistical significance, sample size and the p-value in plain language
The statistics of A/B tests intimidate most marketers - formulas, Greek letters, density charts. But the foundation is simple and can be explained in 4 concepts.
The first concept - the null hypothesis (H0). By default you assume there is no difference between variant A and B. Your goal in the test is to prove with statistical certainty that the null hypothesis is false - that is, that a difference exists. This approach, called null hypothesis significance testing (NHST), has been the standard for 100 years.
The second concept - the p-value. It is the probability of observing a result at least as extreme as yours, assuming the null hypothesis is true. In plain words: how big the chance is that the observed difference between variant A and B is pure coincidence rather than a real effect of the change. A p-value of 0.05 means a 5% chance the visible difference is random - that is, 95% certainty that the change was real.
The 0.05 threshold (95% confidence) is a convention going back to Sir Ronald Fisher in 1925 - not a magic number, just a historical compromise between false positives and false negatives. For low-risk decisions 0.05 is the standard. For high stakes (a deployment affecting a year of campaigns) aim for 0.01 (99% confidence). For pilots and exploration 0.10 may be enough - as long as you know you're doing exploration, not confirmation.
The third concept - the power of the test (1 - beta), usually set at 0.80. It is the probability that if a difference between variants exists, your test will detect it. A power of 0.80 means: if variant B is really better by 20%, you'll catch it in 80% of tests and miss it in 20%. A power below 0.80 is an underpowered test - even if the change works, you'll too often fail to detect it.
The fourth concept - the minimal detectable effect (MDE) and sample size. MDE is the minimal difference between variants that the test will be able to detect statistically. The smaller the MDE you want to detect, the larger the sample size you need. A rule of thumb:
For a baseline conversion rate of 5% and a desired uplift of 20% (so variant B would have a 6% conversion rate) you need about 1500-2000 visits per variant at power 0.80, alpha 0.05. For the same 5% baseline and a 10% uplift (a 5.5% conversion rate) you already need 6000-8000 visits per variant - 4x more, to detect a difference half as large. For a baseline of 2% (typical for B2B SaaS) and a 20% uplift you need 5000-7000 visits per variant.
The practical implications for Wrocław companies: if your landing page gets 200 visits a week, an A/B test with a 5% baseline and a desired 20% uplift takes 15 weeks (1500 visits per variant at a 50/50 split). If you get 50 visits a week, the same test takes 60 weeks - a year and a quarter. Most businesses don't have that patience and should instead focus on traffic (more Ads, more SEO) before introducing A/B testing as a process.
The sample size calculators we use for Dekada72H clients:
- Evan Miller's calculator (the simplest, free): https://www.evanmiller.org/ab-testing/sample-size.html
- The VWO sample size calculator (with a Bayesian option)
- The Optimizely sample size calculator
- The Statsig calculator for custom implementations
An important trap: calculators assume you split traffic 50/50 for the whole period of the test. If you change the split (e.g. 70/30 for safety), the required sample size rises. If you run an A/B/n test (3+ variants), the sample size is multiplied by the number of variants. If you segment post-test, the sample size per segment must meet the requirement.
A proper hypothesis - the structure "Change X will raise conversion by at least Y%"
Most A/B tests in Polish companies begin with "let's try" - "let's try a green button instead of a blue one", "let's try a shorter headline", "let's try changing the hero photo". That is an anti-hypothesis - no structure, no data, no measurable goal. A "let's try" test usually ends with "well, this variant is a bit better" and builds no organisational knowledge.
A proper A/B test hypothesis has four components.
First - an observation based on data. A concrete insight from heatmaps, analytics, surveys or user testing. "The Hotjar heatmap shows 73% of users don't scroll below the fold - we lose them in the first 8 seconds". "Session recordings show 40% of customers type in their email and abandon the form at the 'company' field". Without a concrete observation a hypothesis is a fishing expedition.
Second - the proposed solution. A concrete change meant to address the problem. "We'll change the headline from the generic 'A professional SEO agency' to the specific 'An SEO audit that shows 30+ concrete fixes in 14 days - free'". "We'll shorten the form from 8 fields to 3 (name, phone, email), and collect the extra info in a follow-up call".
Third - the predicted impact (quantified). "We expect scroll depth to rise by at least 30% and the conversion rate by at least 15%". Without a quantified expectation you don't know whether the test "succeeded". A 2% rise in conversion rate can be a success (if you expected 1.5%) or a failure (if you expected 20%).
Fourth - the rationale (justification). Why do you think this change will work? "A specific headline with a concrete number works better in 14 of 18 CXL Institute case studies - a median uplift of 22%". "A short form converts 2-4x better in Unbounce 2023 and Marketingsherpa 2024 tests". A hypothesis without a rationale is gambling, a hypothesis with a rationale is engineering.
The full structure of a hypothesis:
Observation: [concrete data]
Solution: [proposed change]
Expected impact: [quantified uplift]
Rationale: [why it will work]
Primary metric: [conversion rate / leads / revenue]
Second-order metrics: [scroll depth, time on page, form starts]
Sample size required: [N visits per variant]
Duration: [N days]
An example of a full hypothesis from the Dekada72H portfolio:
Observation: Microsoft Clarity heatmaps of a Wrocław law firm client's landing
page show 67% scroll abandonment before the first social proof. 41% of sessions
end in under 12 seconds time on page.
Solution: Move the social proof (3 client logos + the statistic "we serve 87
companies from Wrocław") from the second screen to above the fold, directly
under the headline.
Expected impact: An increase in scroll depth of at least 25%, a rise in
conversion rate of at least 20%.
Rationale: 7 CXL case studies from 2023 show a median uplift of 18-32% when
moving social proof above the fold. For the legal industry trust signals are
critical right after the headline (median of 5 Marketingsherpa tests, 2024).
Primary metric: form submission rate per visitor.
Second order: scroll depth, time on page, form starts.
Sample size required: 1500 visits per variant (baseline CR 4.2%, target 5.0%,
power 0.8, alpha 0.05).
Duration: 21 days at 200 visits a day.
A hypothesis in this structure has three values. First - it forces you to think before the test, instead of "let's throw something in and see". Second - it allows a post-test analysis of "what met expectations, what didn't" - it builds organisational knowledge. Third - it is presentable to management and clients - a clear structure instead of "we're testing various things".
Every A/B test we run for Dekada72H clients has a documented hypothesis before it starts. It's an overhead of 30-60 minutes before the test, but it gives 10x better conclusions after it. For increasing website conversion, hypothesis-driven testing is the difference between CRO as science and CRO as voodoo.
What to test in what order - the priority hierarchy
Time and traffic for A/B testing are limited. Every test takes at least 14 days and eats 3000-15000 visits. If you have 1000 visits a week, you can run 8-12 tests a year - and no more. This limited pool of tests must be invested in the changes with the highest potential uplift, not in "button colour".
The hierarchy of impact on conversion rate, based on 200+ Dekada72H client campaigns and a meta-analysis of 1000+ industry tests:
Tier 1 (uplift 15-30%) - The above-fold headline. The headline is 5-9 words that decide the first impression. A generic headline "A professional marketing agency in Wrocław" converts dramatically worse than a specific "We'll raise your organic traffic by 40-180% in 6 months - a free SEO audit". The headline test is the first thing you test. Iterations: specific vs generic, benefit vs feature, long vs short, with vs without numbers, friendly vs authoritative tone. The median uplift from a successful headline change is 18-25%.
Tier 2 (uplift 10-20%) - The main CTA copy. Not the colour, not the shape - the copy. "Learn more" converts 30-50% worse than "Book a free consultation". "Submit" converts 20% worse than "Get a quote in 24h". Iterations: action verb (get, check, see, book), benefit-driven (see the pricing, get the report, join 287 companies), risk-reduction (a free consultation, no obligation, in 24h). The median uplift is 12-18%.
Tier 3 (uplift 10-20%) - Social proof: type and placement. Client logos vs numbers vs testimonials vs Google Reviews - each variant works differently for different industries. B2B services: well-known client logos dominate (uplift 15-25% vs no logos). E-commerce: numeric testimonials plus review stars (uplift 10-15%). Local services: photos + name + city (uplift 12-20%). Test: position (above the fold vs below), format (logos vs numbers vs reviews), number of elements (3 logos vs 8 logos).
Tier 4 (uplift 10-15%) - The hero image / video. A stock "businessman in a suit" vs a real dashboard screenshot vs a case study video. A hero video with autoplay (muted) usually beats a static image in 60-70% of tests - but the video weight must be optimised (under 2MB, lazy loaded). Iterations: stock vs real, image vs video, person vs product, abstract vs concrete. The median uplift of a successful hero change is 8-15%.
Tier 5 (uplift 10-15%) - The number of form fields. Every additional form field lowers the conversion rate by 5-15%. The standard test: 8 fields vs 4 fields vs 2 fields. As a rule 3-4 fields wins for lead gen, 1-2 fields for a newsletter, 5-7 fields for a qualified lead in B2B (a conscious trade-off for quality). Iterations: number of fields, field order, field type (input vs select vs radio), microcopy under the fields.
Tier 6 (uplift 5-15%) - Body copy structure. Length, hierarchy, sub-headings, formatting. A short landing page (500 words) vs a long one (3000 words) - for impulse purchases the short one wins, for high-ticket the long one wins. Iterations: length, structure (paragraphs vs bullets), tone (formal vs conversational), first person vs third.
Tier 7 (uplift 5-10%) - Trust elements. Guarantees (a money-back guarantee), policies (GDPR, terms), security badges (SSL, payment), awards (Forbes, Bloomberg). Often underrated - adding a "100% money-back guarantee within 14 days" to the landing page of a coaching client in Wrocław gave an 18% rise in conversion rate.
Tier 8 (uplift 1-5%) - Micro-optimisations. Button colour, font, padding, border radius, micro-animations. The classic "A/B testing 101" topic, but the real impact is minimal. Test these only when your conversion rate is above 8%, you have everything from Tier 1-7 optimised, and every 1% is real money.
A practical sequence for a new Dekada72H client landing page:
- Weeks 1-2: launch the baseline landing page, collect data (heatmaps, analytics, recordings)
- Weeks 3-4: identify Tier 1 hypotheses (headline issues from heatmaps)
- Weeks 5-7: Test 1 headline A vs headline B
- Weeks 8-9: analysis, deploy the winner, the next hypothesis
- Weeks 10-13: Test 2 main CTA copy
- Weeks 14-16: Test 3 social proof position
- ...
The cycle takes 6-12 months to get through Tier 1-5. That is a real timeline - shortcuts end in fake winners. For improving website UX the same hierarchy of priorities applies to UX optimisation.
A/B testing tools in 2026 - after Google Optimize
September 2023 is an important date - Google shut down Google Optimize, its free A/B testing tool integrated with Google Analytics. That forced the market to reorganise - small companies that previously had a free tool had to find paid alternatives, free open-source ones, or ones built into CMS platforms.
The 2026 market is split into four tiers of tools. We deliberately don't quote prices - plans change often, so check the vendors' current pricing.
Enterprise tier:
VWO Testing is the market leader for mid-to-enterprise. A full platform with testing, personalisation, segmentation, advanced reporting and AI-powered targeting. The Bayesian engine allows early decisions. Its strongest side: an ecosystem integrated with heatmaps, surveys and recordings - all in one dashboard.
Optimizely Web (formerly Optimizely Classic) is the other leg of the enterprise market. Strongest in the corporate segment - integrations with large CMSs (Salesforce, Adobe, Sitecore), advanced personalisation, multi-armed bandits. A steep learning curve, and developer support required.
Adobe Target is for companies in the Adobe Experience Cloud ecosystem. Strong in personalisation and targeting, weaker in pure A/B testing. Priced by negotiation, together with the rest of the Adobe stack.
Mid-market tier:
Convert.com is my personal recommendation for mid-market. Solid tools with Bayesian statistics, segmentation and heatmap integration. A simpler UX than VWO. For Dekada72H clients in Wrocław with a moderate budget - the sweet spot.
AB Tasty is a French competitor of VWO, popular in Europe. Strong personalisation and targeting tools, integration with large e-commerce stacks (Shopify Plus, Magento, Salesforce Commerce).
Kameleoon is another European tool with a good AI engine and predictive personalisation. They offer a free tier for small companies (up to 30,000 visits a month).
Crazy Egg Test is known mainly for heatmaps, but also offers basic A/B testing. Limitations: only a visual editor (not code), no Bayesian engine, simple statistics. OK for small companies with simple tests, but you quickly outgrow it.
Free / Open-source / Self-hosted tier:
Microsoft Clarity is a free heatmapping tool with experiments on the roadmap (the latest update in 2025: Clarity Experiments in preview). It is not yet a full A/B testing tool, but heatmaps + recordings are a must for every landing page. Microsoft is investing heavily in it - we expect a full-value free A/B testing tool in 2026/2027.
PostHog Self-Hosted is a full free solution for developers. Open source, can be hosted locally. It offers analytics, A/B testing, feature flags and session recordings. Pro: full control, full customisation. Con: requires developer resources to set up and maintain. For Dekada72H with our Next.js stack - the sweet spot.
GrowthBook is a younger competitor to PostHog, focused only on A/B testing and feature flags. Open source, free self-hosted, paid cloud. A Bayesian engine, sample ratio mismatch detection, and Mixpanel/Amplitude integration. It integrates great with a Next.js and React stack.
Statsig is a developer-focused feature flags + A/B testing platform. A free tier for small uses, paid for bigger scale. Its strongest side: a code-first approach, full SDKs for every framework, advanced experimentation.
Custom tier (Next.js / React in-house implementations):
For developer teams (like Dekada72H), custom A/B testing on Next.js is often the best ROI. The setup:
- Feature flags via LaunchDarkly, Statsig, Unleash, or custom (cookies/localStorage)
- Splitting via middleware (Next.js 15+ has native support)
- Tracking via GA4 events, Mixpanel, or a custom warehouse (BigQuery, Snowflake)
- Statistical analysis in a Python notebook (scipy, statsmodels) or ready-made libs (the Spotify confidence package)
Cost: no licence fees plus developer setup time (10-30 hours). Scaling: unlimited. Limitations: requires developer know-how, and there's no no-code editor for the marketing team.
A practical recommendation for Wrocław companies:
- A small company, up to 1000 visits/week: Microsoft Clarity (heatmaps + recordings) plus manual A/B on different URLs. An A/B testing tool is not needed yet.
- A mid-sized company, 1000-10000 visits/week, no developer team: Convert.com or AB Tasty.
- A mid-sized company with a developer team: PostHog self-hosted or Statsig + a custom Next.js implementation.
- Enterprise (10,000+ visits/week): VWO Testing or Optimizely Web.
Whichever tool you choose, the foundation is: a planned sample size, one test at a time per section, the discipline not to peek, documented hypotheses, and segmentation in post-analysis. The tool is only a platform - the methodology is yours.
The most common A/B testing pitfalls - 7 deadly sins
Over the last 5 years we have audited the A/B tests of clients who came to Dekada72H with the context "we did testing, but it didn't help". In 90% of cases the problem lies in one of the mistakes below - sometimes several at once.
Pitfall 1 - Peeking. The most common and most destructive mistake. A test planned for 21 days, the team looks every day. On day 4 variant B leads by 18%, "we're winning", and we stop the test. The problem: stopping early raises the false positive rate from the declared 5% to 20-40%. Half of the "winners" are really statistical noise that will disappear in a longer test. The fix: plan the sample size, launch, and DON'T look at the primary metrics until the end. Look only at the sample ratio (whether SRM is occurring) and obvious bugs.
Pitfall 2 - The multiple testing problem. You run 10 tests at once. Statistics says: one in ten will return a "significant result" by chance, even if no variant is actually better. This is the familywise error rate problem. The fix: the Bonferroni correction - for 10 tests lower alpha from 0.05 to 0.005 (i.e. require 99.5% confidence). Or: limit the number of simultaneous tests. Or: use false discovery rate control (Benjamini-Hochberg).
Pitfall 3 - No segmentation in the analysis. The test shows "variant B wins by 5%". But after segmentation: on mobile B wins by 25%, on desktop A wins by 15%. That's Simpson's Paradox - the aggregate result is misleading. The fix: plan segmentation before the test (mobile/desktop, source/medium, geography), and analyse results per segment. Watch out for the multiple-testing problem - if you check 10 segments, one will "turn out significant" by chance.
Pitfall 4 - Ignoring mobile. 70% of Google Ads traffic in 2026 is mobile. Yet an A/B test is often checked only on desktop ("because it's easier"). A "winner" on desktop can be a loser on mobile - and vice versa. The fix: every test must be verified mobile-first. Heatmaps and session recordings for mobile separately from desktop. Conversion tracking per device.
Pitfall 5 - Short tests (under 14 days). "A 7-day test gave a statistically significant result, we're deploying". The problem: weekly cycles (Monday vs Friday, weekend vs weekday) affect the conversion rate. A 7-day test may capture only one cycle, so days dominated by the weekend can change the results. The fix: at least 14 days, ideally 21-28. A full 2-4 weekly cycles.
Pitfall 6 - Sample Ratio Mismatch (SRM). The test was meant to be 50/50, but after 5000 visits you have 2700 in variant A and 2300 in variant B - that's SRM. Something broke in the randomisation - a bug in cookie tracking, a redirect to a slow landing page, a script timing problem, a CDN cache issue. With SRM any statistical analysis is worthless. The fix: a chi-square test for observed vs expected split. All serious A/B tools (VWO, Optimizely, Statsig) warn automatically. Custom implementations often miss it - check manually.
Pitfall 7 - Optimising conversion rate instead of LTV. Variant B has a 6% conversion rate, variant A has 5%. B wins, right? Maybe not. If leads from variant B convert 30% worse at later funnel stages (to sale), variant A actually generates more revenue. The fix: track post-conversion metrics. Lead quality, conversion to paid, average order value, customer lifetime value. Sometimes a lower conversion rate from better qualification means higher revenue.
Bonus - Pitfall 8 - Confirmation bias in the analysis. The test sponsor has the hypothesis "headline X will win". The test gives mixed results. The person interprets "headline X won in the mobile segment" (cherry-picking) and deploys the variant. The fix: the hypothesis and success criterion defined BEFORE the test. A pre-registered analysis plan. Stick to it, even when the results are boring.
The practical quality check of every A/B test that we use at Dekada72H:
[ ] Hypothesis documented before the test
[ ] Sample size calculated and planned
[ ] Duration at least 14 days
[ ] One test at a time in a given section
[ ] Heatmaps + recordings for qualitative context
[ ] Daily check only: SRM and obvious bugs
[ ] Segmentation planned before the test
[ ] Multiple testing correction applied if >5 tests in parallel
[ ] Post-conversion metrics tracked (LTV, conversion to paid)
[ ] Decision based on pre-defined success criteria
Every test run without this list is gambling, not science. Better fewer tests of better quality than many tests of poor quality - because many poor-quality tests mean many fake winners deployed to production that lower the actual conversion rate.
Multivariate testing - when to use it instead of A/B
Multivariate testing (MVT) tests several variables at once in different combinations. A classic example: 3 headline variants x 2 CTA variants x 2 hero image variants = 12 variants in total (A/B/C * D/E * F/G). Whereas an A/B test shows "which headline is better", MVT shows "which combination of headline + CTA + hero is best" - including interactions between variables.
A practical example of an interaction: a "specific numeric" headline works best with a dashboard hero image (combo X), but an "emotional benefit" headline works best with a happy customer hero image (combo Y). An A/B test of each variable separately misses this interaction - MVT catches it.
It sounds great - so why not use MVT all the time? Three reasons.
Reason 1 - the sample size multiplies. An A/B test with 2 variants and a baseline conversion rate of 5% needs 1500 visits per variant (3000 in total). A 3x2x2 MVT = 12 variants needs 1500 visits for each of the 12 variants = 18,000 visits at a minimum. Realistically more, because a small sample per variant carries more noise. For a landing page with 200 visits a week, a 12-variant MVT takes 90 weeks - nearly 2 years. Pointless.
Reason 2 - statistical complexity. MVT analysis is complicated. A multiple-testing correction must be applied to all pairs of variants. Interactions need ANOVA or multi-way analysis, not a simple t-test. Most organisations don't have the stat capabilities to analyse MVT correctly - the result is fake significance and wrong deployments.
Reason 3 - implementation complexity. 12 variants means 12 separate URLs or 12 separate template variations in the tool. Every variant has to be QA'd. A bug in one variant spoils the whole test. The maintenance overhead is significant.
When does MVT make sense? Under two conditions at once.
First: huge traffic - at least 50,000-100,000 visits a month to the landing page. Then 12 variants get enough data in a reasonable time (4-8 weeks).
Second: an interaction hypothesis - a strong suspicion that the variables interact rather than act independently. Based on earlier A/B tests or qualitative research. "From our 4 A/B tests we conclude that the headline affects CTA effectiveness, so MVT is worth checking".
A practical alternative to MVT - sequential A/B testing.
For most organisations (including all Dekada72H clients) sequential A/B testing works better than MVT. The logic:
- Test 1: First optimise the highest tier (headline). 14-21 days.
- Test 2: After deploying the winner, optimise the next tier (CTA). 14-21 days.
- Test 3: After deploying the winner, optimise the next tier (social proof). 14-21 days.
- ...
In total: 3-9 months to get through Tier 1-5. Cumulative uplift: usually 50-150% from baseline. Sample size: realistic for a landing page with 1000-5000 visits a week.
Sequential A/B testing misses interactions, but in 95% of cases interactions are second-order - the combined effect is greater than the sum of the separate changes thanks to synergy, but the difference between "A+B+C" and "the best combination of A, B, C chosen after MVT" is usually 5-15%. For most companies that margin doesn't repay the complexity of MVT.
An exception - multi-variant feature flags in SaaS. If you are building a product where the possible feature combinations are a multiplication, MVT (or a more advanced multi-armed bandit) may make sense. But that is a different use case from landing page optimisation.
For Dekada72H clients in Wrocław the recommendation: sequential A/B testing for the first 12-18 months. Consider MVT only when the landing page has 10,000+ visits a week and sequential testing has exhausted the Tier 1-5 hierarchy.
Real case studies of Wrocław companies - 3 tests from the Dekada72H portfolio
Concrete examples of A/B tests we ran for clients in Wrocław in 2024-2025. The data is anonymised (company names changed), but the numbers are real.
Case 1 - the law firm "PrawoXYZ" Wrocław (July-August 2025)
Context: a Google Ads campaign for "divorce Wrocław", "debt cancellation Wrocław", "alimony Wrocław". Monthly ad budget 6,200 zł. Baseline landing page conversion rate - 3.4%. Test goal: raise the conversion rate above 5%.
Hypothesis: Microsoft Clarity heatmaps show 67% scroll abandonment before the first social proof. 41% of sessions end under 12 seconds. Hypothesis: moving social proof above the fold will increase scroll and conversion rate by at least 20%.
Variant A (control): a standard landing page with headline + sub + CTA + form, social proof on the third scroll screen. Variant B: an identical landing page, social proof (3 client logos + the statistic "we've served 87 Wrocław companies" + 3 testimonials) moved above the fold, directly under the sub-headline.
Sample size required: 1500 visits per variant (baseline 3.4%, target 4%, power 0.8, alpha 0.05). Planned duration: 21 days.
Results after 21 days:
- Variant A: 1547 visits, 53 conversions, conversion rate 3.43%
- Variant B: 1521 visits, 86 conversions, conversion rate 5.65%
- Uplift: +64.7%
- P-value: 0.0023 (highly significant)
- Scroll depth: A median 42%, B median 71% (a huge improvement)
Conclusion: the hypothesis was confirmed with a stronger effect than expected (20% expected, 64.7% actual). Variant B was deployed to production. Additional savings: Quality Score rose from 5 to 8 thanks to better engagement metrics. CPC fell from 14 zł to 8.50 zł. The ROI from the test: 240% in the first quarter after deployment.
Case 2 - the online shop "ButyXYZ" Wrocław (October 2025)
Context: an e-commerce shop selling work and specialist shoes. A Google Shopping + Search campaign. Monthly ad budget 12,000 zł. Baseline product page conversion rate - 1.8%. Goal: raise conversion rate and AOV.
Hypothesis: session recordings show that 53% of users leave the product page without interacting with the "technical specification" section. Hypothesis: moving the key specs (sizes, material, safety certificates) to the first screen will increase scroll engagement and conversion rate by at least 15%.
Variant A: a standard product page with hero image + price + add to cart + tabs (description, specs, reviews) below. Variant B: hero + price + add to cart + a bullet list of 5 key specs above the fold + the rest below.
Sample size required: 4000 visits per variant (baseline 1.8%, target 2.07%, power 0.8, alpha 0.05). Planned duration: 28 days.
Results after 28 days:
- Variant A: 4127 visits, 76 conversions, conversion rate 1.84%
- Variant B: 4081 visits, 79 conversions, conversion rate 1.94%
- Uplift: +5.4%
- P-value: 0.42 (NOT significant)
Conclusion: the hypothesis was rejected. The 5.4% difference is below the minimal detectable effect and statistically insignificant. Decision: return to baseline, run qualitative research (user interviews, surveys) to better understand the decision-making process.
The educational aspect of the case: a negative result is valuable. Without this test the client could have spent months optimising "specs above the fold", thinking it was the key - actually it isn't. Sample ratio mismatch did not occur, and segmentation (mobile vs desktop) did not reveal a winner in any segment either. The real signal: not all hypotheses work, and A/B testing gives an honest answer.
Case 3 - the business coach "MarcinX" Wrocław (January-February 2026)
Context: a one-person service business selling a coaching programme. A Google Ads campaign for "business coaching Wrocław", "business mentor Wrocław". Monthly ad budget 3,500 zł. Baseline conversion rate 1.2% (typically low for high-ticket).
Hypothesis: the headline "Professional business coaching Wrocław" is generic. Hypothesis: a specific outcome-driven headline with a concrete pathway will raise the conversion rate by at least 25%.
Variant A (control): the headline "Professional business coaching Wrocław - 12 premium sessions for CEOs and business owners". Variant B: the headline "Break out of your 12-month revenue plateau - coaching for founders of companies at a six-figure monthly turnover".
Sample size required: 8000 visits per variant (baseline 1.2%, target 1.5%, power 0.8, alpha 0.05). With traffic of 60 visits a day the test would take 9 weeks.
Results after 63 days:
- Variant A: 1923 visits, 23 conversions, conversion rate 1.20%
- Variant B: 1890 visits, 41 conversions, conversion rate 2.17%
- Uplift: +81.0%
- P-value: 0.012 (significant)
- Average order value: A and B the same
- Lead quality (sales call to closed deal): A 18%, B 31% (better quality; the specific headline in B pre-qualifies)
Conclusion: the specific outcome-driven headline beat the generic one dramatically. Plus: the leads from B were higher quality (closing rate 31% vs 18%) - because the headline pre-qualified customers at a specific stage of business. Variant B was deployed, with significant additional revenue in the first quarter after deployment (the difference of 18 leads x 31% closing x the programme price).
Cross-case conclusions:
- Cases 1 and 3 show an uplift of 64-81% from a single Tier 1 test (social proof above the fold, a specific headline)
- Case 2 shows that even a seemingly "logical" hypothesis may not win - A/B testing requires humility
- All tests lasted at least 21 days despite limited traffic - statistical significance requires patience
- Lead quality (post-conversion metrics) is often more important than the aggregate conversion rate
These three examples are a typical pattern - 60-70% of A/B tests give a positive result, 20-30% a null result, 5-10% a negative one (the variant is worse than the control). A negative result is also valuable - it says "don't invest in this direction".
Systematic A/B testing, based on a solid methodology, leads to compound improvements. A 20% uplift every 3 months for a year = a 110% cumulative uplift = more than doubling the conversion rate. For technical SEO and page speed optimisation the same compound logic applies - small systematic improvements > radical one-off redesigns.
Practical conclusions - how Dekada72H runs A/B testing for Wrocław clients
A/B testing isn't magic, just precision engineering. It requires discipline, methodology and patience - things most companies running tests on the side lack. At Dekada72H we have developed a process that eliminates 90% of the pitfalls described above and gives consistent, repeatable uplifts for clients in the long term.
Step 1 - the foundation phase (weeks 1-8 after the landing page launches). We don't A/B test in this period. Instead: Microsoft Clarity heatmaps, session recordings, post-conversion surveys (5 questions), a Google Analytics deep dive. The goal: understanding customer behaviour, identifying bottlenecks, baseline metrics.
Step 2 - the hypothesis pipeline (week 9+). A list of 10-20 documented hypotheses from the foundation phase, sorted by expected impact (the Tier 1-8 hierarchy). Each hypothesis has the full structure (observation, solution, expected impact, rationale, sample size, duration).
Step 3 - sequential testing. One test at a time per section. A test lasts at least 14 days, with a planned sample size and anti-peeking discipline. After the test: full analysis (aggregate + segmentation + post-conversion metrics), deployment of the winner or keeping the baseline.
Step 4 - knowledge management. Every test (winning, neutral, negative) is documented in a testing log. It builds organisational knowledge - "a specific headline always beats a generic one in our industries", "social proof above the fold usually gives a 15-25% uplift for legal/financial". The conclusions are applied to further clients - shortening the learning curve of testing for every new landing page.
Step 5 - cumulative improvement. After 12 months of systematic testing, a typical Dekada72H client landing page has a conversion rate 2-3x higher than the launch baseline. That is a compound effect - 5-7 tests a year, each with a 15-30% uplift, gives more than a 200% cumulative improvement.
The cost-benefit for a typical client with a meaningful Google Ads budget: A/B testing run by us generates an average 25-60% rise in conversion rate a year, which at the same ad spend means substantial savings every month from better conversion alone. We do not publish price ranges for the testing service: after a short conversation you get a fixed quote within 24 hours, with no hidden extras.
For a growing business, landing page optimisation through systematic A/B testing is one of the highest-ROI marketing investments. It isn't about a single "magic test" - it's about the compound effect of 5-15 tests a year, run correctly statistically, documented and repeatable.
If you run a landing page in Wrocław (or anywhere in Poland) and wonder whether A/B testing makes sense - the answer is yes, but only if you do it properly. 5% of companies do testing correctly and have compound improvements. 95% play optimisation theatre and waste time. The decision about which category you belong to is yours - and it starts with the first test planned with a hypothesis, a sample size and the discipline not to peek.