How Do You Prove a Design Change Worked?

The redesign shipped on a Tuesday. Conversions were up the following week, everyone said so in the channel, and the team moved on. Nobody mentioned that the previous week contained a public holiday, or that a paid campaign started on the Thursday, or that the month before that was never written down anywhere. The change may well have worked. There is simply no way to tell.

Measuring design changes is not a statistics problem for most teams. It is a sequencing problem. Nearly everything that makes a result trustworthy has to happen before the design changes, and nearly everything teams actually do happens after it.

Three things need to exist before the first mockup: a baseline, one named metric, and two or three guardrails you would accept as a veto. With those in place, even an inconclusive result teaches you something. Without them, a genuine win and a coincidence look identical, which is how teams end up wondering why the redesign converts worse than the page it replaced with no way left to find out.

The baseline you did not take

A baseline is a written record of how the current thing behaves, captured before anybody touches it. Not a dashboard you could go and look at later. A written record, with dates, filters and definitions attached, saved somewhere that is not one person’s memory.

It has to be written because definitions drift. Six weeks on, nobody agrees whether conversion meant a form submission or a qualified enquiry, whether bot traffic was filtered, whether the mobile figure included tablets. The comparison then becomes an argument about the numbers instead of an argument about the design, and that argument is settled by whoever is most senior in the room.

A baseline worth having records five things.

• The period, exactly. Which dates, and why those. A baseline drawn from a fortnight that contained a campaign is not a baseline, it is a campaign.
• The definition of the metric. In one sentence anybody could apply. If two people could count it differently, write it more tightly until they cannot.
• The segments you will compare. Device and traffic source at minimum. A change that helps desktop and hurts mobile shows up as nothing at all in a blended average.
• The normal variation. How much the number moves week to week when nothing has changed. This is the most useful item on the list and the one almost nobody records.
• What else is running. Campaigns, pricing changes, seasonal patterns, a competitor’s launch. Write them down even when they look irrelevant, because they will be quoted at you later.

The fourth one deserves a sentence of its own. If the number routinely swings from week to week with nothing changed, then a change that moves it by less than that swing has told you nothing, however confident the title on the slide is.

Name the metric before you touch the design

This is the rule that does the most work and gets broken most often. Decide, in writing, which number the change is supposed to move and in which direction, before the design exists.

The reason is not rigour for its own sake. It is that after any release, some dashboard somewhere contains something that went up. Choose the metric afterwards and it will be chosen from the set that improved, at which point the analysis has become a search for a compliment. Put plainly: a metric picked after the release is praise, not evidence.

One metric, not five

Pick one. A change with five success metrics has none, because any outcome can be narrated as a partial win. If the team genuinely cannot agree which single number matters, that disagreement is more important than the design work and should be settled first, usually by asking what decision the number is going to inform.

The metric should sit as close to the change as possible. A checkout button redesign is measured on checkout completion, not on revenue, because revenue is downstream of a dozen things you did not touch. Choosing the nearest honest number is most of what strategy and analytics work looks like at this scale, and it is mostly an exercise in restraint.

What a usable metric looks like

Three properties. It moves inside the observation window, so a number that only shifts over two quarters cannot judge something you shipped last week. It is attributable, meaning your change is a plausible cause rather than a coincidence in the same period. And it is countable without a new tracking project, because a measurement plan that needs three weeks of engineering gets quietly dropped in week one.

Form work is the clean example. Removing a field from an enquiry form has an obvious nearest metric, completion rate by device, and an obvious downstream risk, the quality of what arrives. Both are countable without heroics. That combination is why how many fields a contact form should have is one of the few design questions that can be settled with evidence rather than taste.

Guardrails: the numbers allowed to veto a win

A guardrail metric is one you do not expect to improve, and which you have agreed in advance would cancel the result if it got worse. Without guardrails, optimisation quietly becomes extraction: you move the primary number by damaging something nobody was watching.

Two of those columns carry accessibility and performance measures, and that is deliberate. Interaction to Next Paint and Largest Contentful Paint belong on almost every visual change, because heavier pages are a routine side effect of better-looking ones. So does keyboard operability, which redesigns break more often than anyone expects and which no conversion metric will ever mention.

Building those checks in from the start rather than meeting them at audit time is the argument in accessibility as a default, and it applies here directly. An accessibility regression is a result. A change that trades one away for a point of conversion has not worked; it has moved a cost somewhere nobody is measuring.

What counts as evidence when traffic is low

Most teams reading this do not have the volume for a clean split test on a marketing page, and pretending otherwise produces the worst available outcome: a test that runs for a fortnight, reaches no reliable conclusion, and gets reported as a win regardless.

When a split test is honest

Run one when the page produces enough of the event you are measuring for the difference you care about to clear the normal noise, when the traffic mix stays stable across the period, and when you can leave it alone until it finishes. That last condition fails far more often than the other two. A test stopped early because it looked good is not a test. It is a screenshot.

What to use instead

Sequential comparison, with the calendar and the campaign schedule written into the caveats. Session recordings and a handful of moderated task attempts, which find usability failures no aggregate number can see. Support tickets and sales objections from before and after, which are the least fashionable data in the building and among the most reliable. And direct instrumentation: a validation error firing on one field and nowhere else tells you exactly what to fix, with no statistics involved at all.

Watching five people attempt the task will find more of what is wrong with a checkout than a month of aggregate numbers. It will not tell you whether the new version is better commercially. Use both and stay clear about which question each one answers, because UX and UI product design decisions usually need the first kind of evidence and budget decisions need the second.

Reading the result without flattering yourself

Assume the result is contaminated, then go looking for how. Four contaminants account for most of it.

Novelty first. Existing users react to any change for a while, in both directions, and the effect decays. If the entire gain appears in the first few days and then fades, what you measured was attention, not improvement.

Then the calendar. Holidays, month ends, invoice cycles, school terms, the last week of a quarter. Compare like periods rather than adjacent ones, even when the adjacent ones are easier to pull.

Then traffic mix. A shift in campaign spend changes who is arriving, and different people convert differently. Segment by source before believing anything, because a mix change can manufacture an improvement out of a page that did not change at all.

Then simultaneous releases. If three things shipped that week, you measured the week. This is the one nobody wants to hear, and it is why we argue for one meaningful change per release even when that slows delivery down. Shipping four changes together and learning nothing from any of them is not faster.

A result you cannot explain a mechanism for is not a result yet. It is a coincidence with good timing.

How to report a change that did nothing

Most design changes move nothing measurable. That is the ordinary outcome rather than a failure of the designer, and a team that cannot say it plainly will keep shipping work it never learns from. Report it like this.

1. Quote the expectation in its original words. Paste in the hypothesis you wrote before the work started. It is the proof that the metric was not selected afterwards to suit the outcome.
2. Give the number, the period and the normal variation. The third of those turns “no change” from an admission into a defensible statement about what was detectable at all.
3. Say what you would have needed to see the effect. “At this volume we could only have detected a movement larger than the usual weekly swing” is an honest sentence and a useful one.
4. Report the guardrails anyway. A change that left the primary metric flat while improving load time or keyboard access is still worth keeping, and the report should say so plainly.
5. Recommend one of three actions: keep, revert, or re-run differently. A report without a recommendation reads as a defence, and defences invite arguments rather than decisions.

Format matters more here than people expect, because a finding nobody can act on is a finding that gets ignored. Every result wants a severity, a priority and a recommendation attached in language a non-specialist can use. That is the shape of a good audit report in general, and it is what a tool such as Prooflin produces: AI-assisted findings, severities, priorities and recommendations resolved into a reviewable report rather than a list of observations somebody else has to interpret.

There is a related honesty problem in reporting periods generally. Ninety days of work does not produce ninety days of proof, and being straight about what a quarter can and cannot demonstrate is the subject of what ninety days can honestly show. The same discipline applies to a single release: state what the window supports, then stop.

The measurement you keep after the release

Instrumentation built for one change is worth more than the change. Most teams dismantle it, or leave it running unattended until nobody trusts what it says, and then build something similar again for the next question.

Keep three things. The metric definitions, so the next comparison counts the same way. The baseline record, so you accumulate a history of what normal looks like on your own site. And the guardrail set, which barely changes between projects and turns naturally into a release checklist. Teams that do this find the third item slowly becoming the measurement half of their design system, which is one of the less obvious answers to when you actually need a design system.

The compounding here is real and slow. The tenth change is far easier to judge than the first, because by then you know how your own numbers behave when nothing is happening. That accumulated knowledge of your own noise is the actual asset in conversion optimisation, considerably more than any individual test result.

Where we would start

Before the next design change, spend one hour. Write the current number down with its definition. Note what else is running. Write the single metric the change should move. List two guardrails and who may call them. That is the whole method, and everything above it is elaboration.

The honest concession: this discipline does not suit every change. Some things should be done because they are correct, not because a number moved. Fixing a failing contrast ratio, removing a misleading claim, repairing keyboard access, correcting a form that loses people’s answers. Measure them if you like, but do them regardless of the result. Measurement is for changes where reasonable people disagree, and a good deal of design work is simply not that.

And when a stakeholder asks whether the redesign worked and the honest answer is that you cannot tell, say it once, then fix the measurement rather than the story. The credibility that buys outlasts the win you would otherwise have claimed. If you want a second read on a measurement plan before a release goes out, send us the hypothesis and the metric you intend to judge it on.