A store experiment should answer a specific question. For example: does clearer delivery information reduce avoidable customer questions without reducing completed purchases? Select the problem with the e-commerce priority guide, then write down the proposed change and the decision the result will inform.
Define the comparison
Record the affected pages, eligible customers, devices, markets and products. Choose a primary outcome that matches the change. For a checkout improvement, that might be completed orders among customers who began checkout; for a sizing change, it might be relevant return reasons among delivered orders.
Confirm how the numerator and denominator are collected. Exclude clearly identified test activity consistently. A different report date, consent setting or event definition can create an apparent improvement without a real change.
Prepare the baseline and test conditions
Record the current outcome, order volume and known variation. Note discounts, stock outages, campaign changes and unusual delivery disruptions. Prefer a suitable randomised comparison where the platform supports it. If using before-and-after periods, document the differences and avoid claiming that the change alone caused the result.
Set a review plan appropriate to the traffic and buying cycle. There is no universal number of days that makes a low-volume test reliable. Include time for delayed outcomes such as returns.
Test the complete order path
- Verify product selection, quantity and displayed total.
- Check intended payment success and failure paths using the provider’s approved test process.
- Confirm the order reaches inventory and fulfilment correctly.
- Check confirmation messages and customer assistance.
- Prepare a rollback owner and a stop condition for serious errors.
Do not send real customer data into an unapproved test service or create accidental live charges while checking the flow.
Review profit and customer outcomes together
Compare completed orders with payment failures, support contacts, dispatch delays and returns. Calculate whether additional revenue remains useful after product, delivery, payment and support costs. A feature that increases sales while creating expensive corrections may need another iteration.
Broad market figures from the Census e-commerce series can provide context, but they do not validate a shop experiment. Keep the store’s comparison and its limitations explicit.
Write the decision record
Summarise what changed, the eligible audience, observations, unresolved questions and the decision: retain, revise, stop or gather more evidence. Keep the previous configuration available until the chosen outcome is verified in normal operation.
