Two Pilots, Two Trajectories: Why One Succeeded and the Other Closed

Revised English edition, 11 October 2026. The original contrasted two pilot accounts. This edition presents the decision framework without treating historical figures as verified results or guarantees.

Two stores can use similar technology, catalog sizes and teams yet reach different decisions after a pilot. Faster pages alone do not resolve an unclear business question. A useful pilot must test both the implementation and the organization's ability to decide what happens next.

A pilot with a clear decision

Before the first implementation task, choose a buyer problem and a measurement window. “Make the site better” is too broad. A category journey, a set of difficult search queries or a catalog content workflow gives the team a boundary it can inspect.

Record the baseline and how the result will be calculated. For a conversion measure, state the numerator, denominator and population. For search, retain the test queries and expected products. For catalog content, retain the source attributes, approved fields and before-and-after versions.

Name one decision owner. Other teams should contribute requirements, but someone needs authority to approve scope, resolve questions and choose whether to continue. Put that name and their responsibilities in the pilot brief.

Write the scope before adding features

A short document can distinguish three categories: included work, excluded work and changes that need a separate discussion. Moving a category listing does not automatically include product pages, the basket, checkout or a redesign.

Search is also a separate workstream when it brings indexing, synonyms, ranking rules and integration questions. Adding it halfway through another pilot changes both the delivery work and the interpretation of the result.

If marketing requests new banners or a different layout, assess whether that change belongs in the experiment. A decision to defer it is easier to explain when the original scope and question are written down.

A pilot with an unclear decision

Consider the opposite setup: success means “we will see how it feels,” ownership means a committee and the scope is an oral agreement. An engineering problem then waits for the next meeting, while each stakeholder expects a different outcome.

The developer may demonstrate lower response time. Marketing may expect more purchases. Product may focus on small design complaints. Each observation can be valid, but none answers an agreed question if the team never defined one.

This creates avoidable delay. New work is added without a revised budget, acceptance moves during implementation and the final review becomes a debate about expectations. A technical improvement can still lead to a decision to stop.

Technology and decision readiness are different

A pilot needs to answer two questions: does the implementation work on the relevant data, and can the organization make the next decision using the evidence? Passing one does not imply passing the other.

Do not interpret a positive laboratory score as proof of more revenue. Likewise, an early conversion fluctuation does not establish a sustained effect. Use technical acceptance first, then the agreed business evaluation with a sample and period that can support the conclusion.

The five fields in a useful pilot brief

Field	What to write	What to avoid
Success	One primary question, baseline, calculation and measurement window	“Better” or an undefined percentage
Owner	A named person with decision authority	“The team will decide” without responsibilities
Scope	Included routes, data and interactions; explicit exclusions	“Add anything that is easy”
Rollback	A trigger, decision maker and tested recovery procedure	“We will handle it if something happens”
Next step	The decision after acceptance, with a separate scope and budget if needed	An automatic commitment to a larger rollout

The thresholds belong to your store. They must account for traffic, seasonality, promotions, device mix and the impact of a failure. A copied threshold from another project is not a substitute for that review.

Make the evidence reviewable

Retain the pilot brief, test data and release version. Compare like-for-like windows and explain missing data. If a catalog problem is corrected during the pilot, record the change: otherwise it may be mistaken for an effect of the new interface.

A result can be “technically accepted, business impact not yet established.” That is a useful conclusion. It tells the owner which decision is supported and which needs more observation.

Keep decisions moving without expanding the experiment

Use a short review rhythm with clear escalation. Engineering questions should go to the person who can resolve them; scope changes should be recorded with their cost and effect on the test. An unresolved assumption should remain visible rather than turning into an implicit promise.

When the checks pass, discuss the next stage as a new decision. When they fail, inspect the cause and decide whether another bounded iteration is worthwhile. Neither response requires treating the pilot as a commitment to replace the whole store.

Prepare your own pilot

Bring the store URL, current platform, examples of the buyer problem, available analytics and the people responsible for the catalog and integrations. These inputs are enough to start a practical discussion of scope, evidence and price.

For a catalog-search problem, review Site Search and its plans and limits. For a storefront change over an existing backend, explore Frontbox. The appropriate pilot depends on the problem you want to decide.