Plan outcomes, sample size, and analysis before collecting data
Choose an outcome and a defensible amount of information before the result can influence the plan.
Field diagramA repeatable study cycle
Estimated reading
11 min
Estimated practice
25 min
Equipment
Paper, calculator, or spreadsheet; supplied pilot data.
Progress stays in this page session. Allow lesson storage in Cookie settings to remember it; browser settings may also prevent saving.
Objective
By the end, you can:
Predefine primary outcomes and decision-relevant comparisons.
Give a transparent rationale for the number of independent units.
Distinguish exploratory analysis from a pre-specified test.
Plan the question you will answer
An outcome is the quantity or classification used to answer the question. Choose a primary outcome before data collection when possible, state its unit and calculation, and identify secondary or exploratory outcomes. A plan prevents a search through many outcomes until one looks favorable. It also makes missing data, exclusions, and transformations visible.
Sample size is a design argument, not a ritual number. Explain the smallest meaningful difference, expected variation, measurement resolution, available units, and decision stakes. A pilot can inform a range of variation but may be too small or biased to establish a final effect. If formal power calculations are appropriate, use a qualified method and state its assumptions; if they are not, provide a resource-based rationale and narrow the claim.
Primary outcome: one result chosen for the main question.
Effect of interest: the difference or relationship that would matter for the decision.
Variation: expected spread from independent units and measurement process.
Analysis rule: summary, comparison, and handling of missing or invalid records.
Stopping rule: when data collection ends, including a justified exception path.
Keep exploration labeled
Exploratory plots and subgroup checks can generate better questions. They should be labeled as exploratory when chosen after seeing outcomes. A post hoc pattern is not automatically false, but it needs confirmation in a future, pre-specified study. Do not increase the sample until a desired result appears while describing the process as fixed; if data collection changes, document the decision and its reason.
Write the analysis plan in dated form before unblinding or outcome review.
Use supplied data to rehearse calculations and inspect assumptions without calling the rehearsal evidence.
Report all planned primary outcomes and explain deviations or additional analyses.
Worked example
A bounded sample rationale
A learner can perform four independent fills per vessel and wants to estimate repeatability, not certify accuracy.
01Primary outcome: standard deviation of fill-level volumes in mL, calculated separately by vessel.
02Rationale: four independent fills are feasible and sufficient for a practice estimate, but uncertainty in that estimate is large.
03Decision: use the exercise to compare observed spread, not to establish a universal superiority claim.
04Future plan: repeat on another day with a pre-specified larger number of fills if the difference matters.
Materials
Set out what you need.
One-page analysis-plan template
Calculator or spreadsheet
Representative pilot dataset labeled as supplied
Safety & stop conditions
Pause if the work no longer fits the plan.
This is a planning lesson. It does not authorize studies involving people, animals, hazardous materials, or regulated decisions.
Method
Work through the steps.
01
Choose the outcome
Write the primary outcome, unit, calculation, and what difference would matter before reviewing final data.
02
Rationalize n
List independent units available, expected variation, meaningful difference, and the basis for the chosen number or range.
03
Freeze and date the plan
Save the plan with version and date; record any later changes as deviations with reasons.
Checkpoints
Observe, record, investigate.
The outcome cannot be redefined by changing units after seeing data.
The sample-size rationale identifies assumptions and limits.
Exploratory results are clearly labeled and point to future confirmation.
Common mistakes
Correct the process, preserve the record.
Saying “three replicates are enough” without context.
Explain the effect, variation, resolution, and decision purpose; no universal count fits every question.
Stopping when p or a graph looks favorable.
Use the pre-specified stopping rule or document the adaptive change and treat the result accordingly.
Treating a pilot as confirmatory evidence.
Label the pilot's purpose and limitations; use it to refine methods or assumptions unless its design supports the claim.
Practice
Write a one-page plan
Use supplied data from two water-volume methods to plan a follow-up. The data show a small spread but do not establish an accuracy reference.
Select one primary outcome and meaningful difference.
Give a resource-based rationale for independent n.
Write an exploratory analysis you would label clearly if added after seeing the data.
Answer: It limits outcome switching and gives the main question a clear analysis. Secondary and exploratory outcomes can still be useful when labeled.
02What should a sample-size rationale contain when no formal calculation is used?
Answer: The independent unit, meaningful difference, expected variation or pilot basis, measurement resolution, resources, and how those limits affect interpretation.
03Can a post hoc subgroup be reported?
Answer: Yes, as exploratory with the selection process and limitations disclosed. It should be treated as a hypothesis for a future pre-specified test, not quietly promoted to the primary result.
