Mattsays ranks wines with a composite, variance-weighted index.
Components of the ranking process include:
Variance from Group Data (Between-Group Variance)
- Measures how much the item’s values deviate from the overall group mean
- This is akin to an ANOVA-style (Analysis of Variance) effect size: items that consistently differ from the group average are ranked higher.
Number of Occasions It Appears (Frequency)
- Items observed more often gain statistical reliability.
- Weighting by frequency prevents rare but extreme values from dominating the ranking.
Its Own Data Values (Magnitude)
- The raw values themselves matter — e.g., higher scores, larger measurements, or stronger effects.
- This is normalized via z-scores to ensure comparability across scales.
Variance of Its Own Data Points (Within-Item Variance)
- Captures consistency: items with low internal variance are more stable and thus more reliable.
- High variance within an item reduces its ranking because it’s less predictable.
Size of the Occasions Its Data Is Recorded (Sample Size per Occasion)
- Larger sample sizes per occasion increase confidence in the measurement.
- This is similar to weighting by inverse standard error in meta-analysis.