Statistical Analysis Skill

Descriptive statistics, trend analysis, outlier detection, hypothesis testing, and guidance on when to be cautious about statistical claims.

Descriptive Statistics Methodology

Central Tendency

Choose the right measure of center based on the data:

Situation Use Why
Symmetric distribution, no outliers Mean Most efficient estimator
Skewed distribution Median Robust to outliers
Categorical or ordinal data Mode Only option for non-numeric
Highly skewed with outliers (e.g., revenue per user) Median + mean Report both; the gap shows skew

Always report mean and median together for business metrics. If they diverge significantly, the data is skewed and the mean alone is misleading.

Spread and Variability

Percentiles for Business Context

Report key percentiles to tell a richer story than mean alone:

p1:   Bottom 1% (floor / minimum typical value)
p5:   Low end of normal range
p25:  First quartile
p50:  Median (typical user)
p75:  Third quartile
p90:  Top 10% / power users
p95:  High end of normal range
p99:  Top 1% / extreme users

Example narrative: "The median session duration is 4.2 minutes, but the top 10% of users spend over 22 minutes per session, pulling the mean up to 7.8 minutes."

Describing Distributions

Characterize every numeric distribution you analyze:

Trend Analysis and Forecasting

Moving averages to smooth noise:

# 7-day moving average (good for daily data with weekly seasonality)
df['ma_7d'] = df['metric'].rolling(window=7, min_periods=1).mean()

# 28-day moving average (smooths weekly AND monthly patterns)
df['ma_28d'] = df['metric'].rolling(window=28, min_periods=1).mean()

Period-over-period comparison:

Growth rates:

Simple growth: (current - previous) / previous
CAGR: (ending / beginning) ^ (1 / years) - 1
Log growth: ln(current / previous)  -- better for volatile series

Seasonality Detection

Check for periodic patterns:

  1. Plot the raw time series -- visual inspection first
  2. Compute day-of-week averages: is there a clear weekly pattern?
  3. Compute month-of-year averages: is there an annual cycle?
  4. When comparing periods, always use YoY or same-period comparisons to avoid conflating trend with seasonality

Forecasting (Simple Methods)

For business analysts (not data scientists), use straightforward methods:

Always communicate uncertainty. Provide a range, not a point estimate:

When to escalate to a data scientist: Non-linear trends, multiple seasonalities, external factors (marketing spend, holidays), or when forecast accuracy matters for resource allocation.

Outlier and Anomaly Detection

Statistical Methods

Z-score method (for normally distributed data):

z_scores = (df['value'] - df['value'].mean()) / df['value'].std()
outliers = df[abs(z_scores) > 3]  # More than 3 standard deviations

IQR method (robust to non-normal distributions):

Q1 = df['value'].quantile(0.25)
Q3 = df['value'].quantile(0.75)
IQR = Q3 - Q1
lower_bound = Q1 - 1.5 * IQR
upper_bound = Q3 + 1.5 * IQR
outliers = df[(df['value'] < lower_bound) | (df['value'] > upper_bound)]

Percentile method (simplest):

outliers = df[(df['value'] < df['value'].quantile(0.01)) |
              (df['value'] > df['value'].quantile(0.99))]

Handling Outliers

Do NOT automatically remove outliers. Instead:

  1. Investigate: Is this a data error, a genuine extreme value, or a different population?
  2. Data errors: Fix or remove (e.g., negative ages, timestamps in year 1970)
  3. Genuine extremes: Keep them but consider using robust statistics (median instead of mean)
  4. Different population: Segment them out for separate analysis (e.g., enterprise vs. SMB customers)

Report what you did: "We excluded 47 records (0.3%) with transaction amounts >$50K, which represent bulk enterprise orders analyzed separately."

Time Series Anomaly Detection

For detecting unusual values in a time series:

  1. Compute expected value (moving average or same-period-last-year)
  2. Compute deviation from expected
  3. Flag deviations beyond a threshold (typically 2-3 standard deviations of the residuals)
  4. Distinguish between point anomalies (single unusual value) and change points (sustained shift)

Hypothesis Testing Basics

When to Use

Use hypothesis testing when you need to determine whether an observed difference is likely real or could be due to random chance. Common scenarios:

The Framework

  1. Null hypothesis (H0): There is no difference (the default assumption)
  2. Alternative hypothesis (H1): There is a difference
  3. Choose significance level (alpha): Typically 0.05 (5% chance of false positive)
  4. Compute test statistic and p-value
  5. Interpret: If p < alpha, reject H0 (evidence of a real difference)

Common Tests

Scenario Test When to Use
Compare two group means t-test (independent) Normal data, two groups
Compare two group proportions z-test for proportions Conversion rates, binary outcomes
Compare paired measurements Paired t-test Before/after on same entities
Compare 3+ group means ANOVA Multiple segments or variants
Non-normal data, two groups Mann-Whitney U test Skewed metrics, ordinal data
Association between categories Chi-squared test Two categorical variables

Practical Significance vs. Statistical Significance

Statistical significance means the difference is unlikely due to chance.

Practical significance means the difference is large enough to matter for business decisions.

A difference can be statistically significant but practically meaningless (common with large samples). Always report:

Sample Size Considerations

When to Be Cautious About Statistical Claims

Correlation Is Not Causation

When you find a correlation, explicitly consider:

What you can say: "Users who use feature X have 30% higher retention" What you cannot say without more evidence: "Feature X causes 30% higher retention"

Multiple Comparisons Problem

When you test many hypotheses, some will be "significant" by chance:

Simpson's Paradox

A trend in aggregated data can reverse when data is segmented:

Survivorship Bias

You can only analyze entities that "survived" to be in your dataset:

Ecological Fallacy

Aggregate trends may not apply to individuals:

Anchoring on Specific Numbers

Be wary of false precision: