Tests whether a population proportion \(\pi\) equals a hypothesized value \(\pi_0\).
The normal approximation requires both \(n\pi_0 \geq 10\) and \(n(1-\pi_0) \geq 10\).
Under H\(_0\), each simulated sample draws \(k^* \sim \text{Binomial}(n, \pi_0)\). We compute \(Z^*\) for each sample. The simulated p-value is the fraction of \(|Z^*|\) values as extreme or more extreme than the observed \(|Z|\).
p > 0.10: Weak evidence against H\(_0\)
0.05 < p < 0.10: Moderate evidence
0.01 < p < 0.05: Strong evidence
p < 0.01: Very strong evidence
Tests whether a population mean \(\mu\) equals a hypothesized value \(\mu_0\) when the population standard deviation is unknown.
The t-distribution has \(n - 1\) degrees of freedom. As df increases, the t-distribution converges to the standard normal. With small samples, the heavier tails of the t-distribution account for the extra uncertainty from estimating \(\sigma\) with \(s\).
Use Z when \(\sigma\) is known (rare). Use t when \(\sigma\) is unknown and estimated by \(s\). For large n, the two are nearly identical.
The t-test requires the population to be approximately normal, or \(n\) to be large enough for the CLT to apply. Tintle et al. use \(n \geq 20\) as the guideline. The traditional CLT threshold is \(n \geq 30\).
Tintle, N. et al. Introduction to Statistical Investigations (ISI).
Under H\(_0\), we generate \(n\) observations from N(\(\mu_0\), \(s\)) for each simulation. We compute the sample mean, sample SD, and \(t^*\) for each. The simulated p-value is the fraction of \(|t^*|\) values as extreme or more extreme than the observed \(|t|\).
Tests whether two population proportions \(\pi_1\) and \(\pi_2\) are equal.
Under H\(_0\): \(\pi_1 = \pi_2\). If they are equal, our best estimate of that common proportion uses all the data from both groups. The pooled proportion \(\hat{p} = \frac{k_1 + k_2}{n_1 + n_2}\) serves as the shared estimate for the standard error calculation.
All four of the following must be \(\geq 10\): \(n_1\hat{p}\), \(n_1(1-\hat{p})\), \(n_2\hat{p}\), \(n_2(1-\hat{p})\).
The two groups must be independent. If subjects are matched or paired, use a different test.
Tests whether two population means \(\mu_1\) and \(\mu_2\) are equal. Welch's version does not assume equal variances.
The Welch approximation for degrees of freedom accounts for unequal variances and unequal sample sizes.
The pooled t-test assumes \(\sigma_1 = \sigma_2\) and uses a single pooled variance estimate. Welch's test relaxes this assumption. Welch is the safer default and is what R's t.test() uses.
Each group needs approximate normality or a large enough sample. Tintle et al. use \(n \geq 20\) per group. The traditional CLT threshold is \(n \geq 30\) per group.
Tintle, N. et al. Introduction to Statistical Investigations (ISI).
Under H\(_0\) (both groups have the same mean), we simulate from N(0, \(s_1\)) and N(0, \(s_2\)). For each pair of simulated samples, we compute the Welch t-statistic.
Tests whether the mean difference \(\mu_d\) between paired observations equals zero. This reduces to a one-sample t-test on the differences.
Pairing removes between-subject variability. Instead of comparing two independent groups (each with their own variability), we analyze only the within-subject differences. This often reduces the standard error and increases power.
where \(\bar{d}\) is the mean of the differences and \(s_d\) is the standard deviation of the differences.
Use the paired test when observations come in natural pairs: before/after measurements on the same subject, matched subjects, or repeated measures. Use the two-sample test when the groups are independent.
The differences need approximate normality or a large enough sample. Tintle et al. use \(n \geq 20\) pairs. The traditional CLT threshold is \(n \geq 30\).
Tintle, N. et al. Introduction to Statistical Investigations (ISI).
Under H\(_0\) (\(\mu_d = 0\)), we generate \(n\) differences from N(0, \(s_d\)) for each simulation. We compute \(\bar{d}^*\), \(s_d^*\), and \(t^*\) for each. The simulated p-value is the fraction of \(|t^*|\) values as extreme or more extreme than the observed \(|t|\).