Intro page

Onboarding

General information

Research

Organization

Untitled page

Offboarding

Explore

Research

Power analysis

Mindhacks

⁠

http://mindhacks.com/2015/06/19/power-analysis-of-a-typical-psychology-experiment/⁠

⁠

Understanding statistical power is essential if you want to avoid wasting your time in psychology. The power of an experiment is its sensitivity – the likelihood that, if the effect tested for is real, your experiment will be able to detect it.

Statistical power is determined by the type of statistical test you are doing, the number of people you test and the effect size. The effect size is, in turn, determined by the reliability of the thing you are measuring, and how much it is pushed around by whatever you are manipulating.

Since it is a common test, I’ve been doing a power analysis for a two-sample (two-sided) t-test, for small, medium and large effects (as

conventionally defined⁠

). The results should worry you.

⁠

⁠

This graph shows you how many people you need in each group for your test to have 80% power (a standard desirable level of power – meaning that if your effect is real you’ve an 80% chance of detecting it).

Things to note:

even for a large (0.8) effect you need close to 30 people (total n = 60) to have 80% power

for a medium effect (0.5) this is more like 70 people (total n = 140)

the required sample size increases drammatically as power drops

for small effects, the sample required for 80% is around 400 in each group (total n = 800).

What this means is that if you don’t have a large effect, studies with between groups analysis and an n of less than 60 aren’t worth running. Even if you are studying a real phenomenon you aren’t using a statistical lens with enough sensitivity to be able to tell. You’ll get to the end and won’t know if the phenomenon you are looking for isn’t real or if you just got unlucky with who you tested.

Implications for anyone planning an experiment:

Is your effect very strong? If so, you may rely on a smaller sample (For illustrative purposes

the effect size of male-female heigh difference is ~1.7⁠

, so large enough to detect with small sample. But if your effect is this obvious, why do you need an experiment?)

You really should prefer within-sample analysis, whenever possible (power analysis of this left as an exercise)

You can get away with smaller samples if you make your measure more reliable, or if you make your manipulation more impactful. Both of these will increase your effect size, the first by narrowing the variance within each group, the second by increasing the distance between them

Technical note: I did this cribbing code from Rob Kabacoff’s

helpful page on power analysis⁠

. Code for the graph shown here is

here⁠

. I use and recommend

Rstudio⁠

Cross-posted from

www.tomstafford.staff.shef.ac.uk⁠

where I irregularly blog

things⁠

I think will be useful for undergraduate Psychology students.

What to do

The software G*Power (

https://en.wikipedia.org/wiki/G*Power⁠

) allows the calculation of sample size and power of an experimental design.

Obtain the app here (Win/Mac):

⁠

https://www.psychologie.hhu.de/arbeitsgruppen/allgemeine-psychologie-und-arbeitspsychologie/gpower⁠

⁠

Here are two pages to describe the parameters to use in one way r.m. anovas and t-tests:

⁠

https://www.scalestatistics.com/sample-size-for-repeated-measures-anova.html⁠

⁠

https://www.scalestatistics.com/sample-size-for-repeated-measures-t-test.html⁠

⁠

If you use G*Power for your research, then consider adding to your report or paper one or both of the following references (depending on what is appropriate).

Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39, 175-191.

Download PDF⁠

⁠

Faul, F., Erdfelder, E., Buchner, A., & Lang, A.-G. (2009). Statistical power analyses using G*Power 3.1: Tests for correlation and regression analyses. Behavior Research Methods, 41, 1149-1160.

Download PDF⁠

⁠

Example

Example of a power analysis with a 1-way 4-levels within-participant design. With 2-way design, consider each level independently.

⁠

Want to print your doc?
This is not the way.

Try clicking the ⋯ next to your doc name or using a keyboard shortcut (

CtrlP

) instead.