More about statistical power, sample size, and how to determine it
The method used to determine your sample size depends on whether you’re doing quantitative or qualitative research. With qualitative research, you want your sample to be big enough to draw valid conclusions about the population in question. Otherwise, the data you collect may not provide much evidential value (unless it’s just a pilot).
In quantitative research, your approach depends on whether your study is observational or experimental. If observational, then you need to decide on an acceptable margin of error for your primary variable of interest, in addition to the level of confidence you want in your finding. For a quick sample size determination, you can use a table like this one.
If experimental research, on the other hand, it is important to understand the concept of statistical power in order to calculate the required sample size. In brief, higher statistical power means that you’re more likely to detect an effect, if that effect actually exists. In more formal terms, power is the probability that you correctly reject the false null hypothesis when a specific alternative hypothesis is true. Consequently, an experiment with more statistical power has a better chance of detecting a true effect.
The question is, how much power do you need to detect the effect you are investigating? And consequently, how many participants do you need to recruit for your study? First, you need to turn to the existing literature or run pilot experiments to decide which effect size you expect. You should then do a power calculation to determine the sample size required to detect that effect, given the p-value threshold you intend to use.
Whilst a sample can certainly be too small (or 'underpowered'), can your sample size also be too large (that is, 'oversampled')? The short answer is: no. The simple reasoning behind this is that increasing your number of participants also increases the likelihood to find the true effect size. After all, the reason you investigate only a certain amount of people is because you cannot investigate the whole population. But you still want to be confident about the conclusions you draw from your sample to the larger population. So the closer your sample size gets to the population size, the more confident you can be in the effect size you find.
Read more about the question of oversampling in our blog!