Sample Size Calculation for Independent Samples t-tests

Yesterday we went over the basics of sample size calculation. What about individual tests? Since we know that sample size calculation will vary by individual test, it is important to calculate the minimum sample size necessary considering the type of statistical analysis you are doing. I am going to cover sample size calculation for independent samples t-tests today.

Power Analysis for Independent Samples t-tests

For this discussion I will refer to Jacob Cohen’s short journal article on power analysis titled, Quantitative Methods in Psychology: A Power Primer. This helpful, little journal article nicely identifies effects sizes for common statistical tests, as well as the necessary sample sizes to find those tests significant, given a particular level of significance (0.01, 0.05, and 0.10), effect size (small, medium, and large), and power of 0.80.

Yesterday we covered the facets of sample size calculation and actually used the example of an independent samples t-test. We need a couple things to calculate the sample size necessary for our t-test:

· Level of significance: Our probability of committing a Type I error, or the probability of falsely rejecting the null hypothesis. Usually this is 0.05, making the probability of committing a Type I error 5%.We decide this.

· Estimated effect size: For simplicity purposes, we are going to think of this as the difference between the two groups. This is an estimate and will be determined by our software for the samples we are using. We have no control over this.

· Power: Our probability of committing a Type II error, or the probability of falsely accepting the null hypothesis. Usually this is 0.80, making the probability of committing a Type II error 20% or four times as likely as the probability of committing a Type I error. We also decide this.

A priori Sample Size for Independent Samples t-tests

Of all the sample size calculations, this is probably the easiest. Page 157 of Quantitative Methods in Psychology: A Power Primer tabulates effects sizes for common statistical tests. Number 1 is t-test for the difference between two independent means or the independent samples t­-test. It tells us that a small effect size is 0.20, a medium effect size is 0.50, and a large effect size is 0.80.

While ideally you should have an effect size from empirical research, we are going to look for a 0.05 level of significance, estimate a medium effect size of 0.50, and look for a power of 0.80. Given these criteria, we turn the page to 158 and look for the same number one. We find that for our criteria, we need 64 participants in each of the groups for a total of 128 participants.

For a customized sample size calculation for your study, thesis, or dissertation, please call Statistics Solutions Inc. (877)437-8622, for a free 30 minute consultation.

Sample Size Calculation

One of the most frequent requests we get as statistical consultants is sample size justification or sample size calculation. Our clients typically have a study they wish to conduct or are working on their dissertation and have completed their proposal or first couple of chapters but the process seems to grind to a crawl when faced with sample size calculation and the appropriate statistical test to test their carefully thought-out research design. Deadlines are looming and you need information fast. Sample size calculation need not be daunting but requires some basic understanding:

1. Sample size is a function of level of significance, effect size, and power.

This means your sample size is going to be dependent on the level of significance, effect size, and power. This also means if you change one of the three measures, your sample size will also change. This is not a complicated concept, but for more information see blog entries on the subject. This is important to understand in sample size calculation.

2. Sample size calculation is dependent on the statistical tests you are conducting.

Since effect size will vary with the statistical tests you are conducting, your sample size calculation will vary depending on the statistical tests you are planning to conduct. Measurements of effect size are different for each statistical test. For example, in sample size calculation, small, medium, and large effect sizes for t-test are 0.20, 0.50, and 0.80, respectively. For a one-way ANOVA the same measures of effect size are 0.10, 0.25, and 0.40, respectively. Since effect size is part of the sample size calculation, your sample size calculation will vary with statistical test.

3. Sample Size Calculation will be unique to your study.

We have established that sample size calculation utilizes the relationship between sample size, level of significance, effect size, and power. We also know that our basic effect size measurements are going to vary with the statistical tests.

However, it may be the case that you are replicating a study, using an instrument that someone has already used in a study, or have information regarding the effect sizes found by other researchers doing something similar. If in your literature review, you find the results of statistical tests conducted by other researchers on something similar to what you are doing, you might find the actual, reported effect size of their study.

For example you might find that a researcher conducting similar research found a large effect size. This would make your a priori sample size requirements considerably smaller, ceteris paribus, than if he had found say a small effect size. For a t-test at the 0.05 level of significance, a power of 0.80, a small effect size, your minimum sample to find your statistical test significant is 393 in each group for a total of 786 participants. With a large effect size your minimum sample would be 26 in each group for a total of 52 participants.

For a customized sample size calculation for your study, thesis, or dissertation, please call Statistics Solutions Inc. for a free 30 minute consultation. 877-437-8622