P-value or False Discovery Rate When Two-Sample t-Test is Employed Instead of Paired t-Test

Abstract

When performing several tests on the same dataset (i.e., multiple comparison testing), the False Discovery Rate (FDR) is commonly used instead of p-values. If all the assumptions of the two-sample independent t-test are met, the p-value is commonly used to test if there is a difference between the population means. What if the two samples are correlated, and a two-sample independent test is carried out instead of the paired t-test? Should we consider p-values or FDRs to make decisions regarding hypotheses? This research explores two methodologies (p-values and FDR) for different magnitudes of correlation between two samples and sample sizes when one hypothesis test (not multiple) is performed. The two strategies are tested using a 100,000-run simulation with correlation coefficients ranging from -0.9 to 0.9 for various sample sizes. The simulation results reveal that if there is no correlation, both approaches are equally valid, which is predicted. However, if a correlation exists, the FDR is recommended to avoid making less erroneous decisions.