iPQB Reproducibility and Rigor: Experimental Controls Module

Introduction

• When you think of bias in an experiment, what comes to mind?
• What is an example of an unavoidable bias? What could be an example of an avoidable bias? An intentionally introduced bias?
• With regard to preparing samples for an experiment, what might constitute bias on the part of the experimenter? How might this type of bias manifest in the design of an experiment?
• How do you think bias might affect what is documented in a protocol, or detailed in a materials and methods section?
• Do you think there is a component of laboratory and research culture that must be considered when addressing reproducibility and rigor in research?
• Would people be willing to admit that their lab might not always conduct the most rigorously designed studies?

Overall questions:

• What is an example of a recent “control” experiment or analysis that you have performed? What does it control for and why do you think it is effective?

Biological and Technical Replicates

Starting Points:

• Replication: requires a precise process where the exact same findings are reexamined in the same way with identical design, power, subject selection requirements, and level of significance as the original research study.1
• Biological replicates are parallel measurements of biologically distinct samples that capture random biological variation, which may itself be a subject of study or a source of noise.
• Technical replicates are repeated measurements of the same sample that represent independent measures of the random noise associated with protocols or equipment.

Lead-in Questions:

• Within an individual experiment, what do you think is the best approach to determine the appropriate number of replicates?
• How did you learn about the need for replicates and the difference between certain types of replicates?

Follow-up Questions:

• Do you think it is common to report data from a single experiment (technical replicates) to generate an “exciting” finding? How often is this type of practice viewed as a way to expedite the research process?
• Since this is a grant application with preliminary results, is it acceptable to include results in such a manner?
• Is it appropriate for the applicant to purposely leave information about the type of replicates out and plot the data in such a way as to suggest significance over multiple experiments? Can it be considered falsification and therefore possible misconduct? If so, what are the potential consequences? What if it was simply an oversight?
• If this was your grant application, how would you have portrayed the data? Would you clearly state the “n” in the figure legend and/or describe this in the body of the grant? Would you have indicated the exclusion of data?
• Do you think papers or grant applications should delineate the use of biological vs. technical replicates in the figure legends (or elsewhere in the document)?
• The reviewer provides an analogy of “taking a thousand cells from one animal” and getting “just one point” from the resulting data. Is this always the case?3
• Do you think the review of the project will be affected?
• Do you think a typical review session discussing this issue would be as collegial?
• The reviewers appeared to be convinced easily that the figure was misleading. Do you think this transition in thought would have been so quick and painless if it were a real review session?

Sample Size Outliers Exclusion Criteria

Starting Points:

• Sample: here, a sample is defined as a single value or observation from the larger set of values
• Sample size: the optimal number of samples that should be used to reach sufficient statistical power; also referred to as ‘n’
• Outliers: an observation that lies an abnormal distance, typically +/- 3 standard deviations, from other values in a random sample from a group of results1
• Exclusion criteria: standards set out before a study or review to determine whether a sample should be included or excluded from the study or analysis2
• Characterization of “normal” for a specific experiment is an important component to identifying outliers and determining exclusion criteria

Lead-in Questions:

• Do you have a standard approach to determining the appropriate sample size and setting criteria for outliers – how you determine the numbers that go into your power analysis?
• How do you know what “normal” is if you don’t know the result? Can you do this initially? Will determination of the best statistical method and approach be useful in defining normal? Follow-up Questions: Lab Management
• Can you relate to this situation – not being able to generate similar results, whether from unpublished data in your own lab or a published paper?
• Have you ever tried to replicate someone’s experimental approach and discovered that information was missing in their lab notebook? Did you feel as though you needed a “Rosetta Stone” to decrypt their handwriting/abbreviations?
• Do you maintain a thorough laboratory record? If so, what methods do you follow to ensure that your lab notebook is comprehensive?
• Do you think an electronic lab notebook would have helped identify the issue(s) faster? What characteristics would the electronic lab notebook need to have?

Statistical Methods and Issues

• Have you ever had data that was “close” to significance? If so, what did you do? How did you interpret these results? Would Dr. Fielding (Harry) have suggested adding a few more samples and trying a different statistical test if they had initially defined their sample size and exclusion criteria, and identified the most appropriate statistical approach?
• Jamal told Robin to drop outliers above a certain value, as it is outside the physiologic range. Do you think this should have been considered further when they established their exclusion criteria? Do you think they actually developed exclusion criteria, or just considered that point as valid (potentially, without confirming) and made it their sole criteria for determining outliers?

Other Issues

• Do you think Dr. Fielding was too hard on Robin? Was there a more appropriate and effective approach that he could have taken when Robin was struggling to replicate Donna’s results?
• Is it realistic to think that most PIs would admit they provided inadequate guidance?
• Do you also think most PIs would take the time to review the lab notebooks themselves to determine what may be causing the discrepancy in the results?

Sex as a Biological Variable

• One of the fundamental variables in preclinical biomedical research is sex: whether a cell, tissue, or animal is female or male. 3 Do you generally consider sex as a variable when designing experiments?
• Have you or someone you know only used male mice in an experiment as a way of avoiding the “sex issue?” Do you think this is appropriate? Does it depend on the type of experiment being done?
• Can an experiment be considered rigorous if sex is not considered?
• A commonly used example advocating for the consideration of sex as a biological variable in research is the zolpidem (Ambien) dosage that was amended in 2013. The drug was found to affect men and women differently, which resulted in a decrease in the recommended dosage for women4 . Would this have occurred if sex was considered in the preclinical and clinical experiments?

CONTROLS

Experimental vs. Technical Replicates / Correlation and Causation

Jim Bull’s Text UT Austin