|
VSC 443/543 Fall 2008
Paula D. Johnson, DVM, MS |
 |
TABLE OF CONTENTSI. INTRODUCTION
II. INITIAL
DESIGN PROCESS
III. DESIGN
PLAN DEVELOPMENT
IV.
FINAL CONSIDERATIONS
|
I. INTRODUCTION The design of a research project affects the outcome of a research study in many ways, and is thus of great importance. All aspects need to be clearly thought out and planned in advance. After an idea for a research project is conceived, a thorough review of the literature and consultation with experts in that field are pursued to refine the problem, question, or hypothesis and to assimilate background information that is necessary for the experimental design phase. Perhaps the most critical step in designing animal research is the identification of the most appropriate animal model to address the experimental question being asked. Other practical considerations include defining the necessary control groups, determining the number of animals needed per group, evaluating the logistics of the actual performance of the animal experiments, establishing the most appropriate methods for statistical analyses, and identifying potential collaborators experienced in the area of study. All of these factors are critical to designing an experiment that will generate scientifically valid and reproducible data, which should be considered the ultimate goal of any scientific investigation. Granting Agencies and the IACUC (Institutional Animal Care and Use Committee) are critical in successful research projects and in addition to serving as “regulatory” agencies, they may also act as consultants for advice. Peer reviewed journals which publish research work are very particular in the type of research and the accuracy in which it is accomplished. It is important to conduct standard laboratory practices, reputable, sound, and repeatable research, in order to be published in a “highly thought of” and “peer reviewed” journal – the goal of most research – to get the word out and benefit biomedical science and all that that implies! SUGGESTIONS FOR OPTIMIZATION OF EXPERIMENTAL DESIGN The steps listed below comprise a practical sequence for designing and conducting scientific studies. We recommend that investigators: 1. Conduct a complete literature review and consult experts who have experience with the techniques proposed in an effort to become thoroughly familiar with the topic before beginning the experimental design process. 2. Ask a specific question and/or formulate an appropriate hypothesis. Then design the experiments to specifically address that problem/question. 3. Consult a biostatistician during the design phase of the project, not after performing the experiments. 4. Choose proper controls to ensure that only the variable of interest is evaluated. More than one control is frequently required. 5. Start with a small pilot project to generate preliminary data and work out procedures and techniques. Then proceed to larger scale experiments to generate statistical significance. 6. Modify original question and procedures, ask new questions, and begin again. This is an obvious place to start, and while in most cases this is clear, it can often be difficult to define. Once the general area of potential research is decided upon (e.g. anatomy, physiology, cellular and molecular biology, microbiology, cardiology, neurology, neurobiology, and the list goes on), the next step is to determine the question to be asked and studied. By having a general area of interest, the literature search is greatly enhanced. A literature search should be performed to determine what is known about the area in question and eliminating possible duplication. A search should be performed in the appropriate data bases or indexes: (See Section VI. for links such as PubMed, etc.). The search should include current and past journal articles and textbooks, as well as information available via the internet. Several “key” words dealing with the subject in general and specifically should be used as the search words. The goals of the literature search are to learn of pertinent studies and methods, identify appropriate animal models, and eliminate unnecessary duplication of research. The “3Rs” of animal research should also be considered at this stage: reduction of animal numbers, refinement of methods, and replacement of animals by viable non-animal alternatives when these exist (See Section VI. for links such as ALTWEB, etc.). The literature search is also an important component of an institutional animal care and use committee (IACUC) protocol submission to provide evidence that the project is not duplicative, that alternatives to the use of animals are not available, and that potentially painful procedures are justified and appropriately addressed. The basis of experimental design is the scientific method. According to many scientists, the scientific method consists of four basic steps: (1) Observation and description of scientific phenomena, (2) Formulation of the problem statement and hypothesis, (3) Use of the hypothesis to predict the results of new observations, and (4) The performance of methods or procedures to test the hypothesis. A detailed description of the experimental manipulations to be used must be carefully thought over and written. The exact methods to be used for data collection must be decided and any known hazards specified. In order to know the potential hazards, the procedures, manipulations, and methods must be extremely well detailed and thought out. All procedures should be detailed in written standard operating procedures (SOPs). This will include determining the testing parameters: what will be measured and how. The overall practicality of the project, as well as the time frame for data collection and evaluation, needs to be planned. Problem Statement, Objectives, Hypotheses A clear definition the problem statement, objectives, and hypotheses is crucial to a successful research project. The problem statement should include the issue that will be addressed experimentally as well as its significance (e.g., potential application to human or animal health, improved understanding of biological processes). The objectives should be stated in a general description of the overall goals for the proposed experiments. The specific questions must also be thoroughly addressed. If a hypothesis is used, it should include two distinct and clearly defined outcomes for each proposed experiment (e.g., a null and an alternate hypothesis). These outcomes may be thought of as the two experimental answers to the specific question being investigated: The null hypothesis is defined as no difference between experimental groups, and the alternate hypothesis is defined as a real difference between experimental groups. Development of a clearly stated problem statement and/or the hypotheses is necessary to proceed to the next stage of the experimental design process. These may (and likely will) be modified as the process continues. Examples of a problem statement and various types of hypotheses follow:
The next step involves choosing an animal model, which requires a thorough literature review. The most appropriate model for the problem being addressed, the lowest on the phylogenic scale, and the costs involved should be considered. In choosing the most appropriate animal models for proposed experiments, the following are recommended:
In knowing the procedures required to complete experiments, it can be determined what additional expertise, if any, will be necessary to perform the research. It is important early in the project development process to identify and consult with potential collaborators to determine who will be working on the project and in what capacity (e.g., as co-investigators, consultants, or technical support staff). Collaborator input into the logistics and design of the experiments and proper sample acquisition are critical to ensure the validity of the data generated. Large research institutions may provide Core facilities that provide many services involving highly technical procedures or require expensive equipment. Early identification of existing core facilities can often lead to the development of a list of potential intramural collaborators. Submission of a proposal to a granting agency requires a detailed budget. This must include: animals (acquisition and maintenance costs), personnel costs (salary, consulting fees, etc.), equipment, including renovations and maintenance fees that are necessary for the equipment and/or facilities, all necessary supplies, travel expenses, publication costs, and any other miscellaneous costs that may be incurred. The appropriate protocol forms must be completely and accurately filled out and submitted to the Institutional Animal Care and Use Committee (IACUC). (Refer to the Lecture on IACUC Protocol Review). Submission of Project to a Granting Agency Once all the paperwork is completed for a specific granting agency, the project is routed through various University or Institutional pathways for appropriate signatures. It is then mailed to the Granting agency. This usually takes place 4-8 months prior to the start date of funding, depending on the agency. A project may not begin with out approval from the IACUC, the Granting agency, and the Institution.Research Plan Development – Scientific Methods As stated previously, a carefully devised and documented description of the experimental manipulations must be made. This requires addressing the problem statement, objectives, and hypotheses. This description should specify the experimental variables that are to be manipulated, suitable test parameters that accurately assess the effects of experimental variable manipulation, and the most appropriate methods for sample acquisition and generation of the test data. The overall practicality of the project as well as the time frame for data collection and evaluation are determined at this stage in the development process. Practical issues that need to be addressed include the lifespan of the animal model (for chronic studies), the anticipated progression of disease in that model (to determine appropriate time points for evaluation), the amount of personnel time available for the project, and the costs associated with performing the experiments. If the animals are to receive chemical or biological treatments, an appropriate method for administration must be identified (e.g., per os via the diet or in drinking water [soluble substances only], by osmotic pump, or by injection). Known or potential hazards must also be identified, and appropriate precautions to minimize risk from these hazards must be incorporated into the plan. All experimental procedures should be detailed through standard operating procedures (SOP), a requirement of good laboratory practice standards (EPA 1989; FDA 1987). Finally, the methods to be used for data analysis should be determined. If statistical analysis is required to document a difference between experimental groups, the appropriate statistical tests should be identified during the design stage. A conclusion will be drawn subsequently from the analysis of the data with the initial question answered and/or the hypotheses accepted or rejected. This process will ultimately lead to new questions and hypotheses being formulated, or ideas as to how to improve the experimental design. The entity under study is the experimental unit, which could be an individual animal or a group. For example, an individual rat is considered the experimental unit when a drug therapy or surgical procedure is being tested, but an entire litter of rats is the experimental unit when an environmental teratogen is being tested. For purposes of estimating error of variance, or standard error for statistical analysis, it is necessary to consider the experimental unit (Weber and Skillings 2000). Many excellent sources provide discussions of the types of experimental units and their appropriateness (Dean and Voss 1999; Festing and Altman 2002; Keppel 1991; Wu and Hamada 2000). N Factor: Experimental Group Size The assignment of an appropriate number of animals to each group is of utmost importance. Although formulas to determine the proper number of animals can be found in standard statistical texts, consulting a statistician to ensure appropriate experimental design for the generation of statistically significant results is recommended. Indeed, the number of animals assigned to each experimental group is often determined by the particular statistical test on the basis of the anticipated magnitude of difference between the expected outcomes for each group. The number of animals that can be grouped in standard cages is a practical consideration for determining experimental group size. For example, a standard 71 sq in (460 sq cm) polycarbonate shoebox cage can house up to four adult mice, so group sizes that are divisible by four will maximize group size and minimize per diem costs. A plethora of variables (e.g., genetic, environmental, infectious agents) can potentially affect the outcome of studies performed with animals. It is therefore important to use control animals to minimize the impact of these extraneous variables or to recognize the possible presence of unwanted variables. In general, each individual experiment should use control groups of animals that are contrasted directly to the experimental groups of animals. Types of control groups include positive, negative, sham, vehicle, and comparative.
Positive Controls
Negative Controls
Sham Controls
Vehicle Controls
Comparative Controls Randomization of the animals assigned to different experimental groups must be achieved to ensure that underlying variables do not result in skewed data for each experimental group. To achieve randomization, it is necessary to begin by defining the population. A homogeneous population consists of animals that are considered to share some characteristics (e.g., age, sex, weight, breed, strain). A heterogeneous population consists of animals that may not be the same but may have some common feature. Generally, the better the definition of the group, the less variable the experimental data, although the results may be less pertinent to large broad populations. Methods commonly used to achieve randomization include the following:
Replication & Statistical Significance The use of appropriate numbers of animals or subjects is extremely important and the correct number is extremely critical, thus a statistician should be consulted. These experts can help to determine the total number of animals or subjects necessary to gain statistical significance. The reasons for replication include: 1) The need to estimate the background noise or error variance (this would relate to variables beyond your control such as presence of infectious agents in your animals that you don’t want or possibly don’t even know their presence), and 2) To make experiments “powerful” enough to recognize true differences between groups (statistical significance). The fewer variables in an experiment the more likely it is to succeed. Thus, it is important for the research and animal care staff to work together to reduce the unwanted variables that can occur. Such unwanted variables can include: infectious disease, parasites, animal room noise, physical plant problems such as temperature and humidity variations or extremes, or animals being frightened by experimental manipulation of other animals in the same room (e.g. someone taking blood samples in the presence of other animals). All animals in the room may experience stress from hearing or from the smell hormones produced from those animals in which the procedure was performed. Some variables may have to be worked into the experimental design. For example, if a new synthetic hormone was tested, the ages of the subjects would be important. The use of immature animals would not be a wise variable to include, thus this should be taken into account. The age range may be tested, but broken into 2 or more sub groups: weanling to puberty, puberty to mid-maturity, mid-maturity to senescence. Treatments of the hormone would then be given to each group with appropriate controls (no treatment and perhaps a comparison to a currently used hormone). Once all data is collected, at least two variables would be studied: treatment variations and age variation. Blindness Bias enters an experiment when the data analyzer (or investigator) knows which group is which. Example: a physician giving some patients an anti-inflammatory drug while others receive a placebo. If either the patient or the physician or both know who’s receiving the real drug and the placebo, the experiment is most likely going to be invalid because more will be read into the results than should have been. The patients may want to feel better on the drug, and the doctor may want to see improvement in the patient (this is human nature). Similarly, if a veterinary pathologist is reading lung sides of asbestos-exposed hamsters treated with an anti-fibrogenic agent and knows which animals received the treatment, he may see less fibrosis when in actuality there is no difference from the positive controls. In both cases, doctor and pathologist, the studies should have been done “blind”, that is the doctor/patient and the pathologist should not know which group is which. In both experiments someone else in the study would have to know which is which to sort it all out in the end. In the case of the arthritis drug, the company or an on-site quality assurance person may have randomize the patients and know who received which treatment. In the case of the hamsters, the research technician doing the dosing might be assigned to know the groups and submit only coded samples to the pathologist. Errors in Logic Researchers must use extreme care in setting up their experiments to ensure that the experiment makes sense and the results will be valid. An example of logic errors would be setting up the experiment using the animals with the range of ages from weaning to senescence without subdividing groups by age. Additionally, it may be necessary to sub-divide by breed or strain to ensure there are no differences in ova production. Extrapolations must be made with extreme care. In many cases, results cannot be extrapolated to other breeds, species, sexes, etc. This is not the case when known associations exist, as when comparing knowledge learned about the pathogenesis of a disease in an “experimental animal model” to the human condition. Problems with Computer Data Analysis The researcher has the ultimate responsibility to make sure that his or her work is analyzed correctly. Unfortunately/fortunately, computers aren’t perfect, or at least the people who work them aren’t. It is easy for data to get “mis-entered”, groups switched, etc., so be sure to check everything over more than once! When it comes to reporting data and coming to a conclusion, it is ones reputation at stake when the results are published, so be sure you are accurate in your data collection, analysis, and reporting. Outlier Some observations will be inconsistent with the rest of the data. For example, one mare produces 100 ova after superovulation when the mean of all other mares is 20. When outliers are found, first verify that the records are correct, that data entry error was not made, or to see if there was some unusual occurrence or circumstance in how the experiment was carried out with he individual unit. If nothing unusual occurred and the data is indeed correct, you may have reason to throw out the outlier. If multiple outliers are observed, a close look at the experimental methods may show signs of sloppy work. A statistician should be consulted to determine if a large number of “outliers” is actually that, or true data not to be discarded, but rather considered as outcomes of the research. Experimental Protocol Approval Animal experimentation requires IACUC approval of an animal care and use protocol if the species used are covered under the Animal Welfare Act (regardless of funding source), the research is supported by the National Institutes of Health and involves the use of vertebrate species, or the animal care program is accredited by the Association for the Assessment and Accreditation of Laboratory Animal Care International. In practice, virtually all animal experiments require IACUC approval, which entails full and accurate completion of appropriate protocol forms for submission to the IACUC, followed by clarification or necessary modification of any procedures the IACUC requires. Approval must be obtained before animal purchase or experimentation, and is required before submission of a grant proposal by some funding agencies (e.g., National Institutes of Health). If the research involves hazardous materials, then protocol approval from other intramural oversight committees or departments may also be required (e.g., a Biosafety Committee if infectious agents or recombinant DNA are to be used, or a Radiation Safety Committee if radioisotopes or irradiation are to be used). Animal welfare regulations and Public Health Service policy mandate that individuals caring for or using research animals must be appropriately trained. Specifically, all personnel involved in a research project must be appropriately qualified and/or trained in the methods they will be performing for that project. The institution where the research is being performed is responsible for ensuring this training, although the actual training may occur elsewhere.Pilot studies use a small number of animals to generate preliminary data and/or allow the procedures and techniques to be solidified and "perfected" before large-scale experimentation is begun. These studies are commonly used with new procedures or when new compounds are tested. Preliminary data are essential to show evidence supporting the rationale of a proposal to a funding agency, thereby increasing the probability of funding for the proposal. All pilot projects must have IACUC approval, as for any animal experiment. As soon as the pilot study is completed, the data will allow further evaluation of the question, hypotheses and procedures. If modifications are necessary these can be evaluated by an additional pilot study or a full study may begin, once IACUC approval has been obtained. The researcher has the ultimate responsibility for collecting, entering, and analyzing the data correctly. When dealing with large volumes of data, it is especially easy for data entry errors to occur (e.g., group identifications switched, animal identifications transposed). Quality assurance procedures to identify data entry errors should be developed and incorporated into the experimental design before data analysis. This process can be accomplished by directly comparing raw (original) data for individual animals with the data entered into the computer or with compiled data for the group as a whole (to identify potential “outliers,” or data that deviates significantly from the rest of the members of a group). The analysis of the data varies depending on the type of project and the statistics required to evaluate it. There are many outstanding books and articles on statistical analysis. A few suggested readings include: Cobb 1998, Cox and Reid 2000, Dean and Voss 1999, Festing and Altman 2002, Lemons et al. 1997, Pickvance 2001, Wasserman and Kutner 1985, Wilson and Natale 2001, and Wu and Hamada 2000. Detection of flaws in the developing or final experimental design is often achieved by several levels of review that are applicable to animal experimentation. For example, grant funding agencies and the IACUC provide input into the content and design of animal experiments during their review processes and may also serve as advisory consultants before submission of the grant proposal or animal care and use protocol. Scientific peers and the scientific literature also provide invaluable information applicable to experimental design, and these resources should be consulted throughout the experimental design process. Finally, scientific peer-reviewed journals provide a final critical evaluation of the soundness of the experimental design. The overall quality of the experimental data is evaluated and a determination is made as to whether it is worthy of publication. Obviously, discovering major experimental design deficiencies during manuscript peer review is not desirable. Therefore, pursuit of scientific peer review throughout the experimental design process should be exercised routinely to ensure the generation of valid, reproducible, and publishable data. Barrow J. 1991. Theories of Everything. New York: Oxford University Press. Bennett BT, Brown MJ, Schofield JC. 1990. Essentials for animal research: A primer for research personnel. In: Alternative Methodologies. Beltsville: USDA National Agricultural Library, University of Illinois at Chicago. p 13-25. Blount RL, Bunke VL, Zaff JF. 2000. Bridging the gap between explicative and treatment research: A model and practical implications. J Clin Psych Med Set 7:79-90. Cobb GW. 1998. Introduction to Design and Analysis of Experiments. New York: Springer. Cox DR, Reid N. 2000. The theory of the design of experiments. In: Monographs on Statistics and Applied Probability, 86. Boca Raton: Chapman & Hall/CRC Press. Dean AM, Voss D. 1999. Design and Analysis of Experiments. New York: Springer. De Boer J, Archibald J, Downie HG. 1975. An Introduction to Experimental Surgery: A Guide to Experimenting with Laboratory Animals. New York: Elsevier. Diamond WJ. 2001. Practical Experiment Designs for Engineers and Scientists. 3rd ed. New York: Wiley. EPA [Environmental Protection Agency]. 1989. Good Laboratory Practice Regulations. Federal Register 40, chapter 1, part 792. FDA [Food and Drug Administration]. 1987. Good Laboratory Practice Regulations. Federal Register 21, chapter 1, part 58. Festing MFW, Altman DG. 2002. Guidelines for the design and statistical analysis of experiments using laboratory animals. ILAR J 43(Suppl):000-000. Holmberg P. 1996. From dogmatic discussions to observations and planned experiments: Some examples from early aurora borealis research in Finland. Sci Educ 5:267-276. Keppel G. 1991. Design and Analysis: A Researcher’s Handbook. 3rd ed. Englewood Cliffs: Prentice Hall. Kuhn T. 1962. The Structure of Scientific Revolutions. Chicago: University of Chicago Press. Larsson NO. 2001. A design view on research in social sciences. Syst Prac Act Res 14:383-405. Lawson AE. 2002. What does Galileo’s discovery of Jupiter’s moons tell us about the process of scientific discovery? Sci Educ 11:1-24. Lemons J, Shrader-Frechette K, Cranor C. 1997. The precautionary principle: Scientific uncertainty and type I and type II errors. Found Sci 2:207-236. Pickvance CG. 2001. Four varieties of comparative analysis. J Hous Built Env 16:7-28. Russell WMS, Burch RL. 1959. The Principles of Humane Experimental Technique. London: Methuen & Co. Ltd. [Reissued: 1992, Universities Federation for animal Welfare, Herts, England.] http://altweb.jhsph.edu/publications/humane_exp/het-toc.htm>. Silverman J, Suckow MA, Murthy S, NIH IACUC. 2000. The IACUC Handbook. Boca Raton: CRC Press. Spector R. 2002. Progress in the search for ideal drugs. Pharmacology 64:1-7. Sproull NL. 1995. Handbook of Research Methods: A Guide for Practitioners and Students in the Social Sciences. 2nd ed. Metuchen: Scarecrow Press. Wasserman W, Kutner MH. 1985. Applied Linear Statistical Models: Regression, Analysis of Variance, and Experimental Designs. 2nd ed. Homewood: RD Irwin. Weber D, Skillings JH. 2000. A First Course in the Design of Experiments: A Linear Models Approach. Boca Raton: CRC Press. Webster IW. 1985. Starting to do research. Med J Aust 142:624. Whitcom PJ. 2000. DOE Simplified: Practical Tools for Effective Experimentation. Portland: Productivity. Wilson EB. 1952. An Introduction to Scientific Research. New York: McGraw-Hill. Wilson JB, Natale SM. 2001. “Quantitative” and “Qualitative” research: An analysis. Int J Value-Based Mgt 14:1-10. Wu CF, Hamada M. 2000. Experiments: Planning, Analysis, and Parameter Design Optimization. New York: Wiley. Zolman JF. 1993. Biostatistics: Experimental Design and Statistical Inference. New York: Oxford University Press. VI. WEB SITE LINKS – Databases and Indexes AGRICOLA THE AMERICAN ACADEMY OF CLINICAL TOXICOLOGY http://www.clintox.org/AcademyLinks/NonGov_Tox.html CBIR (CREATIVE BASED INFORMATION RESOURCES) http://www.buffalostate.edu/orgs/cbir/ MEDLINE http://www.medlineplus.gov/ MEDNETS http://www.mednets.com/ NCBI (National Center for Biotechnology Information) - http://www.ncbi.nlm.nih.gov/ PUBMED - http://www.pubmed.gov SCICENTRAL - http://www.bio.net/bionet/mm/cellbiol/1998-April/008580.html SCIRUS - http://www.scirus.com/srsapp/ TOXLINE - http://toxnet.nlm.nih.gov/cgi-bin/sis/htmlgen?TOXLINE UNCOVER the NET – Science - http://www.uncoverthenet.com/dir/13/1.php Links for look for ALTERNATIVES TO ANIMALS AS MODELS (as required by the Animal Welfare Act) It is recommended to use at least one search engine to specifically address alternative techniques (in addition to the research links). The following sites can help you develop strategies for searching for alternatives and can link you to data bases that will perform the searches: The 3R’s: ALTWEB: Alternatives to Animal Testing on the Web serves as a gateway to alternatives news, information and resources on the Internet and elsewhere. - http://altweb.jhsph.edu/ ALTBIB: Within the National Library for Medicine website. - http://toxnet.nlm.nih.gov/altbib.html AnimAlt-ZEBET Database: This German site provides a full-text database of alternative methods (3Rs) to animal experiments in biomedicine and related fields. http://www.dimdi.de/static/en/db/index.htm In Vitro Methods: ECVAM SIS (Scientific Information Service) & EURCA (European Resource Centre for Alternatives in Higher Education): Provides factual and evaluated information on advanced non-animal methods for toxicology assessments; offers full method descriptions, including development and validation status. It also provides information about alternatives to using animals in higher education. http://www.nc3rs.org.uk/category.asp?catID=3 INVITRODERM: Provides alternatives to skin irritation/corrosion testing in animals. - http://www.invitroderm.com/ Refinement Methods: ALTWEB Pain Management (Anesthesia/Analgesia): Information about anesthesia and analgesia for most commonly used laboratory animals. http://apps1.jhsph.edu/altweb/aadb/aadb_search.cfm ALTWEB Humane Endpoints: Designed to help researchers find the earliest "endpoint" that is compatible with the scientific objectives of their research, i.e., the earliest point at which an experimental animals pain and/or distress is terminated, minimized, or reduced. http://apps1.jhsph.edu/altweb/humane/ AWIC (Animal Welfare Information Center): Alternatives: This site provides many avenues to consider for searches and gives information on how to design your search for alternatives. http://riley.nal.usda.gov/nal_display/index.php?info_center=3&tax_level=1&tax_subject=183 UCCAA (University of California Center for Animal Alternatives): http://www.vetmed.ucdavis.edu/Animal_Alternatives/database.htm
Page designed and maintained by K. Coronado |
 |
 |
 |
 |
 |
