Return to Index Page
Return to Chapter One

Chapter 2

Method

The guidelines used for selecting and evaluating ABE reading instruction research are based on those developed by the National Reading Panel (NRP) in their review of research related to reading instruction at the K-12 level (National Reading Panel, 2000a, pp. 5-6, 27-33). For the NRP review, major topics for study were established, studies were located through a literature search, and studies were evaluated using a set of "evidence-based methodological standards" (NRP, 2000a, p. 2). The RRWG made several modifications to the approach used by the NRP. Important modifications included the addition of topics especially important to adult reading professionals, the inclusion of studies related to the assessment of reading ability, and the inclusion of non-experimental studies as well as those involving the use of control groups.



Selecting Topics


Following the lead of the NRP, core topics are based on those aspects of reading found by the National Research Council (NRC) and others to be most important in learning to read: alphabetics, fluency, and comprehension (NRC, p. 2; Snow, Burns, & Griffin, 1998). After the topics were discussed among members of the RRWG planning committee and researcher group, additional topics important to the field of ABE were added to the core categories. All topics are listed below.

A. Instructional Focus

  1. alphabetics instruction (including phonemic awareness and word analysis)
  2. fluency instruction
  3. vocabulary instruction
  4. comprehension instruction

B. Instructional Goals and Setting

  1. general functional literacy instruction
  2. workplace literacy instruction
  3. family literacy instruction

C. Instructional Methods and Material

  1. teaching strategies and techniques used for reading instruction
  2. instructional materials used for reading instruction
  3. technology and reading instruction
  4. intensity and duration of reading instruction
  5. teacher preparation and reading instruction

D. Learner Characteristics

  1. functional reading level and reading instruction
  2. ESOL and reading instruction
  3. learning disability and reading instruction
  4. motivation and self-esteem and reading instruction

E. Assessment of Learners' Strengths and Needs for Reading Instruction



Selecting Studies for Inclusion


Four main sources were used to locate relevant research articles: The PsychINFO and ERIC databases, reference lists from relevant articles, and recommendations from adult literacy researchers for relevant research articles that may have been missed in the database searches.

To determine whether or not a study should be included in the review, the following criteria were used.

A. A study must focus on ABE learners' reading development. ABE students are those low-literate adults aged 16 and older who are no longer being served in a secondary education program (Adult Education and Family Literacy Act, Title II of the Workforce Investment Act, PL105-220, 1998). This would include adults scoring on Levels 1 and 2 of the reading test developed for the National Adult Literacy Survey (Kirsch, Jungeblut, Jenkins, and Kolstad, 1993), or those with grade equivalent scores of K through 12 on a test of reading.
B. A study must include reading as an outcome measure. Reading outcome measures used by the NRP include reading real words in isolation or in context, reading pseudowords that can be pronounced but have no meaning, reading text aloud or silently, and comprehending text that is read silently or orally, including both individual vocabulary words and extended text (NRP, 2000a, p. 5).
C. Studies published in refereed (peer-reviewed) journals are given the highest priority. These journals' editors select an editorial board and other independent peer reviewers who use a common set of criteria to review studies submitted for publication. Based on reviewer comments, the editors select articles for publication. The reviews are usually "masked" (information about authors is not provided) and help to insure that only studies without major flaws are published. Articles from non-peer-reviewed journals are reviewed separately and "treated as preliminary/pilot data that illuminate potential trends and areas for future research" (NRP, 2000a, p. 29). In addition, only five non-journal sources have been selected for inclusion. Two of these were selected because they describe in more detail data referred to in journal articles. The others report results from three national surveys of adults (the National Assessment of Educational Progress Young Adults Survey and the National Adult Literacy Survey in the United States, and a large survey in Britain).
D. A study must contain a full description of outcome measures.
E. A study must contain careful and complete descriptions of the adults participating (age, demographic, cognitive, academic, and behavioral characteristics) and must contain enough information to make judgments related to validity (NRP, 2000a, p. 28).
F. Any interventions (and assessment procedures) used in a study must be described in sufficient detail to enable the study to be replicated.
G. Results from intervention studies using an experimental or quasi-experimental design are given highest priority. These are both referred to as experimental results. Experimental results must be based on valid comparisons between groups with differences between groups tested statistically for significance. Correlational and other non-experimental results may be used to support experimental studies in a topic area, or as preliminary/pilot data if no experimental data exist in a topic area.
H. Non-experimental results from qualitative studies must be based on a sound analytical framework. Qualitative reading research focuses on literacy processes as opposed to quantifiable, numerical data related to growth in reading. The following description of qualitative research is paraphrased from the Handbook of Qualitative Research (Denzin & Lincoln, 2000, pp. 3-8): Qualitative research includes case study; personal experience; introspection; life story; interview, artifacts and cultural productions; and observational, historical and visual texts. It is not defined by any distinct set of methods or practices. It includes ethnography, interviews, survey research, and participant observation. It emphasizes the qualities of entities, and processes and meanings that are not experimentally examined or measured (if measured at all) in terms of quantity, amount, intensity or frequency. It emphasizes the socially constructed nature of reality, the intimate relationship between the researcher and what is studied, the situational context that shapes the inquiry, and the value-laden nature of inquiry. In contrast, quantitative research measures and analyzes causal relationships between variables, not processes, and claims a value-free framework.
Like quantitative-descriptive research, qualitative research may be used to support results from experimental studies (and vice versa). Qualitative research may employ multiple methods and so, as Denzin & Lincoln (2000) state, may include the use of quantitative measures, although these are not the focus of the research. In cases where a qualitative research study includes numerical analyses of data, these may be categorized according to the quantitative framework described above. The highest quality qualitative studies are those that collect data using multiple methods and use triangulation of these methods to support findings and any conclusions drawn from them. For techniques such as data coding (whether from transcripts, video tapes, or field notes), training and inter-rater/coder reliability should be performed.
Only a few qualitative studies have been selected and all are case studies of individual adult learners. Other non-experimental results selected include only those studies with comparisons made between two groups, or comparisons made of one group at two or more points in time.
I. Correlational and other non-experimental results are appropriate when considering studies in the Assessment category. Assessment studies may simply describe ABE students' reading, unlike instructional studies which look for change over time resulting from some aspect of instruction. In assessment studies, planned comparisons between groups over time may not be appropriate. Assessment studies using adequate sampling procedures and inferential statistics when appropriate to analyze data are given highest priority and the same weight as experimental studies.

As mentioned in the introduction, results from the NRP K-12 research synthesis are used, when appropriate, to support and augment findings from the ABE reading instruction research base. Criteria used for including NRP findings are listed below, roughly in the order they are used, with those listed first given a higher priority:

A. The instructional research at the K-12 level supports limited, research-based findings at the adult level.
B. The instructional approach derived from K-12 research can plausibly be used with adults. This "plausibility criterion" is needed so that research-based results that may be effective at the K-12 level are not blindly applied to adults. For example, research may demonstrate that a particular entry-level basal reader is extremely effective with children, but this research result may not necessarily lead to a recommendation that the basal reader be used in an adult workplace literacy program.
C. The instructional approach derived from K-12 research is based on a strong body of evidence. The stronger the result at the K-12 level, the more likely it is to eventually be shown to be effective at the adult level. Strength can be measured along two dimensions: depth and breadth. A finding has depth when it has been replicated and the effects summarized over replications are strong. The NRP, for example, defines a strong finding as one for which statistically derived effect sizes are moderate to large. A finding has breadth when it applies to a wide range of conditions. These conditions may be related to the learners, for example. A broad finding would be one that holds for learners at different age or ability levels. A finding may hold for various instructional settings or conditions, such as in- or out-of-school settings, small group, classroom, or tutoring situations, various subject or content areas, or different levels of teacher preparation or expertise. The same finding may result regardless of the types of assessment used (informal or formal, for example). A finding that has both depth and breadth is probably one that could be tried with adults, absent research-based direction at the adult level.
D. The instructional approach has been shown to work at the K-12 level with those who have not followed normal age and ability level development in their reading. Adults in ABE programs are, in a sense, "out of grade level." They may be working on skills that others (and they themselves) worked on as elementary or high school students. They are older learners of specific reading skills. They may also be more likely to have a reading disability (Snow & Strucker, 2000). Therefore, those results at the K-12 level that apply to reading disabled or relatively older students may be of interest to adult educators. Instructional practices that work with younger disabled readers, those who have received instruction but whose reading is well below average, may be of use to adult educators working with adults who are also older and well below average in their reading ability.


Deriving Principles


Principles were derived from qualifying research studies by first placing the studies into the categories identified by the RRWG, based on reading outcome measures and independent variables. Studies with common themes within each category were grouped together and their results were summarized as succinct principles or trends.

Findings from groups of studies that contained two or more experimental studies with compatible results (and any number of non-experimental studies) were labeled principles. Findings based on fewer than two experimental studies were labeled trends. Non-experimental findings were used as convergent evidence for the Trends and Principles. In the assessment categories, assessment studies with adequate sampling procedures and inferential statistics were given the same weight as experimental studies.

Exceptions to the above were made for the most recently published national, large-scale studies of adults learners, including the National Adult Literacy Survey (Kirsch et al., 1993), the NAEP (National Assessment of Educational Progress) study of young adults (Gallo, 1972), and the NFER (National Foundation for Educational Research) study in Britain (Brooks, Davies, Ducke, Hutchison, Kendal, & Wilkin, 2001). In the case of results from the NALS, for example, some principles were based on results from the NALS alone because the sampling procedures used allowed the generalization of findings to the whole adult population.

All except a few of the principles derived from the research might be considered "emerging principles" because they are based on a relatively small body of experimental research.

Back to top
Back to Chapter One
Next