Back to Page 1


_

Appendices

Appendix A: Educational Research Design: Testing for Effectiveness
Appendix B: The Content of Phonics Instruction
Appendix C: Options for Calculating Readability
Appendix D: A Rule-Based Procedure for Summarization

Appendix A
Educational Research Design: Testing for Effectiveness

This appendix is included to provide more depth on the subject of research design than we provided in the main text, but it is not a comprehensive discussion of this subject. We have avoided the technical details whenever possible. Our goal is to explain the underlying concepts so that you can be a more informed consumer of educational research. If you understand the thinking behind the process of proving effectiveness, you will know what kind of questions to ask when you are confronted with a teaching method or product that purports to be research based.

Introduction

When an educational researcher wants to test the effectiveness of a particular instructional method, approach, or product, he or she must find out whether students who receive this new "treatment" (also often called an intervention) learn more than those who don't. This kind of study, then, requires two groups of students: those who are taught using the new method and those who are taught using the current methods. The researcher can compare pre- and posttest scores of the students in each group to see if one group made greater gains than the other.

However, even if the treatment group (let's call them Group A) did better on the posttest or showed greater gains than the other group (Group B), we still can't be sure the difference in learning was caused by the treatment. Maybe Group A had better skills at the beginning. Maybe they were more motivated. Maybe Group B included more students with special learning needs. Maybe some of the Group A students had extra tutoring or other advantages. Any of these factors (variables) could have affected the learning gain.

In order to isolate the learning that may be attributed to the new instructional approach, the researcher must "control" for these and other variables that may also influence the learning outcome. It's fairly easy to control some of the factors. For example, a researcher can make sure that the same amount of time is spent on instruction for both groups. But what about the student characteristics? How can the researcher control those variables?

Remember that the goal is to ensure that Group A and Group B are very similar, so that the only important difference is the treatment--the instructional approach, method, or product. So she tries to "match" the groups on relevant characteristics. If the groups are similar and the researcher is changing only one thing that happens in the classrooms (the treatment), it is logical to assume that if the groups have different outcomes, it's the treatment that caused that difference.

The key is to work with groups that are very similar, and there are two ways to approach this goal.

Experimental Research

In a true experiment, the researcher assigns students to the two groups using a random procedure. For example, he or she might "draw names from a hat" or choose every other name on the enrollment list. A random method gives every person an equal chance to be assigned to either group and makes it likely that any differences in the groups will "balance each other out." This approach prevents a researcher's unintended biases from influencing the composition of groups.

It does not, unfortunately, ensure that groups are closely matched, and researchers may use other methods (too technical for this book) to address these concerns. One way to get around some of these difficulties is to work with large numbers. When the number of participants (the n) is large, the odds are better that group differences will be balanced out by the random assignment procedure.

Large numbers, then, lend credibility to a study. If the number of students being considered for participation is large enough and assignment is left to chance, these conditions increase the probability that Groups A and B will not be markedly different on relevant characteristics (skills, native language, prior education, etc.).

random elements pointing to either Group A or Group B

In education, this kind of research is often difficult because researchers can't assign individual students to different groups when they are already assigned to classes. For this reason, educational researchers often have to work with existing groups, usually in the form of classes. When researchers work with existing groups and random assignment of individuals is impossible, they are doing quasi-experimental research.

Quasi-Experimental Research

Following are two approaches to quasi-experimental design.

Random assignment of groups. In one approach to this type of design, the researcher randomly assigns groups (instead of individuals) to receive--or not receive--the treatment. So for instance, if several Adult Basic Education (ABE) programs are participating in a study testing a new reading comprehension strategy, half of the classes would be randomly assigned to the treatment Group A, and half would be in Group B.

Pre-test score equivalency. Random assignment of groups isn't always possible or practical, however. Another approach to the quasi-experimental design involves working with existing groups and administering a pre-test to the students in each class in order to establish equivalency on reading skills.

Matching. Of course, the groups still may differ in important ways, so the researcher must check on a number of other characteristics. For example, he or she will check to see if the ages of the two groups are similar and if both groups have approximately the same percentage of adults whose first language isn't English.

In addition, when working with groups, classroom characteristics are important. Adult education classroom schedules and instructional formats may differ in significant ways, making comparisons between classes problematic. For instance, one class might meet on Monday and Wednesday, while another meets four days a week. One class might have average attendance of only 50%, while another that is offered on the worksite or provides child care might have attendance closer to 90%. These variations in program settings result in large differences in the amount of instruction learners receive, and "time on task" is obviously an important variable.

You can see that both the characteristics of the learners and the characteristics of the instructional setting are important. A researcher should consider both types of variables when assembling groups to participate in a study. One approach is to first define the individual and classroom criteria for inclusion, exclusion, and matching, and then select the classes that are roughly similar for participation in the study. This approach is not as good as individual random assignment but does address many of the practical restraints of applied research.

In the example detailed in Table A-2 students in eight classrooms comprised the population initially being considered for the study. After applying the exclusion criteria, Classroom #3 was excluded because most of the students were recent immigrants. Their English language skills were much weaker than the students in the other classes, so the researcher decided this group was too different to be fairly compared with the others in a study of English reading-strategy instruction. In the table, selected characteristics of the remaining seven classes are compared. As you can see, the two groups, although not identical, are similar.

Participating Classes*
Student & Class Characteristics Group A--
Classes: #4, #6, #8
Group B--
Classes #1, #2, #5, #7
Instruction Treatment Group: Instruction in new comprehension strategy Comparison Group: Reading instruction as usual
Population Adults enrolled in ABE; voluntary participants; 32% Hispanic/Latino (including 25% recent immigrants); 30% African American; 10% other non-native speakers of English; 28% White native speakers of English Adults enrolled in ABE; voluntary participants; 30% Hispanic/Latino (including 22% recent immigrants); 34% African American; 12% other non-native speakers of English; 24% White native speakers of English
Age: range & average 18-42; average 23 20-39; average 26
Years of school completed: range & average 4-11; average 8.5 5-11; average 9.5
Reading comprehension pre-test score (grade equivalent) 3.5-10.9 GE; average 6.2 GE 4.2-11.2 GE; average 6.7 GE
Class instructional hours 6 hours weekly (3 days x 2 hours) 6 hours weekly (2 days x 3 hours)
Average daily learner attendance 72% 67%

*After applying the researcher's exclusion criteria, Classroom 3 was excluded from consideration for participation in the study because most of the students were recent immigrants.

Table A-2


When individual and classroom variables are "controlled" in this way, the researcher may draw tentative conclusions about the effectiveness of the method or approach being tested. Because the groups are similar in these important ways, if Group A shows significantly greater learning gains, the results will point to the instructional treatment as the cause of the difference. Again, a larger number of classrooms would lend even greater support to these conclusions.

Replicated Research

Of course, no single study provides conclusive evidence of effectiveness. Other researchers may try out the intervention with different learner groups or in different program settings to see whether, or to what extent, the findings may be generalized. When several studies report similar findings, the research has been replicated, and results may be accepted with greater confidence.



Appendix B:
The Content of Phonics Instruction

This outline is based on the phonics content described in the sources listed below.

Consonant and short vowel sounds
  • Consonants: The sounds for all the consonants are introduced (a few at a time) because they are fairly consistent in the sounds they represent, and this knowledge is immediately useful. The two sounds for c and g are taught, but are not generally introduced at the same time. So, for example, /k/ as in cat and /s/ as in city are not taught together.
    The continuant sounds, like /s/, /m/, and /f/, may be among the first consonant sounds taught because you can hold onto the sounds and "slide" into the vowel (Gunning, 2001). That makes it easier to blend sounds to form a word. Stop sounds, like /p/, /d/, and /k/ can't be continued in that way, so the speaker has to add a vowel sound. The sound of the letter d ends up as "duh" and that added vowel sound makes it harder to recognize the word.
    A common practice is to teach a key word for each sound as a memory aid. A good key word begins with the target letter sound (not a consonant blend like br or sp, just the consonant followed by a vowel) and is a concrete term familiar to the learners: bus, cat, dollar, fish, gift, etc.
  • Consonant digraphs: Consonant digraphs are two letters that stand for one sound: ch, sh, th, wh, and ph. Key words may be used for these sounds, too.
  • Short vowels: The vowel sounds are introduced in the same way. The short vowels are sometimes taught first because there are more different spelling patterns for the long vowel sounds.
    Typically, students learn a few consonant sounds and at least one or two vowel sounds right away, so they can decode words immediately.
Sight words

The most frequently encountered words are taught as sight words (the, in, of, etc.). (You can find a list of these words in The Reading Teacher's Book of Lists, Fry et al., 2000.) Knowing these high-utility words allows beginners to read simple materials independently and allows teachers and learners to compose sentences so they can practice decoding words in a context.

Onsets and rimes

Many programs introduce onsets and rimes (also called word patterns and phonograms) early in the sequence, after one or two vowels and several consonants have been learned. This is a good approach for dealing with the vowel sounds because learners may have a hard time detecting a vowel sound in the middle of a word. They may find it easier to recognize a larger word part, like ap, or ot. So, for instance, after working with short a, they might learn the rimes ad, an, or ack and read words with different onsets, like bad, had, mad, and dad. (Of course, they can only decode words with letter sounds they already know, so this example works only if they know the sounds for b, h, m, and d.)

Long vowel sounds

The long vowel sounds may be first introduced with spellings that exemplify the silent e rule, because this rule is fairly consistently applicable. (Examples: take, dime, fine, hope) Onsets and rimes are also useful in working with the long vowels. (Examples: ay, ail, eed, ight)

Consonant blends

The consonant blends or clusters (br, cr, dr, bl, pl, sn, st, scr, etc.) are usually taught directly. Learners may attach these new onsets to the rimes they've already learned to create many new words.

Other vowel sounds

The other common vowel combinations are usually introduced after the long and short sounds.

Examples:

  • oi / oy, au / aw, ou / ow, oo (as in zoo), oo (as in book)
  • (shwa: upside down and backwards e the schwa sound represented by the a in ago or the i in pencil)
  • The r-controlled vowels (as in her, bird, card, north, and burn)
Structural analysis

Analyzing words by recognizing larger word parts is often included in phonics instruction. Simple examples are below.

  • Plurals: --s, --es
  • Verb endings: --s, --ing, --ed
  • Common prefixes: re, un, anti, ex, non, pre, post

For more detail on content, consult the following sources:

Fry, E.B.; Kress, J.E.; & Fountoukidis, D.L. (2000). The reading teacher's book of lists, (4th ed.). San Francisco, CA: Jossey-Bass. (See Suggested Phonics Teaching Order, Phonics, Example Words, and Phonograms," pages, 9-42)

Gunning, T.G. (2001). Building words: A resource manual for teaching word analysis and spelling strategies. Needham Heights, MA: Allyn & Bacon. (See "The Content of Phonics," pages 6-10)

Miller, W. H. (2002). Reading skills problem solver: Ready-to-use strategies and activity sheets for correcting all types of reading problems. Paramus, NJ: Center for Applied Research in Education. (See pages 72-82)

Skinner, L., Gillespie, P., & Balkam, L. (1998). Teaching adults who learn differently: An extensive guide for literacy teachers and tutors. San Diego, CA: Red Van Publishers. (See "General Scope and Sequence of Structured Literacy Program," pages 212-213)

Wilson, R. & Hall, M. (1997). Programmed word attack for teachers. Upper Saddle River, NJ: Merrill. (A self-study guide for teachers on word analysis terminology and concepts)


Beginners need a structured, carefully sequenced curriculum designed to develop decoding skills, so your program will need to learn about existing, proven programs. Examples of programs that have been used with adult learners are in Appendix B.



Appendix C: Options for Calculating Readability

Flesch-Kincaid

The Flesch-Kincaid readability formula is based on the number of words, syllables, and sentences in a passage. It yields a "reading ease" score and a grade level. It's easy to use with electronic text because it's one of the tools available in Microsoft Word and Word Perfect. In Microsoft Word, you can find it via the Tools menu, under Spelling and Grammar.

Fry's Readability Graph

The Fry graph is also easy to use, and you can use it with material that you don't have in electronic form. You just select a 100-word passage from the text to be analyzed, count the number of sentences and the number of syllables, and plot those numbers on the graph to calculate the grade level of the material. The graph and directions may be found online at the Discovery School site school.discovery.com/schrockguide/fry/fry.html and in The Reading Teacher's Book of Lists, (Fry, Kress, & Fountikidis, 2000).

Gunning-Fog Index

This formula is based on sentence length and the number of "hard words" (defined as those with more than two syllables). Directions like those below may be found at various sites online.

  • Calculate the average sentence length (ASL)
    Choose a sample at least 100 words long. Count the words and the sentences. Divide the number of words in the sample by the number of sentences to compute the average sentence length.
  • Find the percentage of hard (polysyllabic) words (%P)
    Count the words that have three or more syllables. (Do not include those whose third syllable is es or ed. Do not include proper nouns and other capitalized words, technical terms, or compounds composed of small words.) Divide this number by the total number of words in the sample to compute the percentage of hard words (%P).
  • Add the ASL and the %P and multiply the sum by .4
    ASL + %P = ___ x .4 = grade level

Appendix D:
A Rule-Based Procedure for Summarization

The example below is a procedure for summarizing a paragraph (McNeil & Donant, as cited in Duke & Pearson, 2002).

Rule 1: Delete unnecessary material.
Rule 2: Delete redundant [repetitive] material.
Rule 3: Compose a word to replace a list of items.
Rule 4: Compose a word to replace individual parts of an action.
Rule 5: Select a topic sentence.
Rule 6: Invent a topic sentence if one is not available.

Here's how it might work.

Paragraph to be summarized:
Today's employers are looking for workers with many skills. They want people who are flexible and can do many things well. Good reading and math skills are the minimum requirements for most jobs. Employees must also know how to learn. They must be able to read new information and learn new skills by reading. They need to be able to read technical manuals with charts and graphs and statistics. Many jobs also require computer skills. To use a computer, workers must at least be good readers, but jobs involving technology also demand analytical thinking and problem solving. Finally, employers want people who have good communication skills and who know how to work well with people. These are often called "soft skills," but for many jobs these skills may be the most important keys to success.
1. Delete unnecessary material.

Today's employers are looking for workers with many skills. They want people who are flexible and can do many things well. Good reading and math skills are the minimum requirements for most jobs. Employees must also know how to learn. They must be able to read new information and learn new skills by reading. They need to be able to read technical manuals with charts and graphs and statistics. Many jobs also require computer skills. To use a computer, workers must at least be good readers, but jobs involving technology also demand analytical thinking and problem solving. Finally, employers want people who have good communication skills and who know how to work well with people. These are often called "soft skills," but for many jobs these skills may be the most important keys to success.

2. Delete redundant material.

Today's employers are looking for workers with many skills. They want people who are flexible Good reading and math skills are the minimum requirements for most jobs. Employees must also know how to learn. [They must be able to read new information and learn new skills by reading. They need to be able to read technical manuals.] Many jobs also require computer skills. [To use a computer, workers must at least be good readers,] but jobs involving technology also demand analytical thinking and problem solving. Finally, employers want people who have good communication skills and who know how to work well with people. These are often called "soft skills," but for many jobs these skills may be the most important keys to success.

(Result)
Today's employers are looking for workers with many skills. They want people who are flexible. Good reading and math skills are the minimum requirements for most jobs. Employees must also know how to learn. Many jobs also require computer skills. Jobs involving technology also demand analytical thinking and problem solving. Finally, employers want people who have good communication skills and who know how to work well with people. These are often called "soft skills," but for many jobs these skills may be the most important keys to success.
3. Compose a word to replace a list of items.

Communication skills = soft skills, work well with people. Computer skills = technology. Problem solving = analytical thinking.

(Result)
Today's employers are looking for workers with many skills. They want people who are flexible. Good reading and math skills are the minimum requirements for most jobs. Employees must also know how to learn. Many jobs also require computer skills and demand problem solving. Finally, communication skills may be the most important keys to success.
4. Compose a word to replace individual parts of an action.

Are looking = want, are minimum requirements, must, require, demand, may be most important.

(Result)
Today's employers are looking for workers with many skills: flexible, good reading and math skills, know how to learn, computer skills, problem solving, and communication skills.
5/6. Select a topic sentence or invent one if necessary
Today's employers are looking for flexible workers who have good reading and math skills, who know how to learn, who can use a computer to solve problems, and who have good communication skills.

This procedure may be most appropriate for fairly advanced readers. To know what is unnecessary the reader must already have at least a sense of the main idea of the paragraph. You might have learners create paragraph "maps" first, to identify the main idea and/or work with a partner to think through the decisions to delete material.


Back to top