Dr. Eunice Askov’s Response to Questions
Dr. Eunice Askov’s Response to Questions from the Workplace Literacy Discussion List
In August 1997, the NIFL Workplace Listserv invited Dr. Eunice (Nikkie) Askov to provide responses to questions about workplace literacy program evaluation. Dr. Askov has had ten years’ experience evaluation programs funded by the United States Department of Education National Workplace Literacy Program. She also serves as an external evaluator of three NWLP projects.
The questions posed by the workplace literacy listserv members focused mostly on evaluation. The Q and A session has been divided into the following categories:
- General information on workplace literacy programs and evaluation
- Long-term authentic evaluation
- Legal aspects of evaluation
- Evaluating workplace literacy materials
- Participatory workplace evaluation
- Qualifications for being an outside evaluator
Inviting the listerv to participate (Barbara Van Horn)
Dear subscribers,
Our guest speaker, Dr. Nickie Askov, will be available all this week to
answer your questions on evaluating workplace literacy/workforce education
programs. Don't be shy about asking -- we are looking forward to hearing
your questions and concerns about program evaluation!
Dr. Askov is considering the questions and will reply throughout the week.
Response (Dr. Eunice Askov)
Dear NIFL-Workplace List Serv:
Thanks for your terrific and challenging questions! I am really glad that you all are thinking in depth and showing lots of experience in the issues around evaluation of workplace literacy programs. My hope is that you all will contribute to the answers to these questions. I will take a stab at answering them, but please chime in with your thoughts. My answers are based on experience over the past ten years of evaluating programs funded by the US Department of Education's National
Workplace Literacy Program. I currently serve as external evaluator of 3 NWLP projects--Community College Board of Colorado ("front range" community colleges), Wisconsin's Technical and Adult Education Board (including many
of the 2-year postsecondary institutions in Wis.), and College of Lake County (north of Chicago).
On general developing a comprehensive workplace evaluation plan
1. I would appreciate comments re: evaluating my workplace (preparation)
program. I administer (and teach) "workplace know-how skills" ranging a
full gamut from reading to math to computer literacy, to writing,
listening, problem solving, etc. -- preparing adults with disabilities
and /or disadvantaging conditions for employment. I would like to form an advisory council to assist in this process, as well as program promotion, fund seeking, etc.
Response (Dr. Eunice Askov)
Thanks for your question about how to evaluate the "full gamut" of
your programs. My question to you is: Who needs the information and why? That
will shape your evaluation. If you need evaluation data to demonstrate
program effectiveness to funding agents, then you need to involve them in
determining the types of data that would be most useful. If you are
working with a company delivering workplace literacy, then obviously the
company becomes an important stakeholder. (August 5, 1997 answer to J.M.L. Fulton)
do evaluation of workplace literacy programs.) An advisory council for
your program would be very helpful, especially in being responsive to
community needs as well as in developing multiple funding sources. We rely
heavily on stakeholder groups (which is really an advisory council) in
designing training evaluation for the workplace. (See my response to Paul
Jurmo for the name and ordering info. of our manual.) If you would like to
ask a more specific evaluation question, go ahead. Thanks.
2. Could you provide us with a general overview of your approach to workplace literacy evaluation, then provide answers to the specific question proffered?
Response (Dr. Eunice Askov)
First, let me describe my approach to workplace literacy evaluation that I am using in the 3 NWLP projects where I serve as external evaluator. I follow Kirkpatrick's four
levels of evaluation because they seem to work and get at essential information
for summative evaluation. (Reference to follow.) I also do formative
evaluation by asking the same questions in interviews over the 3-year time
period so I can provide useful feedback for program improvement. The first
level of Kirkpatrick's "hierarchy" is satisfaction on the part of all
stakeholders--workers, supervisors, training director, management, union,
educational provider, instructor, etc. Usually this is measured with a
brief questionnaire by the provider or company. I do in depth structured
interviews also to get an objective look at satisfaction as well as
suggestions for improvement. The second Kirkpatrick level is mastery of
the training content. The provider needs to demonstrate that the workers
have learned the content that has been taught. To me, this seems to be the
point of the program! (If you can't demonstrate gains in relevant basic
skills in the work context, you are in trouble!) The third level is
transfer to the workplace. Here you look for carryover of instruction into
the daily jobs. This can be measured by supervisor and worker interviews
(which I do as external evaluator) or questionnaires. One of my evaluation
sites--Community College Board of Colorado--has done a lot with this by
having workers write down on a card what they have learned at the end of
each class. They are then instructed to write down on the same card when
they apply the skill in their jobs. The cards are then brought back to the
instructor who collects them as part of project evaluation. This approach
encourages workers to think about application which encourages transfer; it
also demonstrates that the classroom instruction is being applied in the
workplace. (Mary Gershwin has presented on fourth level this approach at several
workplace literacy conferences.) The fourth level is impact on the
organization—bottom-line results. The company needs to identify what their
concern is--why they want a workplace literacy program. That's the clue as
to what they should measure. Note: They have to gather this type of data.
Providers simply don't have access to this type of info. Best to use data
that they already collect--you are more likely to get access to that rather
than asking that they collect something new.
3. What range of evaluative methods are currently being used for
workplace programs across the country?
Response (Dr. Askov):
I really don’t know. I do know that there is a tremendous range of approaches being used in the NWLP projects. Almost any text on evaluation will outline methods ranging highly prescribed to participatory evaluations. The approach you use may depend on your philosophy. I lean toward the eclectic with a participatory bent!
4. With a variety of businesses interested in workplace literacy/training programs is there general baseline data that should be collected and monitored for literacy program evaluation?
Response (Dr. Askov)
I find Kirkpatrick’s levels of evaluation very useful (See above). You, of course, must collect information to demonstrate basic skills gains. What is transferred to the workplace and bottom-line measures will depend on the needs of the specific workplace. Find out where they “hurt” and measure that.
5. What types of pre- and post-testing are businesses interested in giving as part of a program (i.e., competency-based assessments, standardized tests, other)?
Response (Dr. Askov)
Businesses don’t usually know how to assess workplace literacy skills pre- and post- . they rely on educational providers to do that. My preference is for competency-based assessments that measure basic skills in the specific workplace context. That usually means that you have to get into test development. Using standardized tests pre- and post- usually doesn’t work since they are not sensitive to the curriculum which you are delivering. You want to be able to show mastery and gains in basic skills. [You might be interested in our discussions of assessment issues in our most recent publication from the Institute for the Study of Adult Literacy, which gets intot he pros and cons of various types of assessments. Ordering information is available from Kitty Long at kpl1@psu.edu.
6. What does a workplace literacy program evaluation involve?
Response (Dr. Askov)
See above for my approach.
7. What types of data should be collected for literacy programs delivered in: manufacturing, service, and government settings?
Response (Dr. Askov)
I assume you mean bottom-line data. Again, finding out what the company wants in almost any setting (whether it’s the number of widgets, hamburgers, or customer calls received) is a major issue along with product quality. Safety is particularly important in manufacturing since it affects bottom-line results as well as their workers’ well-being.
8. What procedures would you use to analyze the data?
Response (Dr. Askov)
I'm not sure there is a generic answer to this question. If you have the capability to perform statistical tests to measure pre- and post- test gains with T-tests, good. If not, perhaps you can work with a graduate student from a university to get help with statistics. However, the assumptions behind most statistical tests are difficult to meet in a workplace program. The optimum is having a randomly selected control group or comparison group which receives no training/treatment or the usual training. If you can set up that situation, then measure both groups pre- and post- with tests that will reflect mastery of the training.
On long-term authentic workplace evaluation
1. I work with the Workplace Learning Resource Center, part of San Diego
Community College District. We contract with companies to provide basic
skills instruction in the workplace. I would like to find hard data re: how basic skills improvement affects the bottom line, to assist us in marketing these classes to industry. We do a good job of assessing learner progress, collecting anecdotal evidence of
positive changes in the workplace, and demonstrating immediate gains. We
haven't been as successful in documenting more long term, concrete changes,
things like "fewer defective parts" or "less down-time of machinery" or
"more effective participation in team meetings", mostly because we aren't
there any more after the course is over to collect the data, follow up, etc.
We are focused on the next project, and the last one gets lost, and the
company as well has moved on to the next thing. Suggestions? Experience? Assessment tools? Examples of data?
Response (Dr. Askov)
Marian raises an excellent issue that all of us who operate on "soft" money
face. How can you take time to do an evaluation which is not funded as
part of a grant? On the other hand, how can you NOT take time to do an
evaluation which is going to provide evidence of effectiveness for future
contracts? She is absolutely right in pointing out that the best way to
"market" services is to demonstrate a past record of accomplishments.
Tom Sticht pointed out in a paper on workplace literacy evaluation written
for the US Department of Education in the late 80's that educational
programs should not be accountable for producing "bottom-line" gains. He
stated that the purpose of workplace literacy programs is to improve basic
skills. If the program leads to better functioning in the workplace, and
to bottom-line improvements, then great! (We need to remember that
perspective! We can't be remedies for all bottom-line difficulties!)
However, companies that are engaging providers may be looking for a more
direct payoff. My experience is that companies enter into a contract with
an educational provider for the purpose of improving their so-called
bottom-line. However, once they see the effectiveness of the program
demonstrated through their workers (through improved morale, self-esteem,
attitudes, work behaviors, etc.), they often stop asking for bottom line
proof. They are more likely to point to changes in individual workers as
evidence of program effectiveness. However, top managers on up the line,
usually at other locations, do question the cost effectiveness of workplace
literacy programs so the question remains a good one.
The only way that an educational provider can gather bottom line evaluation
data on effectiveness is with the cooperation of the company. We do not
have access to their records! Somehow, you need to get company buy-in on
data collection so that you can show impact. In fact, they need to do the
data collection. (Find out where they "hurt"--where their bottom line
problems are. Make that the focus of the evaluation.) Because it is
difficult to get companies to gather data for you (they may be "too busy
with production" or unwilling to share confidential info.), most providers
rely on testimonials of CEOs or training directors. This approach is often
effective if you have a convert who is willing to speak out at business
roundtables, etc. What are the rest of you doing? The NWLP projects offer
experience in funded evaluation efforts. (I'll describe my approach in
another message. This is long enough!) Please share ideas. Thanks.
On legal aspects of workplace evaluation
1. What legal issues have been encountered when evaluating job-literacy
requirements? FYI- we evaluated a particular category of workers for reading skills
- and those who passed at a certain level demanded (and won) a higher
grade of pay - they were not HIRED to read at a specific level. (We're union) .Can a person be fired due to lack of literacy skills? In a complex manufacturing setting - what do you feel realistic base reading grade levels should be? Can you recommend any studies that show concrete proof of higher reading levels equate a better (more productive) employee?
Response (Dr. Eunice Askov)
Peg's questions take us in a slightly different direction from program
evaluation. My suggestion is that you hunt up this reference for guidance
on legal issues: Arthur, Diane (1994). "An Employer's Guide to Policies
and Practices: Workplace Testing". New York: Amacom. It is available from
the American Management Assn., 135 West 50th Street, New York, NY 10020.
Try your library first--it's not cheap! As to your question about reading
grade levels for advanced manufacturing jobs, I avoid using grade levels
since they are meaningless for adults, especially in a workplace where
content knowledge and experiential background are so important. Any answer
I would give would really not be accurate or meaningful. I also can't
provide "concrete proof" that higher reading levels translate into better
worker productivity. It makes sense that higher reading capabilities would
enable a worker to deal with the literacy demands of the job, but there are
many other factors as well, such as attitudes, work habits, technical
knowledge and experience, etc. We do know that higher education levels
translate into better salaries, but I'm not sure that is what you are
asking. Any thoughts from others? Thanks.
On workplace education material evaluation
1. Do you recommend evaluating the reading ease or difficulty of on-the-job
printed materials in every workplace? Telling people, "You'll have to learn
to read well enough to understand your on-the-job materials" doesn't seem to
be the only answer. I think we need to look at the difficulty of the
materials and help the writers learn some new skills, too. Several HR
materials I've analyzed were written at graduate school level, poorly
organized, and badly in need of some graphic design and layout changes. How
do you recommend evaluating the materials and their characteristics? Then
what do you recommend be done about making them more reader-friendly?
Response(Dr. Eunice Askov) on evaluation of materials
Here are my thoughts to Audrey's concerns. She is absolutely right about
workplace materials. They are often obscurely written by engineers (or
someone far away from the frontline workers). You might do as she
suggests--apply a readability formula (Fry Graph or whatever) to a sample
of materials just to show the company that the workers will have difficulty
reading the required materials. (You need to remember, however, that
workers who are familiar with the company can read materials at levels
higher than predicted by a readability formula because they are familiar
with the content.) Often workplace literacy programs become involved in
writing a "translation" for technical manuals as part of the contracted
training services. The key to success in rewriting materials is working
closely with the training director so that your version of the materials is
not only readable but accurate! Don't forget to pilot test the rewritten
materials on new and experienced workers to be sure that they do work!
Because we lack the technical background, we may omit or distort important
material that workers need to know. We also need to identify the essential
workplace vocabulary (including abbreviations) and teach those words as
part of the rewritten materials rather than avoid terms just because they
may elevate a readability formula. So, while it is useful to use a
readability formula to demonstrate need for revised materials and training
for a company, don't rely on a formula too heavily for your own rewriting
of materials. You may also be able to identify some audiovisual training
materials that could replace some of the company's written training
materials. The point to keep in mind is this: Workers really need to be
able to interact with training materials in some way, such as by
computer-assisted instruction, role play, workbook activities, etc. Simply
reading materials (whether in their original or rewritten forms) is usually
not effective for learning. Audrey, please share your experiences. Others
also. Thanks.
On participatory workplace evaluation
1. I'm hoping it's not too late to add a question for Dr. Askov...
If not, it would be great to hear her thoughts about what *participatory*
evaluation has looked like or could look like in workplace literacy
evaluation. Are there any classic examples in circulation or in process?
Response ( Don Cichon)
I'm just winding up a participatory evaluation of a 3-year workplace ed
program across 7 companies in NY State. We'll have draft reports ready
in early September. If you send me your mailing address, I'd be glad to
get you copies, then "talk” if you’re interested.
Response (Paul Jurmo)
In 1993 and 1994, the National Institute for Literacy funded a one-year
project in which a team of researchers developed a "participatory"
(a.k.a. "collaborative" or "team-based") approach to workplace education
evaluation. The results and methodology were summarized in a series of
case studies from the seven workplace education program sites which
participated in the study, along with a guidebook which outlined the
evaluation process used.
These documents (authored by Laura Sperazi and Paul Jurmo) are available
from Learning Partnerships at the address below. (A third document by Paul
Jurmo -- an overview of issues and new directions for workplace education
evaluation, based on a literature review and interviews -- is also
available.)
This collaborative approach to evaluation grew out of an earlier evaluation
project funded by the Mass. Workplace Education Initiative. This
evaluation approach was later built on in a series of guidebooks issued
by ABC CANADA (416-442-2292) which laid out principles and practices of
a collaborative approach to workplace needs assessment, evaluation, and
curriculum.
Underlying this approach to workplace education and evaluation (whether
called "participatory," "collaborative," or "team") is the notion that
key program stakeholders can build a deeper understanding of the
potential and outcomes of workplace education if they take the time to
reflect on program goals and monitor what the education effort is
achieving. By working in a planning and evaluation team, stakeholders
can also build collaborative planning mechanisms consistent with a more
participatory workplace.
This model has been tried out more recently in the 3-year federal
workplace education program in NY State which Don Cichon refers to
below. This and other experience using this model (in the US and
Canada) show that, although a collaborative approach to workplace
education has significant potential, it also requires time, common
values, and expertise, assets which busy stakeholders often find hard to
come by.
Response (Dr. Eunice Askov)
Paul, thanks for chiming in. I was going to reference your materials--just
had to get the correct citations with me at the computer! You did a great
job of describing the participatory approach. We have followed a similar
approach--in which the stakeholder group really takes charge of the
evaluation--in a manual created as part of the National Workforce
Assistance Collaborative (NWAC) funded by the US Department of Labor to the
National Alliance of Business (NAB). [Askov, Eunice N., Hoops, John, &
Alamprese, Judith (1997). "Assessing the Value of Workforce Training".
Washington, DC: National Alliance of Business.] You can order the guide
for about $7 from NAB at 1201 New York Ave, NW, Suite 700, Washington, DC
20005 or call 1-800-787-2848. You will also be able to download it from
NWAC's web site (http://www.ed.psu.edu/nwac/index.html). Don't look there
yet for it, but you might find other helpful reports for workforce and
workplace training and literacy which are already in place. Thanks.
On outside evaluator qualifications for workplace evaluation
1. This is a slightly different type of question concerning evaluation. If
you want to hire an external evaluator to assist in evaluating a workplace
literacy program, how would you identify a qualified evaluator (i.e., what
qualifications should the individual possess) and how do you determine a
fair amount to pay the evaluator (and/or what should the evaluator be
expected to do as part of the evaluation)?
Response (Dr. Eunice Askov)
This is a good question. Of course, you want to consider a person's formal
training and experience. Experience in and knowledge of the literacy field
are essential. (Some evaluators say that this is not necessary, but I
don't agree. An evaluator needs a sense of the adult literacy delivery
system, how instruction is carried out, the nature of the students, etc.)
It is important to interview your potential evaluator to determine his/her
philosophy toward evaluation. My recommendation is to choose someone who
believes in a participatory approach so that all the stakeholders are
involved. It is also important to determine the evaluator's philosophy
toward assessment. For example, sometimes an evaluator believes in using
only standardized tests. The program decision-makers have to decide
whether or not they agree with that point of view. Then, you also have to
reach agreement on the extent of involvement in the program. The evaluator
should be involved upfront and throughout the project in both a formative
capacity as well as a summative evaluation capacity. You need to decide
what level of depth is necessary for your purposes. Finally, you need to
decide why you need an external evaluator--why you can't do it for
yourself. If an outside point of view is essential, then go ahead. If you
think you can perform the evaluation function, then check out our book from
the National Alliance of Business as part of the National Workforce
Assistance Collaborative ("Assessing the Value of Training") which tells
you how to do your own evaluation in a workplace setting. Good luck.
On references about workplace literacy program evaluation.
1. What publications, journals and books are foundational to understanding workplace literacy program analysis and evaluation?
Response (Dr. Eunice Askov)
I promised you the reference that I use in my evaluation plan as an
external evaluator for NWLP projects. It is Kirkpatrick, Donald L. (1994),
"Evaluating Training Programs: the Four Levels". San Francisco:
Berrett-Koehler Publishers. (Address is 155 Montgomery St., San Francisco,
CA 94104-4109. Telephone: 415-288-0260; fax: 415-362-2512) I like
Kirkpatrick's four levels because they make sense to people. While the
levels are not strictly a hierarchy, they do represent a natural
progression in evaluation.
To continue the list serv chat on program evaluation, it is now popular to
try to figure the ROI or Return on Investment. Jack Phillips wrote a
series of articles on the subject in "Training" last spring, complete with
formulas for figuring impact. Ed Gordon is also working on ROI formulas to
be published shortly. It appears that this publication (see below) may be
similar. If you are going that route, you may also want to take a look at
Mikulecky, Larry, & Lloyd, Paul (1992), "Evaluating the Impact of Workplace
Literacy Programs" available from the National Center on Adult Literacy,
University of Pennsylvania, Philadelphia, PA.
NWAC's web site (http://www.ed.psu.edu/nwac/index.html). Don't look there
yet for it, but you might find other helpful reports for workforce and
workplace training and literacy which are already in place. Thanks.
Response on the value of the Guest Speaker Forum
(Carolyn Talarr) To everyone who responded to my question, Thanks so much for the wealth of references and leads on participatory workplace literacy evaluation. I'm going to take you up on your offers and *will indeed* be contacting those of you who indicated that you'd be willing to share materials. And generally I'd like to thank Barb Van Horn and Dr. Askov for this week of rich, focused information sharing. Makes me want to suggest similar events on other lists...
(Barbara Van Horn) Carolyn Talarr said that she wanted to suggest similar events (discussion with guest speaker) on other lists. I would like to encourage subscribers
who enjoyed this week's exchange to send me suggestions for other topics of
interest and/or suggestions for guest speakers. I would like to do this
more often!
(Josephine Kramer) Yes, Nickie and Barb, thanks for the week's exchange . It was very
enlightening. Stimulated a lot of good thoughts and interaction.
How about Larry Mickuleky sometime? Or Mary Gershwinn (Colorado)?
(Dr. Eunice Askov) Thanks for inviting me to be the guest resource on the NIFL-Workplace list serv last week. I enjoyed thinking about and responding to the challenging questions. Your suggestion of Mary Gershwin who is a practitioner on the
front line is a good one. The Community College Board of Colorado NWLP
project (which she directs) has done some really interesting and innovative
things in their project. Enjoy!