I would like to welcome our guests Jennifer Rafferty and Shannon Young. They have both posed some interesting questions as catalysts for discussion, and I encourage subscribers to take a look at these, ask your own questions, or respond to any ideas that strike you.
Let's hear from programs and states that are now engaged in offering distance learning opportunities: what are your successes and challenges? Do Shannon's and Jennifer's questions and points feel familiar to your experience? Do you have strategies for tackling
assessment and accountability mandates? What are they?
Did you read through the resource "Pre-Assessments for Online Learning"? What did you think of that discussion?
For full information on this discussion including the announcement, guest profiles, and suggested preparations, go to: http://lincs.ed.gov/lincs/discussions/assessment/07assess_distance.html#announce.
Thanks and looking forward to hearing from you!
The following is posted on behalf of a subscriber:
Do states that fund and support distance learning have a coordinating entity to provide leadership, oversight, and accountability? How do DL programs within a particular state get evaluated in terms of their effectiveness?
I can speak only from my experiences in New York State, which has a large number of students in Distance Learning programs. The State Education Department, which sets policies on management and assessment in these programs, have started conducting analyses on the effectiveness of distance learning programs vis-à-vis regular classroom programs (not that much difference in effectiveness) and programs themselves are encouraged to look at the NRS reports of distance learning programs separately. Distance learning programs are evaluated in NY the same way as other services are, using NRS guidelines. Starting in FY2008 (December 2009) the Federal Department of Education is going to ask all states provide disaggregated reports on students in distance learning programs, so the focus on these services is intensifying.
Director of ALIES/Data Analysis
Literacy Assistance Center
What are 'disaggregated reports'?
Sorry, by that I meant reports prepared not for the whole program or agency but just for the distance learning program. This allows a program to evaluate the performance of a specific distance learning product or all distance learning offerings.
It means separated out into various groupings.
Mary Jane Jerde
Thank you for you posting, Venu. You mention two distinct groups below that can provide us with data to analyze the effectiveness of distance learning, the classroom programs and distance learning programs. I am wondering if New York allows learners to co-enroll in both a classroom and a distance learning program?
Distance Learning Project Manager
Adult Literacy Resource Institute
Piggybacking on Jennifer's question, if you have any co-enrolled students, have you developed a policy for how to categorize these students in terms of NRS reporting?
Shannon J. Young
Program Manager, Project IDEAL / AdultEd Online
Senior Research Area Specialist, Program on Teaching, Learning, & Technology
Institute for Social Research, The University of Michigan
Since my experience comes from working on pilot project in Ontario, Canada, I can't comment on how individual States in the U.S. accomplish this but I can share some things from North of the border with you and I hope they are to some value to this discussion.
During the pilot phase of the project from 2003 to 2006, AlphaPlus Centre [http://www.alphaplus.ca] was contracted to provide the coordination of communications, research, professional development and to the extent of the research data gathered also accountability. Four literacy programs were contracted by the government directly to explore, implement and research distance, online, blended and/or flexible learning models in partnership with AlphaPlus. I was working as the project coordinator of this project at AlphaPlus Centre. In case you haven't seen the link posted on the introduction page to this discussion, the research report and an executive summary can be accessed online in pdf format. [http://distance.alphaplus.ca]
Since the pilot project phases finished the Ontario government has been working on a accountability framework that would make the evaluation of the effectiveness of the programs and reporting to the government as the funder of ABE in Ontario more effective.
Hi NIFL List:
In response to this question on evaluation, here in Arizona the state hired Sheryl Hart, Matthew Piech, and one other representative to handle the distance learning needs. Then the state offered additional funding to seven programs to run pilot sites for a three year period. These seven programs were chosen from a pool of current GED providers statewide. As of now we have completed our three year pilot and are going to scale, which means that all GED programs in Arizona must have a distance learning component whether they operate their own or partner with another provider.
Coordinator, Developmental Education
Verde Valley Campus
Thank you for your post, Tina! Great to see you here. My question to you is not related to assessment, but it is a follow-up question on what you have described below for going to scale. If all programs in AZ are required to offer a distance learning option, how is this funded? Did programs have to choose to offer a dl program service option and give up a f-2-f class in order to fund the component, or is there funding available for the required distance learning component?
It's very quiet out there! I've had many new subscribers join our List, so I know this is of interest to you all. I know you have questions and I know you have experiences to share. Please post them now!
Here is one of my questions for this discussion, which actually has already been asked within the suggested preparations by our guests Shannon and Jennifer:
Post-testing: How is this done? How do you get that important follow-up measure from students? I have experience in DL within professional development only (not with ABE/ESOL students) - and I know how difficult follow-up is with the professional population. The same goes for regular classroom-based programs - getting full test scores can be really hard. So I am extremely interested in hearing from folks about the strategies and methods they use to get DL students to complete the full cycle including the post-test.
Do federal requirements allow for any flexibility/creativity in this regard? Can you use formative assessment and other types of measures to help demonstrate achievement?
Do Project IDEAL states do their assessment in the same way, or are you able to alter your approach depending on certain things (levels of funding; timing; some F2F ("face-to-face")/no F2F; staffing; etc)?
Let me raise another question. I am new to this list so it may be something that many of you have already discussed.
I work with a national labor/management partnership that supports training and education for the incumbent healthcare workforce. We are supporting several distance learning and hybrid nursing programs. Because these programs are asynchronous for the most part and can fit into many workers busy schedules, there is a great deal of interest in them. However, because we realize that distance learning may not be the best modality for everyone, we want to be able to be able to screen for students who are most likely to be successful in this kind of program and also pinpoint other students who may be successful with some extra help up front.
I'd love to hear others experiences with this type of screening. Do you find it necessary? Is it helpful? What have you learned? I'm looking forward to hearing from you.
H-CAP National Coordinator
Hi Laura, Welcome and thanks for your question - which is an excellent one! I'm also looking forward to hearing from subscribers on this question.
Some discussion on this can be found in one of the suggested resources, entitled "Pre-assessments for on-line learning". Go to:
The information in the discussion opens with a focus on "wait periods" or "tryout periods" so that prospective students can self-determine their ability and commitment, but it winds around to discuss assessing skills that people need to engage in technology and distance learning.
I'd be very much interested in what List Subscribers think about that discussion and Laura's questions below.
Assessment Discussion List Moderator
I think the idea of a tryout period that Marie mentioned is a good one. We also recommend careful pre-screening. During the first round of Project IDEAL experiments, several states opened their distance programs to any and all who expressed interest. This approach proved problematic in many cases - students whose skills, abilities, and/or dispositions didn't match well with distance study quickly became frustrated. As a result, drop-out rates were high. Programs discovered that they needed to carefully pre-screen students in several arenas to ensure a good fit. Many now begin by surveying students on their learning styles and preferences, time management skills, and motivation, using their responses as a jumping off point to talk about what distance study entails. Once some students understand what's involved in being a distance learner, they realize a classroom program makes more sense for them. Prescreening also includes assessing students' technology skills and access as well as their basic literacy skills. This combination of pre-assessments helps teachers better determine (1) which students are potentially good candidates for distance and (2) what, if any, baseline training is needed.
These days, there are a variety of "is distance learning for me?" self-assessments available online for students. I've included a sample of some I've come across to give you an idea of what types of topics/issues distance programs are addressing.
Guilford Technical Community College
--An original and entertaining approach to helping students determine whether distance learning makes sense for them.
Student Online Readiness Tool (SORT) http://www.alt.usg.edu/sort/
--Six section self-assessment created by the University System of Georgia designed to help students determine whether they should study online. The six areas are: Technology experience, access to tools, study habits, lifestyle, goals and purposes, and learning preferences.
Online Readiness Quiz (http://www.pima.edu/cgi-bin/onlineReadiness/quiz.pl)
--This 41-item quiz from Pima Community College assesses potential distance
students in four areas: computer and technology skills, time management, learning environment, and study and reading skills.
Minnesota Virtual University: Are Distance Learning Courses for You?
--Provides a computer scored self-assessment for distance learners.
Hi Shannon and everyone,
Wow, thanks for all these great resources! I see that you advocate a variety of pre-assessments in order to best determine a person's readiness, which makes a lot of sense to me.
It made me think of something else: what is there in the way of curriculum and materials used in DL? Do folks focus on 'commercial packages' or materials generated by the feds or states, or do programs develop their own curriculum and materials? I ask this because if there are particular requirements around assessing and reporting, then each of these pieces would be affected by the other. And I would assume that some things work better together than others.
How does all that work in programs?
Assessment Discussion List Moderator
Different states have purchased different software packages for their dl classes. Arizona purchased MHC Online. Later we added a Pre-GED LAN only software program to help with students who were not ready for the MHC Online materials. This summer Arizona purchased a newer version of Pre-GED that is web-based similar to MHC Online.
Each program in our state has the ability to choose supplemental materials. Our college also has Skills Tutor, so we were blessed to have both of these electronic materials for our students. In addition, we had a few older McGraw-Hill GED and Pre-GED textbooks that we could loan to students. I encouraged students to buy a Complete Pre-GED book to have at home, but certainly did not require it.
In IDEAL's course DL 102, the online instructors networked together to locate videos, Web sites, materials to scan and send to students, and generally worked creatively to find ways to meet curricular needs to explain challenging concepts at a distance. Much has been done, but we are still in the early stages of online education.
I'm just getting caught up with the conversation and haven't read all the threads yet. I wish it wasn't the last day for the discussion. There is so much to talk about with regards to dl.
I work with dl at my school and we use MHC online, Skills Tutor and another program called A+. I like all of them and all of them have their strengths and weaknesses.
I also make supplemental materials available to my students - anything from workbooks to worksheets to face to face tutoring if they really get stuck. We communicate a lot by email and by phone. Like Tina, we're still in the early stages of our dl process.
In addition to the pre-assessment resources Shannon has shared, I would also like to share with you the URL for the Massachusetts ABE Distance Learning Project. www.anywhereanytimeabe.org . Over the past two years, we have developed the Orientation Modules section. It is actually being revised and updated as we engage in this discussion, so I encourage you to visit the site again in the next month because you will find that the resources have additional notes to the instructor as well as more interactive features and revised handouts.
The Orientation Modules were created thinking of both the distance learner and the classroom learner. One of our visions for this section of our website was to see classroom programs incorporate content from the Orientation Modules into their curriculum so that classroom learners would be prepared for distance learning should they ever need to "stop out" of the classroom. In fact, the content developers, who are both classroom and dl instructors, use the materials with both groups of learners.
We created the modules thinking they could be used according to the needs of the learner. Our dl pilot programs are not required to complete the modules with distance learners during the orientation process, but they are encouraged to use the resources and adapt them. We generated all of the topics through discussions and focus groups with our dl pilot programs. The topics include: setting goals, barriers and supports, independent learning, technical skills, time management and study skills, and communication skills.
As for your question about distance learning curricula, Massachusetts has implemented a variety of commercially produced curricula over the past eight years. During the pilot phase from 2004-07, programs were funded to pilot commercially produced curricula for which the state had purchased licenses. They were not funded to create curricula, although there is one ESOL dl pilot that had developed a homegrown curriculum prior to being funded. Of course, teachers supplement these curricular packages with their own materials to fill in gaps and to provide additional practice. We call these other materials "supplemental", based on Project IDEAL's system for collecting proxy seat time for distance learners. There are currently three models I am aware of that have been created to approximate seat time for distance learners: the teacher judgment model, the student mastery model, and the clock time model. So, depending on the curriculum/a and the kind of delivery (e.g. multimedia, computer-based) a state chooses, this will dictate the way the proxy seat time is collected. Project IDEAL has a publication on this topic of seat time, it is Paper 2#, Measuring Contact Hours and Educational Progress in Distance Education, and can be accessed at their website at www.projectideal.org under the Publications section.
One of the big questions we asked ourselves while piloting the commercially produced curricula was, do these curricula align with our state Curriculum Frameworks and our standardized assessment for ABE learners (the MAPT)? This past fiscal year, we examined the frameworks and the ABE curricular package that is currently being used in MA with dl pilots, so we have a better sense of where there will be a need to supplement our ABE dl curriculum.
Hi Marie and everyone.
In Prince William County, VA, we are using Project Connect for ESOL. This is our second session using it. If you are not familiar with the package, you can learn more about at http://pbslearnenglish.org/learn/page1.xml As an instructor, I find Project Connect easy to use, and in general, I think the students feel it is pretty simple. However, students still need a live help. Another experienced ESOL teacher (whom I think is on this list) holds live sessions on Friday nights. During the live sessions, I am logged into
Project Connect from home so students can get instant responses to email and feedback on their work units.
We provide students a handbook that illustrates Project Connect's key functions as well as helpful tips for students who might not be accustomed to online learning. In addition to this and the Project Connect online orientation, we will be providing live, monthly orientations in which we go through the handbook and program and catch up on any questions students might have about the technology. This helps me get some face-to-face time with the students as well. Last term, I worked with students online and spoke to them on the phone. However, since they only had live contact with the Friday night teacher, some students had difficulty with the concept I was a real person. This is common in most DL classes, so hopefully, the orientations will help eradicate the myth that I am a computer : ) And of course, they will continue to benefit from the blended method already in place.
Katherine Mercurio Gotthardt, ESOL DL Instructor
Prince William County Public Schools
I've attached the new NRS Guidelines for Distance Learning that were sent out to State Directors on June 6, 2007 with a memo from Cheryl Keenan. The distance education policy includes a definition of distance learners, guidance on how to measure contact hours for the learners, and describes assessment and reporting requirements which are summarized in the memo. It will be an interesting discussion around how states implement these policies, and you have posed some great questions below.
Massachusetts NRS Trainer
You will notice that the attachments Donna mentions were not attached! (Thanks Donna!!) We are trying to avoid sending attachments via the emails because some people's systems are not prepared to handle this type of thing.
I am working on having the documents Donna talks about posted at the same site where the rest of the information for this discussion is. I'll let you know as soon as they are up there. In the meantime, perhaps some of you are already aware of the new guidelines or have access to them.
Assessment Discussion List Moderator
Donna, Marie and Everyone,
You can get these documents on the NRS web site. http://www.nrsweb.org.
Look under What's New.
NRS Project Director
Here's a follow-up to the discussion on distance learning assessment. I have a question about the changes to the NRS guidelines for distance education learners. (http://www.nrsweb.org. Look under What's New.)
Item 3: "For NRS reporting, states can count a student only once, as either a distance education student or traditional classroom learner."
Presumably that means the student can be counted only once annually, and that the program needs to decide if the student is _primarily_ a distance learning student or _primarily_ a face-to-face (classroom or tutorial) student. Presumably this gives programs and students the freedom to offer distance learning options and face-to-face options to the same student within the year, that a face-to-face student could also have supplemental online learning. Presumably, the intent of this is to avoid double-counting the same person, not to restrict how students access learning (whether distance learning or face-to- face or both).
Donna Cornellier, or Larry Condelli, are my presumptions correct?
David J. Rosen
Yes, you are correct in your interpretation. The only intent for NRS purposes is to avoid double counting, while at the same time giving states as much latitude as possible to provide both distance and other instruction. States need only explain (in their distance education policy) how they make the determination for counting purposes.
Ok, how does this work if say for instance, a student is enrolled in a physical class that runs from 8-12 and then is enrolled in a distance learning class for time outside the 8-12 boundary or for day he/she is absent. They are only counted once even though that is two separate classes, rosters and contracts?
The primary counting issue we've seen so far has to do with hybrid or blended learning students. In some instances, students study in a classroom program for some months and then switch to distance learning (e.g., during winter months). Does an agency assign status based on which program (classroom or distance) the student entered first? on which approach accumulated more hours?, etc. In other instances, the student may
participate in a combination of distance and classroom study concurrently. At what point do you determine whether a student should be considered a distance or classroom learner? And, what are the benefits/drawbacks of counting them one way or the other?
Traditionally, counting "seat time" hours has been used to determine when to post-test a student. Now that both distance and classroom hours can be accumulated, what impact, if any, should this have on post-testing timeframes?
We will be working with Project IDEAL states this August on developing their policies and plans for counting distance learners using the new NRS guidelines. We will also have a paper (due out this fall) that addresses some of these questions. What other questions are you exploring as you review the new NRS guidelines and think about the ramifications for data collection and reporting?
Where exactly would the double counting happen? If I understand the new reporting policy correctly, states are being required to submit two new tables, 4c and 5a, just for distance learning students but to include all students in the old NRS tables 1 - 6. This tells me that 4c is a subset of table 4 and 5a is a subset of table 5. On which table would
we make the mistake of counting a student who is enrolled in both distance learning and traditional classroom instruction twice? We already make sure we don't double count students enrolled in more than one class. Is this not similar?
Director of ALIES/Data Analysis
Literacy Assistance Center
I believe so. A student may only be counted one time even though he/she is enrolled in separate distance and classroom-based classes. Larry, would you confirm that this is the case?
Hi Shannon and Katrina,
NRS requires a non-duplicated count of students. So the state has to have a policy on how to classify them (DL or not) for reporting. It's just the same as if a student took two separate classes at the same time -- they are only counted once in NRS.
Of course you may have other procedures for local contracts, management, scheduling, etc., according to state/local policy. Again, it's the same as if a student took multiple classes at the same time.
Good morning, afternoon, and evening to you all.
Today is our final day of conversation with Jennifer and Shannon on assessment in distance learning. I encourage you to post your questions and experiences now. I'm looking forward to reading the replies from yesterday's posts.
Here's my next thought on this, I'm back to Post-Testing: does DL post-testing ever involve groups of learners together? Or is it usually a unilateral event for that one student? I'm asking because I know this is a tough area, and I was wondering if some amount of work with peers might alter this. What are people's thoughts and experiences on this?
I don't know if I understood you correctly, but logistically it is not always possible to get the dl learners together to be post-tested. In Massachusetts, our dl programs run with an open enrollment schedule, so students start at different times and they accumulate "seat time" hours at different rates depending on the intensity of their studying. As is with the orientation process, our programs have to be prepared to provide support individually if the timing for orientation and pre/post-testing doesn't fit the learners' schedules. I have seen some dl students go to classroom-based programs when post-testing is being conducted in a group. This seems to be a good logistical set-up, to look at the scheduling of post-testing for classroom programs in the agency, and send the dl learner to be post-tested there if it fits their schedule and scenario. When I was teaching in a dl ESOL program back in 2003, we were using the BEST to assess learners and it was challenging to coordinate because each learner needed to be assessed individually and the test needed to be administrated by another instructor. We ended up doing an exchange where I would assess classroom learners and a classroom teacher came to my program to assess my distance learners.
Hi Jennifer - thanks so much for this. Yes, I didn't make myself that clear in my post, but your reply definitely speaks to some of the questions that I do have.
I was indeed asking about what you discuss below: I am interested in how post-testing might get accomplished when you can be in f2f situations, and whether or not being able to group students together would make any difference at all.
But I was also wondering (and I'm showing the true colors of my lack of knowledge around the details of DL - but that's ok because I am here to learn) these things:
-I assume that some DL is purely DL - no f2f (is that right?) -If so, how do you use required tests that do not lend themselves to on-line delivery? -And if post-assessment *can* be accomplished purely on-line, are there ways to make that post-test more of a group task, which might lead to more people completing that piece of the puzzle? (this group task idea is obviously an assumption on my part)
Ok, thanks and looking forward to hearing what people's experiences are with this.
Assessment Discussion List Moderator
You've raised good questions many programs have to address. Most Project IDEAL states have policies requiring their students to be post-tested in a proctored f2f setting, regardless of whether they're pure distance students. I think this usually applies only to standardized testing, though I know states like North Carolina require students to come in for progress testing after each unit of study. NC's ESOL distance students receive "seat time" credit only after demonstrating mastery of the materials. I know Missouri at one point experimented with offering the TABE online, but I believe students were still required to complete the test in a proctored setting. Other curriculum-specific quizzes and tests are usually completed without a f2f component.
Some programs and states (particularly ones that have large, rural populations) have set up partnerships with local libraries, schools, and other social service agencies to handle post-testing. That way, while students may have to drive some distance to participate in orientation, they can then go to a place that's closer to home to post-test.
Last year, we surveyed Project IDEAL teachers to find out how they got students to return for post testing. One frequent answer: throw a party. When students knew they'd be seeing and socializing with other students from orientation (as Tina mentioned), they were more likely to come back in for testing.
Does any state have a policy allowing post-testing to be conducted online and without a proctor? If so, how do you ensure validity?
DL isn't always purely Distant. Students get stuck with a problem, or have an issue and he or she may not understand an area and may need some momentary face to face interaction with a real person. My DL students know that I teach from 8-3 but that any time after 3 they can come in and meet with me. I'll even give up my lunch time if need be to meet with my students. Additionally, I will give my DL students my IM information and they know they can find me online if they want "real" time interaction outside of my normal workday hours.
The NRS requirments for DL were just recently posted to the list. One of the requirements mentioned in the document is that pre and post testing has to be done face to face. We already require our students to do that and all GED practice tests have to be done online. Additionally, one thing we've found is that it's helpful to have some kind of face to face interaction with the students. It's not as impersonal as it could be if the DL student didn't see a warm face and know they could really get the help they needed if they have questions or problems.
We normally do posttesting in a group setting to save instructor time and energy. One teacher has offered posttesting done at the same time she is giving pretesting when she does orientations. She will send out an invitation to her online students to come in at a certain time, and she has had some success with that setup. She does these on Saturdays. She also has her students make arrangements with the campus instructor to do posttesting for her.
I ask dl students to come to campus during the face-to-face (f2f) week of posttesting and give them a list of locations and times for them to give flexibility. I do posttesting one week per month. By the end of the semester, many students have come to at least one of these sessions. I also try to get the student to stay for the entire three hours class period to encourage some personal interaction with the staff and other students as well as to ask questions in person.
Because these students enter our program in group orienatations with the f2f students, the followup posttesting in group settings with other students is a natural process and seems to be a welcoming system. We also do go out of our way to accommodate students with individual posttesting. Some students are actually nervous about meeting in groups, and others have scheduling or transportation issues that we are willing to work around within reason.
We do pre and post testing face to face with DL students as well. They test in a group environment at least for the Pre Test. The post test are usually scheduled with me after so many hours of instruction and progress as we work towards their GED test. If they're not post tested before the end of a session, we'll bring them in as fits their schedule to get their post testing done. Flexibility has definitely been a key component of the DL process. One of the things I'm learning from the DL students is not that they don't want face to face interaction but don't have the time or resources to come to class as often as they would like to and DL provides them an avenue to make progress and positive steps forward in a way that fits their life commitments.
I would like to thank our guests, Shannon Young and Jennifer Rafferty, for taking the time to share their experiences with us in last week's discussion on assessment in distance learning. I encourage subscribers to check out the resources that were shared, and if you do, let us know what you think. Thanks also to those subscribers who posted their
questions and thoughts on this topic.
For now, if you have just subscribed to the List, you can read the discussion at the archives. Go to: http://lincs.ed.gov/mailman/listinfo/assessment and click on Read
Current Posted Messages dated from last week. I will also prepare the discussion in user-friendly format and post it at the archives; I'll send email when it's ready - probably in about a week.
If people still have thoughts on this topic, please feel free to carry on with the discussion.
Even though this discussion is officially over, we can still talk about the subject I hope.
The Healthcare Career Advancement Program is, among other things, working with several labor management partnerships on distance learning nursing programs. One of the things we are interested in is the interaction between students and instructors. In some instances we understand that there is even more interaction in DL than would be possible in some of the lecture hall classes that many students experience in general education and nursing lecture courses. We are also looking at the issue of sheltered entry to college utilizing distance learning. For example, workers who have been out of school a long time often have difficulty making the transition to college and need a lot of support as they are beginning their gen ed courses and pre-requisites for nursing. We are considering a hybrid on-ground/on-line entry to college that starts with more classroom face time (and group computer time) and ends with more individual and asynchronous on line time. We think this would be particularly helpful for students interested in on line programs who need to get up to speed with using the computer and getting into distance learning.
Do any of you have examples of these kinds of programs?
Thanks for posting. The hybrid/blended program that you describe with more face-to-face in the beginning phase of enrollment is an approach that we have used in MA with our distance learning pilot programs. While we never coined it as a model per se, we discussed what we recognized as a trend with some learners who gradually made their way into the independent learning mode. I always imagine a spectrum when I see distance learners. At one end is the group that enters and can immediately engage in distance learning because they have the independent learning skills and other skills needed to persist. At the other end of the spectrum are those who require a blended model with more face-to-face time. Some of these learners who start with the blended model eventually move on the spectrum towards the independent learning mode as they gain comfort with the new delivery. Others continue to need regular face-to-face support. In short, we have used a flexible approach that addresses the needs of the individual.
Please note: We do not control and cannot guarantee the relevance, timeliness, or accuracy of the materials provided by other agencies or organizations via links off-site, nor do we endorse other agencies or organizations, their views, products or services.