Assessment in Distance Learning - Pre-Assessments for On-Line Learning

Pre-Assessments for On-Line Learning

Excerpt from:

Discussion on Community Literacy, June 25 - 29, 2007

National Institute for Literacy Special Topics Discussion List

To view the full discussion in the archives, go to:
http://lincs.ed.gov/mailman/listinfo/SpecialTopics

Distance Learning Technology Colleagues,

Earlier this week, in response to a message Heidi Silver-Pacuilla had posted, Holly Dilatush wrote:

"too many learners register, show up one week, then drop before/without completing 12 hours/first episode. Follow-up (to attempt to determine WHY) has been challenging -- guesswork more than documented responses. There are SO many extenuating circumstances."

For many adult learners, especially those who choose online options, and for many reasons, we need to design/include/expect a "tryout period" a short online learning experience -- perhaps two weeks -- sampling the material, process and technology used in the longer, online learning. At the end of the tryout, participants can stop (if they were experimenting with the medium, are not happy with the content, can't make the longer commitment, or for any other reason). Those who are ready to commit, can do so, and at that point begin to be counted in the DOE-funded system.

Does an example of this already exist somewhere? If so, how is the tryout period funded? (State and local funders and private funders need to pick up the costs of this "tryout")

I believe we need the same sort of tryout period for teachers doing online professional development. Does a model of this exist somewhere?

Your thoughts?

David J. Rosen

Hi,

Nancy from Toronto here again in response to the issue of learner retention.
Recruitment and retention have been the focus of much discussion during the
distance delivery research we have been engaged in at the AlphaPlus Centre with
the four sites that have been exploring distance delivery.

Retention has also been a noted interest in what happens in short online courses
that we have developed and delivered as part of AlphaRoute for the past three
years. Our emerging and currently only form of new content for AlphaRoute
this year is online courses. Our courses are four weeks long. For the first two years they were delivered in a discussion forum - Web Board (supported by an external course web site) as part of AlphaRoute. This year we are offering the courses in Moodle. You can take a look at the course topic list and external course web sites at: www.resources.alpharoute.org > Resources > Online courses for adult literacy students

What we have found in terms of retention is that for the first two weeks, students are gung-ho, and then we see a drop-off in week three that is sustained through week four. Generally we have found that a third of the students that enroll in a course complete it and receive a course certificate of participation.

Interesting what you share David about a two-week period being perhaps the right
amount to try-out time.

I should note that we acknowledge there is a continuum of learning for us all, and so for a student to take the step and enroll in an AlphaRoute online course is in fact a learning step. Trying it out for a week or two to see how it works and then not continuing is also learning. Committing to take the course and complete the work to achieve the certificate at the end (and learning some cool stuff along the way) is a goal for us. And in the world of demonstrating literacy learning and a range of skills, can make for a great demonstration. But we would like to think that our courses are so valuable to learners and engaging that they will all move from start to finish. It isn't realistic that they will though - because of that continuum of learning.

From the stats and knowledge of the programs that the students enrolled in the courses come from, the highest retention rates come from students whose instructor has included the AlphaRoute online course within their instruction and where students are in a computer lab at the same time taking the course. So motivation and support (instructor and peer) are onsite. However, the course facilitator is at a distance and does contribute to a motivating and retention aspect of student support. The development of online courses in AlphaRoute has not been done within a research project or model. We are learning as we go and writing articles and sharing information as we learn. Our challenge at this point is to focus on that three week drop-off reality and see what we can do from the course content and development side to attempt to support learners in sustaining their involvement past that drop-off point.

Any suggestions or ideas from the range of instructors and researchers participating in this discussion are more than welcome!

Nancy Friday

I'm a little late getting back to this, but I love the point you made, Nancy, about the try-out being a learning experience even if the course isn't completed. I've found that to be the case with PD online as well, with teachers trying out the online experience and deciding they definitely don't like it and leaving; others finding that the material or the
participant interaction to be so valuable that they persist in the face of all kinds of technical and personal difficulties. Not too different from our students, I think. In this age of accountability we focus perhaps to much on outcomes and not enough on process and what can be learned from it, regardless of outcome.

Wendy Quinones

I agree. Some of the states in Project IDEAL have found that students who have an extensive orientation to the distance material - including plenty of time to "play" and get a feel for this instructional approach - tend to persist longer. If we want students to be successful, it seems reasonable that we help them find the instructional method or class that best fits their needs and learning styles. It's hard for them to make an informed decision about distance if they're unfamiliar with it, so the idea of try-out periods makes a great deal of sense.

Leslie Petty

I'm jumping in here mid-stream so this may have been discussed previously.

I have rarely seen in DL courses, especially for DL literacy courses, any kind of distance learning readiness assessment. You're so right when you say that "...it seems reasonable that we help them find the instructional method or class that best fits their needs and learning styles." I'm in a 100% online grad program and several of the folks have dropped out because their technical skills simply weren't adequate. I've found too many educators who have discounted the DL readiness piece and go on to blame the students for not being successful.

Melinda M. Hefner

I have seen exactly such readiness assessments. I'm pretty sure I had to take one before I took an online course here in 2000... but it wasn't a grad program. Starting recently, *all* our students have to take a "computer competency" assessment and course placement is recommended from that. (We have several levels of basic computer competency classes.)

Susan Jones

I'm so glad to hear that your students' computer literacy skills are assessed. We've found that students often think that the computer skills they have are sufficient for a DL course, when they aren't. (They may e-mail and use the internet, but not know how to save files, upload files, use discussion boards, use digital drop boxes, use virtual classrooms, etc., etc.).

You didn't mention this, but I assume that the assessments also measure self-management skills, learning styles, lifestyle considerations, hardware and software requirements, etc. Measuring students' perceptions of DL courses is also helpful since many first-time DL students think that DL courses are easier than seated or hybrid courses.

I've seen lots of DL readiness assessments, but often they only address computer skills rather than also addressing the areas I mentioned above. As DL courses become more and more common and as student learning and student satisfaction data are analyzed, I believe that the importance of comprehensive distance learning readiness will emerge.

Melinda M. Hefner

Melinda -

I want to respond to your statement:

"We've found that students often think that the computer skills they have are sufficient for a DL course, when they aren't. (They may e-mail and use the internet, but not know how to save files, upload files, use discussion boards, use digital drop boxes, use virtual classrooms, etc., etc.)."

The problem with assessing for these skills seems to me to be that they are course and interface specific. As a doc student, I have used several versions of blackboard, three university email systems and programmed web sites to accept user content. I still have to learn how to save files, upload files, use discussion boards, use digital drop boxes, use virtual classrooms, etc., etc anew with every new system I encounter. I know many ways this CAN work, but it is essential that technical support for the specific interface be included in the instruction and targeted at the learning group expected to use the interface. This is not like climbing the stairs to enter the building - its as if new ways of building stairs were being invented every day - sometimes at the same building! Interestingly enough, this is occasionally an exercise in internet history, while many commercial websites make this interface seamless and intuitive - many educational providers are either at the mercy of an antiquated system or buying an interface that is not consistently applied.

Lissa Probus

You have described precisely one of the problems my department's literacy students encounter. Can you imagine an adult student whose only computer experience has been working with educational software or web-based programs in an ABE/GED class, completing his/her GED, going on to take college level classes, and enrolling in a DL course? We offer our adult literacy students computer literacy training, and we also offer them transition to college classes, but it is beyond our scope and, quite honestly, the capability of most of our instructors (this is not meant to be critical of our instructors) to teach them about using various content delivery systems, web portals, online advising systems, etc.

Blackboard, WebCT, Campus Cruiser, homegrown content and learning delivery systems, etc. generally offer tutorial information, but all too often educational institutions merely regurgitate the tutorial information. Presenting information does not mean that learning has taken place. I've heard too many times from the IT folks, "Why don't they know how to use Blackboard? We offer workshops each semester."

Educational providers need to be knowledgeable, savvy, and willing to spend some money to get systems that allow for superior online teaching and learning, not just to act as repositories for materials, electronic portfolios, or as databases for grades. It's an upfront investment that I think has tremendous benefits for learners, educators, and institutions. Those schools that are doing it well have fabulous results with student learning.

Melinda M. Hefner

Almost all of my distance students who did not continue in the MHC program had either technical difficulties (computer virus, Internet turned off) or personal problems (got a job, kids sick, don't have time, Etc.). Even though most had good technical skills- and the program really doesn't require much- the personal discipline required was the obstacle. I'm doing the stats on the course now, and it's not looking pretty!! I will need to screen next year for reading levels. My DL grant is funded "on probation" and the stats so far have a very low success rate with lower level readers (below 8th grade). For those who can do supported distance only, I will give some Internet resources for those who are on a lower level, but they'll have to do that independently.

Maureen Hoyt

Don't forget that normal academic attrition will include things like (got a job, kids sick, don't have time, Etc.), this is neither a failure of your program or of personal discipline. It's just life. Life happens. Including basic access to tech support for virus protection could be helpful - my university offers some, but I paid for more. Internet turned off is a huge problem (and one that I often face at the end of the semester as internet access is expensive!). Consider including some support for planning the expense, back up resources - can the course be done at a local library?

If many of your students have job and kid problems, is there a way to make participation asynchronous and allow them to work over longer periods?

Personal discipline is a matter of working with individual motivation, and you can find supporting research in searches of SRL in DL (self-regulated learning). Even minimal personal feedback has been shown to improve SRL in transitional learners. There is a link between learning level and SRL, as well as an engagement issue if you are dealing with adult learners. If your adult learners are experiencing low SRL, try to tie the learning to things they need to do. If the new capacities are understood to be both valuable and achievable, and the course criteria are such that design meets audience needs, then assess attrition based on issues that would not allow an I or W at a community college. make sure not to blame your self or your students for poverty, family or the digital divide - these are the reasons for your intervention in the first place.

Lissa Probus

I require only 5 hours on line a week and excuse students who have any kind of excuse. Next year the program(s) will be available on the Internet, which may help some. The course can run indefinitely- I have a few students who have been with me over a year. I hope that the new ITTS program will help students with lower math skills. I also offer tutorials for students who have time to come in, but most don't have the time.

The frustrating thing for me is the large number of students who come in with good intentions, do the assessment, load the program at home, then stop and do not respond to my emails or calls- There are those few who stick with it and have some success, which is gratifying.

Maureen Hoyt

Hi everyone - great to see the conversation keeping up at such a robust level! I just wanted to raise a point from my research review as well as my personal work and research with students. I would caution against pre-assessments that screen people out of classes. Pre assessments in our literacy and ESOL world should screen people *into* the right environments for their skills, otherwise we will lose them - again. Students who responded to evaluations of their online learning experience unanimously say that they learned computer skills AND self-directed learning habits *by* participating. This is a very fine line in service delivery, I know, but I think the key is to encourage students to try and then have supports on hand as/when they need them. We also have to keep pushing to produce and use better, more responsive instructional materials that teach the skills necessary to learn from them.

Good luck to all of us!

Heidi Silver-Pacuilla, Ph.D.

I agree with you Heidi that any form of external or self-assessment of students should draw them in rather than screen them out.

AlphaRoute was created initially to be distance learning resource for adult literacy students in Ontario whose reading, writing and numeracy skills were below what at the time was grade 9 level. Over the years the largest number of students using AlphaRoute have come in at or below the grade 6 level and have used AlphaRoute onsite in their programs more than at a distance from their home.

The idea behind AlphaRoute was to create a secure place online for this audience where students, even those at the earliest stages of reading, could explore and gain basic computer and online learning skills. So within AlphaRoute there are very simple mouse manipulation and clicking games, drag and drop jigsaw puzzles, learning activities that save work to a portfolio that students can review, audio supported text, basic email (no attachments) without the bells and whistles and without spam or fear of introducing a virus, discussion forums, chat rooms, online courses using Moodle which introduces a messaging system, and a short online assessment tool for reading, writing and numeracy. The idea is to give students a place to try out and if they are interested in so doing, master some of the basic tools and skills required in external online learning environments that are more sophisticated and complex.

We have never developed an exit assessment for AlphaRoute that determines if in fact a student has achieved competency in all these areas. That would be something an instructor or program coordinator may have developed.

It seems that many people engaged in this discussion are working with students at much higher levels than we are. The highest end of literacy and basic skills delivery in Ontario falls at the pre-GED preparation level and or grade 12 equivalency delivered through the colleges.

My response here is to insert AlphaRoute as an entry place for gaining computer competency skills and online learning skills for students across the literacy and basic skills spectrum in Ontario.

I will add that AlphaRoute doesn't include a keyboarding competency component and over the years this kind of content piece has been requested by practitioners and students. We believe there are many very good educational software programs available, the Mavis Beacon Typing Program being a very popular one in literacy programs, that can provide a curriculum and paced-keyboarding instruction, so we haven't integrated keyboarding into AlphaRoute.

Nancy

"Blackboard, WebCT, Campus Cruiser, homegrown content and learning delivery systems, etc. generally offer tutorial information, but all too often educational institutions merely regurgitate the tutorial information. Presenting information does not mean that learning has taken place. I've heard too many times from the IT folks, "Why don't they know how to use Blackboard? We offer workshops each semester.""

It's not just understanding how to use the basic tools and features of a delivery system. Course developers and creators can create extremely confusing and frustrating courses. I have found myself "Clicking in Circles" trying to find a particular handout to download for the particular assignment I'm supposed to be working on on a particular day. You click on the course outline link and then on the explanation for the day and then on the calendar and then on the discussion forum, and then on and so on and so forth until you find a link to the handout which was actually never uploaded to the server so you can't download it anyway! You send an email to the instructor and 4 days later you get a reply saying it works fine on their computer! etc...

Barry Bakin

AMEN.

I've seen it done beautifully *right,* too, in various forms, which makes the bad ones more painful.

This makes me think... is there a way a savvy person could document this lapse of professional standards? I'd put the recipe for that on my blog in a heartbeat just in case somebody could hold one of these folks accountable for this kind of stuff.

Susan Jones

Many very good points made here. I can tell the story of one student to illustrate those regarding screening, prerequisite computer skills, and the perhaps un-screenable factor of motivation.

During my most recent recruitment for our program's English for All-based course (mostly DL with optional supplementary support available), one person I phoned from our waitlist was unable to speak with me in English. I explained to Luis (in my elementary-level Spanish) that the course was likely to be challenging, but he was welcome to try it and see how he felt. At the face-to-face orientation, he had great difficulty conversing with the other students but was able to follow along by watching how to navigate the course website.

At home, he had numerous technical issues:

First, he didn't have lots of computer experience and didn't own a computer, but bought one, motivated by the course.

Then, he couldn't view the videos because of a software snafu. I (and my more fluently Spanish-speaking husband) explained another way he could access the videos using iTunes, which he had his more computer-savvy brother help him work out.

He also has had language difficulties:

When I called to check in on Luis because I saw he wasn't moving forward in the course, he said he was having difficulty understanding the videos. I explained how he could resize the iTunes window and the English for All window, so he could read the text of the video as he viewed the movie in order to aid his comprehension.

When I asked him how he feels in the course, what with the difficulties and all, he says he's learning the language, and he does not have another opportunity to learn because he isn't enrolled in any other course.

(Luis was able to hold this conversation completely in English! Struggling, but all in English.)

So, Luis is an example of someone who could have been screened out due to his limited computer and language skills. However, he's persisted in this primarily DL course for 2-1/2 months, he's improved his computer knowledge through his involvement with the class, and he's earning mostly 100% scores on his coursework.

Other students who passed the screening because of their stronger computer and language skills haven't persisted as Luis has.

Diana Satin

Hi Diana,

I'd like to respond to your story about Luis and his persistence. I see this happen a lot within the online environment of AlphaRoute, and with the literacy student that I myself tutor face-to-face. Some students refuse to be defeated by technology and are so eager to learn something new and participate with others that they will plow through incredible barriers to achieve their goals. I am always amazed and impressed.

Of course there are also many students who do give up at the first technology glitch and write off a resource because it didn't work the first time, or created some challenges along the way.

AlphaRoute was developed from its inception to be an environment in which the student has full access to everything available - to be able to explore say what a level 5 activity is like when the student is at level 2. To enroll in an online course (everything is free and non-credited) and to lurk, engage of part, or complete to earn a certificate of participation. This puts the self-management piece into the hands of the student.

Within AlphaRoute there is a small group of students who form a community of sorts. I am always amazed when I speak with a couple of those core students by how much they know about other students across the province because of their interactions by email or chat within AlphaRoute.

There are students who take the AlphaRoute online courses from home and program computers that use dial-up! It can take a long time for some content to load, but the students are so keen to participate and learn online, that they persist when I would not.

I know how much I learn when I refuse to be defeated by an obstacle and work through a problem until I get some resolution - and how good I feel afterwards that I prevailed.

I congratulate Luis for his persistence and to you and the program staff for removing the first barrier and giving him his chance. What was key for his success, apart from his own determination was his self-knowledge that he could succeed and the support along the way from you and your husband that he required and requested.

We have learned from our research into the use of AlphaRoute and what students indicate they require in learning online is that the support of an online mentor as well as onsite support as required is key. Students value and welcome the assistance they need to explore their limits and pursue their goals.

As students move on from basic literacy programs we know they will encounter external assessment tools that will assess and measure their competency in a range of areas, including comfort and facility in online learning environments. And that seems to be where many people engaged in this discussion are interacting with students who have reading, writing and numeracy skills but perhaps not sophisticated enough or honed enough distance online learning skills, including self-management and self-direction to succeed.

I just want to insert what is happening at the early end of the literacy continuum to introduce and prepare students for what is ahead.

Nancy

Hello everyone

I'm Nancy Friday's colleague working with four literacy programs exploring distance online delivery in Ontario since 2003. We developed a number of forms for data collection purposes of the research aspect of the project, which ended up giving the pilots a good indication about how "ready" their learners were for online, blended or flexible delivery of their literacy program. One was a survey adapted from Kentucky Virtual High School to generally determine learners' preferences and predispositions for distance learning environments, another one was adapted from an earlier research study conducted by AlphaPlus to see if learners self-management skills had increased through online learning. Both forms are in the Appendix F (pp. 102-105) of the report at http://distance.alphaplus.ca/

As Nancy pointed out already, the pilot programs agreed that orientation sessions are absolutely necessary. Pre-enrollment and on-going orientation sessions increased retention and success rates of most of their learners. In the beginning of the project, there were doubts if basic literacy learners would be able to cope with the demands of computer-based and web-based learning environments but the ones that decided not to exclude these learners found that it is possible as long as adequate orientation and support is provided. Orientation in this context did not only refer to the technical skills needed to participate in online learning but also and just as importantly to a comprehensive orientation to what it means to learn this way, what is expected of learners and what they can expect of teachers.

Cheers, Matthias