General Assessment of Instructional Needs (GAIN) - Discussion Transcript

General Assessment of Instructional Needs (GAIN)
Discussion Transcript

Announcement | Guest Participant | Preparation



General Assessment of Instructional Needs (GAIN)
Discussion Transcripts from February 23 - 24, 2010



Hi everyone,

I would like to welcome Bradley Olufs of Wonderlic Inc., who we are hosting today and tomorrow on the List to answer questions for us on the new GAIN Test.

Subscribers - are you looking forward to adding the GAIN to your present selection of assessments? What are your thoughts regarding this new test? What are your questions? Are you thinking of pursuing the use of GAIN, and if so, why?

Below is some information on the test that appeared on the announcement flyer:

*******

Wonderlic's new test, General Assessment of Instructional Needs (GAIN), has been approved for use in all state ABE programs funded by OVAE.

Help your participants gain ground, and give your teachers an extremely accurate assessment and placement instrument with GAIN!

Why should your program use GAIN?

  • EFL aligned - directly measures Levels 1-6!
  • Fewer steps and more accurate results - no locator!
  • Shorter administration time - only 90 minutes total length!
  • Simpler results - GAIN tests are automatically scored online!
  • Easier student tracking - included online system exchanges data with state data management systems!
  • Smaller bottom line - less time spent on administration, scoring and tracking!

*******

Full information on this discussion is at:
http://lincs.ed.gov/lincs/discussions/assessment/10gain

Thank you!

Marie Cora

Assessment Discussion List Moderator


Hi Bradley,

I understand that OVAE has approved the GAIN test, but as I understand the situation, Missouri has not, at least so far. Without picking on my state in particular, could you explain the process for states to approve it? -- Does it differ from state to state? Have some states approved it? Do others automatically accept anything OVAE accepts?

Also I'm curious to know if GAIN must be taken online or if there's a paper version, and, if there is a paper version of the test, if it can be scored on paper. It sounds like it's an online test that performs the function of a locator with programmed-in adaptation of the questions' difficulty level as students answer right or wrong -- is that correct? To get complete and accurate results in 90 minutes instead of some 4 hours would be great, but even if my state approves it, I would be concerned about the cost of the test itself -- sounds like it would be ongoing rather than a one-time, possibly grant funded, investment -- as well as the cost (in fact the impossibility) of getting computers online at every site we run.

Debra Smith

GED Coordinator

Parkway AEL, Missouri


Good afternoon Debra:

Now that GAIN is approved for use at federally funded ABE programs, each State will need to update their State Assessment Policy to include GAIN. In the meantime, GAIN can be used as a progress test at ABE Programs. It is my understanding that Workforce programs and non-federally funded adult education/literacy programs can implement GAIN immediately. Based on my meetings across the country, I am confident that GAIN will be approved by many of the States.

GAIN can be administered online or paper/pencil - both versions are machine scored by Wonderlic. The online version is automatically timed and scored by the Wonderlic test server (via an Internet connection). Results are available within minutes via the Wonderlic Online interface. The paper/pencil version can be template scored via the Wonderlic Online interface, with results available upon completion. Wonderlic also supports a fax-back scoring service.

GAIN is not a computer adaptive test, nor does GAIN require a locator test. GAIN was designed with sufficient items at each EFL to accurately measure any student's skills, regardless of their EFL level, on the same test form. Both sections of the GAIN can be administered in 90 minutes total.

Wonderlic supports a tiered price scale for GAIN. The larger number of GAIN units purchased, the lower the unit price. A GAIN unit includes both sections of the test, scoring, and reporting. For significant savings, purchasing at a State or District Level is recommended. There is no fee for staff to become GAIN Certified Test Administrators.

Training is typically conducted via Webinars and runs 1-2 hours.

In addition to a shorter test session, Wonderlic can automatically export the student and GAIN scores to your State database, saving you more time.

I look forward to answering additional questions.

Best Regards,

Bradley Olufs

800-370-9317


Bradley,

I went to your webpage, and there was no mention of alternative formats, accommodations??? Is there a Braille format? Large Print? Audio? The search engine returned nothing. I hope this isn't another standardized test that NRS approved without ADA guaranteed alternative formats.

Michael Tate


Good morning Michael:

The everythingtogain.com website will be providing more information in the next day or so. There are currently 2 forms of GAIN approved by NRS (Form A and Form B). It is recommended to use one form on the pretest and an alternative form for post-testing.

Reasonable accommodation has long been recognized as an essential component of the testing process and it is legally mandated under the Americans with Disabilities Act (ADA). While the ADA does not define reasonable accommodation, it provides a list of examples of what might constitute a reasonable accommodation. A major focus of the ADA is that reasonable accommodation is not effectively addressed by a general policy: rather accommodations are best addressed only on a case-by-case basis. Once it has been documented that the test taker has a disability that is covered by the ADA, the test administrator should initiate discussions to help identify an appropriate reasonable accommodation (extended time, rest breaks, assuring test site is accessible to a person with a mobility issue, reading test questions and recording responses for the test taker).

GAIN content was developed specifically using the EFL definitions. There are charts and graphs making a Braille format more challenging. Large print format can be provided based on demand.

Best Regards,

Bradley Olufs


Bradley,

Actually, ADA requires that businesses, educational programs, transportation, test makers, etc. take "forward actions" (if I am remembering the legal term correctly) which means that alternative formats must be ready and available in advance of the service.

Many service providers believed they could wait, for example, until the first blind student comes in to Braille their program materials, but that actually works to keep blind people from ever coming in, since the program doesn't appear to be all that welcoming to those who are blind if it's going to be a big fuss when they come in. If you are a wheelchair user, you avoid going to the restaurant that is exempted from having ramps.

I wish test makers would think about the alternative format tests that will be needed as they are creating questions, instead of complaining that it is complex to make the alternative when someone asks about the alternative format. For me, this just points up the failings of the standardized test approach.

I am very disappointed that NRS chose, once again, to approve another standardized test, GAIN, without seeing alternative formats. I tended to forgive their approval of CASAS and TABE, because those tests were largely developed prior to ADA (even though there have been disability rights laws on the books since 1973), but to approve a test with no alternative format 18 years after the passage of ADA make one wonder what message NRS is sending.

Thanks to dedicated staff like Ginny Posey, CASAS finally has a Braille test. When can we expect to see Wonderlic's alternative formats?

Michael Tate


Hi Bradley –

As many ABE programs are administered in marginalized areas with a diversity of cultures and experiences, I wonder if this has been addressed for the GAIN? There have been many issues with cultural bias on standardized tests - has Wonderlic worked to prevent this?

I am also wondering how not using a locator or having an adaptive function on the e-version works. With a range of students from 0.0 GLE - 12.9+ GLE, how do you know where to start a student? I can't imagine you give the same test to students of all ability levels. I also would be interested in Michael Tate's questions.

Thanks very much.

Mary Lynn Carver

ABE/GED Instructor

ABE Department Chair

College of Lake County

Grayslake, IL 60030

Phone: 847/543-2677
mlcarver at clcillinois.edu


Good morning Mary Lynn:

During the development of GAIN, a Fairness and Sensitivity panel was assembled by an external consultant in order to provide an independent fairness and sensitivity review of GAIN items. The panel consisted of five members representing four distinct cultural groups (African American, Hispanic, Asian, and Caucasian) and both genders. All five panel members held Masters Degrees or Doctorates, four of which were in the field of Education.

GAIN items were evaluated based on the Fairness Review Guidelines established by the Education Testing Service. These guidelines included:

  • treat people with respect in testing materials
  • minimize the effect of construct-irrelevant knowledge or skills
  • avoid material that is unnecessarily controversial, inflammatory, offensive, or upsetting
  • use appropriate terminology
  • avoid stereotypes
  • respect diversity in depictions of people

Wonderlic carefully considered the recommendations of the Fairness and Sensitivity panel and made adjustments and modification to item content as appropriate. Field testing included adult education sites across the country and included a diverse population of cultural groups and gender.

Your second question about no locator is one of the most commonly asked questions I receive. GAIN items were created using the EFL definitions. There are sufficient questions at each EFL on a single test form to accurately score any individuals EFL level. The forms are designed with the easier questions (EFL-1) at the beginning and gradually getting more difficult. The GAIN is scored using Item Response Theory (IRT) scoring which derives a score that represents a test taker's skill level. The difficulty level of an item and its susceptibility to guessing are all factored into the equation. The IRT model is sensitive to the unique pattern present within an individual's responses, whereas more traditional methods simply focus on the number of items the individual answers correctly. IRT provides an enhanced measurement precision in scoring.

Best Regards,

Bradley


Bradley,

If every NRS level in three subject areas is included in 90 minutes of GAIN test items, how accurate are the test scores for each subject, and at each level?

Sally Gullion

Assessment Coordinator

Learning Works, Vermont's AEL System


Hi everyone,

A colleague just pointed out to me that the GAIN has been approved for about 3 years now. Below is a page from the Federal Register, Feb. 2, 2010, that lists which assessments are NRS approved:

Tests Determined To Be Suitable for Use in the National Reporting System for

Adult Education (NRS) http://www2.ed.gov/legislation/FedRegister/finrule/2010-1/020210e.html

Have any subscribers on the List used GAIN before? Or are at least familiar with it? If so, tell us what you think.

Thanks,

Marie Cora

Assessment Discussion List Moderator


Oops, let me clarify! The Wonderlic Basic Skills Test (WBST) and the General Assessment of Instructional Needs (GAIN)are 2 different assessments.

The WBST was approved for EFLs 3-5 a few years ago. Wonderlic did not resubmit the WBST for re-approval in April 2008, as we created and submitted GAIN for EFLs 1-6.

Bradley Olufs


Hi Bradley - thanks so much for correcting my information!

Marie Cora


I asked in my school system and was told that in Florida, state approval was necessary before any test could be used even if the test had NRS approval. And to the best of my knowledge, I have not seen anything from the state of Florida approving the use of the test.

Robin Matusow


When a program can start to use GAIN will depend upon a number of factors.

1. If your ABE program is receiving Federal Funding and you report to NRS, then the first step is for your State to update their State Assessment Policy to include GAIN.

2. If your program is not receiving Federal Funds, you may be able to start using GAIN right away.

3. Workforce Development programs can start using GAIN as the DOL TEGL 17-05,c2 automatically adopts the NRS list of approved assessments.

4. Any program can use GAIN as a "progress" test.

Bradley Olufs


This question may have been asked and I missed it on the list. What is the minimum required instructional hours between pretest and post test? Are there correlations to other NRS approved assessments?

Jason Mullen


Jason, this is a major hot potato that has a major affect on NRS performance, but the regulations are not perfectly clear nor consistent. As a result I'll be curious to see if anyone can offer you a straight answer for this question.

My understanding is that it all depends on what test is being used, what state you are working in, and how the OVAE officer interprets the agreement between your state plan and the test publisher's recommendations. As a CASAS user in Iowa, we cannot post-test until after they have attend a minimum of 40 hours despite CASAS recommendations that post-testing also occur when a student is leaving the program or at the end of the program year to maximize paired scores. As a result, we can either force students to hang around for 40 or more hours whether they need it or not or suffer the consequences of them not completing a post-test. It is a no-win situation.

Jim Schneider


Thanks Jim, We use CASA as well and have the same 40 or more requirement. My question was in reference to the NEW GAIN assessment: what are the minimum instructional hours for it and does GAIN have correlation to CASAS and or TABE.

Jason Mullen


As Mike Dean explained at the 2010 National State Adult Education Director's, now that the NRS has conducted an official review of the assessments, the test publisher's educational hours between pre- and post-test are a requirement, rather than a recommendation. The individual State Assessment Policy may contain limited exceptions to the hours ruling.

Bradley Olufs


Good morning Jason:

Wonderlic recommends 60 educational hours between GAIN pre and post test.

During the GAIN development it was critical to focus on the content validity of this assessment and its links to NRS EFLs. Also relevant is the relationship between GAIN scores and scores obtained using other measures of similar constructs. In short, "evidence that two measures are highly related and consistent with the underlying construct can provide convergent evidence in support of the proposed interpretation of test scores as representing a candidate's standing on the construct of interest." (SIOP, 2003, p5).

Construct validity evidence for the GAIN was obtained through two separate studies. The first examined the relationship between GAIN scores and scores on another measure of basic English and Math skill, the Test of Adult Basic Education (TABE). The second examined the relationship between GAIN scores and teacher evaluations of student's English and Math competency levels.

Findings from this study revealed a strong correspondence between GAIN and TABE scale scores. GAIN English scores significantly correlated with TABE Language (r=.78, p<.001, N=84) and Reading (r=.81, p<.001, N-101) scale scores. Significant correlations were also found between GAIN Math scores and TABE Total Math scale scores (r=.77,p<.001, N=96). The magnitude of these relationships is remarkable given the fact that the data was archival and that no statistical controls were possible with respect to time differences between the dates of the two test administrations. This finding provides strong evidence of convergent validity between the GAIN and the TABE.

Best Regards,

Bradley Olufs


Just a question. I'm wondering if it is only small rural programs that have a major problem reaching 60 hours for our students. For this current year (beginning July 1, 2009), of the 135 students in our regular program who have attended at least 12 hours, 25 have attended at least 60 hours. That's less than 20%.

The figures for our corrections students are even worse, because they are in the county detention center. Very few stay long enough to get the 12 hours, and we have none with 60 hours or anything even approaching that target number.

How can we show that our students have learned anything if we cannot test them before 60 hours?

Carolyn Dickinson

Western Nebraska Community College


Carolyn,

Your concerns are not unusual. Does your State Assessment Policy contain any exception provisions that enable you to test with less than the required educational hours?

Bradley Olufs


Only by going through obnoxious hoops, getting specific permission from the Department of Education by explaining extenuating circumstances. The mere fact that a student wants to the the GED ASAP and does not want to wait to put in 60 hours doesn't count.

Carolyn Dickinson


I had looked at the page you reference previously, Marie, and my impression was that GAIN and a whole list of other tests have just been approved or re-approved, some for seven years and some for three. GAIN is among the tests approved for three years, starting this month and continuing for three years. (I.e., the three years is future, not past.) The three-year tests have to meet certain conditions the government sets out to be approved for continuing use.

The header says this: The following tests are determined to be suitable for use at all ABE and ASE levels of the NRS for a period of three years from the date of publication of this notice:

A question for Bradley: Can we see a sample test? I have a lot of questions, but I'd rather see for myself.

Debra Smith

GED Coordinator, Parkway AEL

Missouri, the Show-Me State


Good morning Debra:

Thank you for clarifying the Federal Regs for everyone.

The fastest way to get more information, such as sample GAIN questions, is to go to www.everythingtogain.com and register. This will send an email to my attention, and we will follow up the same day.

Thanks for your continued interest,

Bradley Olufs


Dear Colleagues,

I would like to thank Bradley Olufs for responding to our questions regarding the new GAIN assessment, and thank her as well for extending the time she spent with us last week. I hope that everyone found the Q&A useful - thanks as well to the subscribers who engaged in the discussion. As usual, I will prepare the posts in a transcript and post it at the LINCS website; I will alert the List when this is ready to access.

Thank you!

Marie Cora

Assessment Discussion List Moderator