Questions of Validity Surround Teacher Tests

On the night before Nicole M. Malec took the Massachusetts Educator Certification Test (MECT) last month, she obsessed over sharpened pencils instead of poring over review problems.

The review books, which had been unavailable to earlier test-takers, gave little indication of what to expect, but she hoped a good night's sleep and a trip to Staples to buy an electric pencil sharpener would serve her well.

Yet, her trip to Boston's English High School the following morning did little to put her mind at rest--she encountered confused proctors, poor test conditions and a lengthy, awkwardly worded test.

"I was assuming that by [the] January [administration] it'd be cleaned up," she says. "I couldn't believe just how unprofessional the atmosphere would be."

Malec's complaints are not unique.

Since its inception last April, the test has been praised by politicians and criticized by educators.

The MECT attempts to measure the basic skills of prospective teachers by testing candidates in three subject areas over an eight-hour period: writing, reading and a subject area of their choosing.

While politicians have used it as a rallying point during the election year, teachers argue that the test's administration is faulty and its questions misleading.

Bay State politicians used the results of the initial April testing--where 59 percent of the 1,800 prospective teachers failed to pass all three areas-- as campaign fodder, demanding higher standards from Massachusetts teachers.

Though passing percentages for the July, October and January tests have risen, educators and prospective teachers still worried that the MECT is not adequately testing their skills--and that it has become a political tool.

A Fair Test?

On Feb. 11, a team of educators, by organizing a press conference on the steps of the State House, helped to push the issue teacher testing back into the spotlight.

The team--educational writer Anne Wheelock,Boston College Professor Walter Haney and SalemState Professor Clarke Fowler--published a reportcritiquing the MECT as unreliable, citing a highmargin of error, a huge variance between readingand writing scores and test administrationtroubles.

"It's a pretty devastating critique. I thinkthey raise very important issues that shouldresult in an investigation," says Jerome T.Murphy, Dean of Harvard's Graduate School ofEducation. "It does call into question thevalidity of the tests."

But some term the report biased.

"I'm shocked, shocked to hear that they thinkthe tests are unfair," says Abigail Thernstrom, amember of the State Board of Education.

"They don't believe in testing students andthey don't believe in testing teachers," she says.

But Fowler says it is the MECT specifically,and not testing as a concept, that he objects to.

"I support having a test for prospectiveteachers. It's not an unreasonable thing to askthat teachers have basic skills," he says.

Fowler, along with the report's co-authors,points to NES's lack of pre-testing or externalreview as its major problems.

Problems with the NES

While NES had begun developing the test inadvance, they still had less than five months tofinalize the exam, which was to run in a trialstate for the initial April and July sittings.

But because of Massachusetts politicking, theNES was given only five months to create the test.Further, the April and July administrations becameactual exams, not trials.

Across the nation, Massachusetts has become thecase study of what not to do in teacher testing,says Gene J. Campbell, director of licensing forthe Colorado Department of Education.

He says he believes politics unduly influencedthe process of testing and worries whether theintegrity of the test was compromised.

"The Massachusetts experience is well-knownacross the country. The unfortunate thing is thatit has put a shadow of a doubt on [teacher]testing across the nation," he says.

Campbell's state of Colorado, which has usedNES since it initiated teacher testing in 1994, isone of NES's larger clients.

While the NES-administered teacher tests havehad a successful start in Colorado, Campbell saysthe development of the Colorado test in 18 monthswas "tight."

Even considering the 18-month time frame,Campbell said there were logistical problems.

"I don't think, considering the size of ourtask, I'd feel comfortable at all [with fivemonths]," Campbell says of Massachusetts' timeframe. "I am not sure, given the time constraints,[NES] could have avoided these problems."

But in the report, Fowler and his co-authorsalso bring up the questionable work of NES inconducting a state teacher test in Alabama.

In the 1989 case Richardson v. Lamar CountyBoard of Education, the Alabama Supreme Courtfound NES to have "violated the minimumrequirements for professional test development"when it had created teacher tests for Alabama,according to the report.

For Haney, the Alabama decision addscredibility to his report's claims of poor testadministration.

"This company has been documented to produceshoddy work," he says.

NES spokesperson Dominic F. Slowey says thesecharges are less than credible.

"That test [in Alabama] was developed twentyyears ago," he says. "That test bears noresemblance to the Massachusetts test."

Slowey stresses NES's 25-year track record as anational testing organization and its extensivework in multiple states.

He also says that even before the report, NEShad been conducting an in-house assessment of itstesting.

Fowler says this system of evaluation is notonly inherently biased toward NES, but alsoout-of-line with the expected norm forstandardized test companies.

"I'd suspect that NES is going to conclude thatNES did a great job in creating this test," hesays. "This is asking the used car dealer to havehis mechanic inspect the car they're going to sellyou."

Slowey characterizes some of the criticismsaimed at NES as deriving from people looking for ascapegoat.

"It's not unusual to have the test challengedbecause people who failed look for an excuse," hesays.

State education officials echo this comment,saying that those who fare badly tend to be themost vocal.

"People who are satisfied aren't going tocomplain. People aren't going to jump up and downand say, 'What a fantastic test," says KevinCarleton, spokesperson for John Silber, Chair ofthe State Board of Education.

Political Power Play

In a strange series of events, politics andeducation coalesced to produce the April versionof the MECT.

Even before the April results were fullyreleased, Massachusetts Speaker of the HouseThomas M. Finneran (D-Mattapan) bashed teacherswho failed as "idiots."

Cellucci latched on to teacher testing as arallying point in his raucous gubernatorial raceagainst Attorney General L. Scott Harshbarger '64.

After finalizing the contract with NES todevelop the exam one year ago, Commissioner ofEducation Robert Antonucci resigned his post twomonths before the initial April 4 tests.

Next, the Department of Education withdrew astudy guide early in the month because testcontent had been altered.

They dropped a bombshell 10 days before thefirst April testing: they required prospectiveteachers to take--and pass--the exam to achievecertification.

The Board of Education voted on June 22 tolower the minimum passing score because the testwas untried. But ten days later, at the behest ofthen-acting Governor Cellucci, the Boardreconvened and voted to raise the cut-off point,prompting the resignation of Interim CommissionerFrank Haydu.

""I think it was highly political. The debatewas filled with finger-pointing," Murphy says."Candidates were trying to outdo themselves withteacher-bashing."

Acting Commissioner David Driscoll hasattempted to guide the department through the lastturbulent few months, vigorously defending theteacher tests.

"The focus of the criticism of the tests ismisguided. The issue is about the answers to tests[which] clearly indicate poor basic literaryskills," he said in a press release.

Teachers' Tribulations

But the intertwining of education and politicshas resulted in a paradox.

As Massachusetts weeds out supposedlyunqualified candidates, it faces a dire teachershortage, as a majority of the teacher populationis reaching retirement age.

"How in the hell do we attract more people intothe profession?...I don't think anybody has ananswer," says Mark Roosevelt '78, a co-author ofthe 1993 Massachusetts Education Reform Act.

Haney agrees. "If you don't have an oversupplyof teachers, it doesn't matter even if you have avalid test," he says.

Meanwhile, teachers question whether the testadequately assesses their knowledge, and whetherNES is a qualified administrator.

Malec, for example, says that in her test, thetape recorder used in the dictation portion wasgarbled, the wait to take the test lasted over anhour and the questions were ambiguous.

She says she is worried that these problems,though small, are representative of NES's biggerproblems.

"If they can't fix these little things, how doyou deal with the big ones?" she asks.

Cheryl T. Haynes, who has been teaching atCambridge's Martin Luther King Jr. School for 24years, thinks there is another way to testteachers.

Haynes says that an in-school evaluation likethe King School uses, where the principalevaluates the teachers in a class-room setting, ismore relevant.

For Katherine Kelley, president of theMassachusetts Federation of Teachers, the teachertests have become a political tool.

"[Teachers] are constantly castrated for notdoing their jobs when they're spending hours uponhours doing extra work," she says. "They've gottenno credit and they've gotten constant bashing."

Kelley says the recent debate has brought themorale of teachers to what she says is the lowestin thirty years.

Malec seems to exemplify this.

Although Malec, despite her tribulations,passed the teacher exam, she is looking to otherprofessions as a 27-year-old mother of twochildren.

"It's frustrating because it's really affectingpeople's career choices. This is really affectingpeople's lives. It's really a shame," she says.

"I find it hard to get excited about a jobwhere you are certain to take home work everynight, you can only go to the bathroom atspecified times of the day, and everyone (parents,politicians, etc.) complains that you have it easyor make too much money," she said in an e-mailmessage.CrimsonSarah E. HenricksonTEST PREP At the Martin Luther KingJr. school on Putnam Ave., teacher CHERYL T.HAYNES (rear) oversees her eighth graders.

The team--educational writer Anne Wheelock,Boston College Professor Walter Haney and SalemState Professor Clarke Fowler--published a reportcritiquing the MECT as unreliable, citing a highmargin of error, a huge variance between readingand writing scores and test administrationtroubles.

"It's a pretty devastating critique. I thinkthey raise very important issues that shouldresult in an investigation," says Jerome T.Murphy, Dean of Harvard's Graduate School ofEducation. "It does call into question thevalidity of the tests."

But some term the report biased.

"I'm shocked, shocked to hear that they thinkthe tests are unfair," says Abigail Thernstrom, amember of the State Board of Education.

"They don't believe in testing students andthey don't believe in testing teachers," she says.

But Fowler says it is the MECT specifically,and not testing as a concept, that he objects to.

"I support having a test for prospectiveteachers. It's not an unreasonable thing to askthat teachers have basic skills," he says.

Fowler, along with the report's co-authors,points to NES's lack of pre-testing or externalreview as its major problems.

Problems with the NES

While NES had begun developing the test inadvance, they still had less than five months tofinalize the exam, which was to run in a trialstate for the initial April and July sittings.

But because of Massachusetts politicking, theNES was given only five months to create the test.Further, the April and July administrations becameactual exams, not trials.

Across the nation, Massachusetts has become thecase study of what not to do in teacher testing,says Gene J. Campbell, director of licensing forthe Colorado Department of Education.

He says he believes politics unduly influencedthe process of testing and worries whether theintegrity of the test was compromised.

"The Massachusetts experience is well-knownacross the country. The unfortunate thing is thatit has put a shadow of a doubt on [teacher]testing across the nation," he says.

Campbell's state of Colorado, which has usedNES since it initiated teacher testing in 1994, isone of NES's larger clients.

While the NES-administered teacher tests havehad a successful start in Colorado, Campbell saysthe development of the Colorado test in 18 monthswas "tight."

Even considering the 18-month time frame,Campbell said there were logistical problems.

"I don't think, considering the size of ourtask, I'd feel comfortable at all [with fivemonths]," Campbell says of Massachusetts' timeframe. "I am not sure, given the time constraints,[NES] could have avoided these problems."

But in the report, Fowler and his co-authorsalso bring up the questionable work of NES inconducting a state teacher test in Alabama.

In the 1989 case Richardson v. Lamar CountyBoard of Education, the Alabama Supreme Courtfound NES to have "violated the minimumrequirements for professional test development"when it had created teacher tests for Alabama,according to the report.

For Haney, the Alabama decision addscredibility to his report's claims of poor testadministration.

"This company has been documented to produceshoddy work," he says.

NES spokesperson Dominic F. Slowey says thesecharges are less than credible.

"That test [in Alabama] was developed twentyyears ago," he says. "That test bears noresemblance to the Massachusetts test."

Slowey stresses NES's 25-year track record as anational testing organization and its extensivework in multiple states.

He also says that even before the report, NEShad been conducting an in-house assessment of itstesting.

Fowler says this system of evaluation is notonly inherently biased toward NES, but alsoout-of-line with the expected norm forstandardized test companies.

"I'd suspect that NES is going to conclude thatNES did a great job in creating this test," hesays. "This is asking the used car dealer to havehis mechanic inspect the car they're going to sellyou."

Slowey characterizes some of the criticismsaimed at NES as deriving from people looking for ascapegoat.

"It's not unusual to have the test challengedbecause people who failed look for an excuse," hesays.

State education officials echo this comment,saying that those who fare badly tend to be themost vocal.

"People who are satisfied aren't going tocomplain. People aren't going to jump up and downand say, 'What a fantastic test," says KevinCarleton, spokesperson for John Silber, Chair ofthe State Board of Education.

Political Power Play

In a strange series of events, politics andeducation coalesced to produce the April versionof the MECT.

Even before the April results were fullyreleased, Massachusetts Speaker of the HouseThomas M. Finneran (D-Mattapan) bashed teacherswho failed as "idiots."

Cellucci latched on to teacher testing as arallying point in his raucous gubernatorial raceagainst Attorney General L. Scott Harshbarger '64.

After finalizing the contract with NES todevelop the exam one year ago, Commissioner ofEducation Robert Antonucci resigned his post twomonths before the initial April 4 tests.

Next, the Department of Education withdrew astudy guide early in the month because testcontent had been altered.

They dropped a bombshell 10 days before thefirst April testing: they required prospectiveteachers to take--and pass--the exam to achievecertification.

The Board of Education voted on June 22 tolower the minimum passing score because the testwas untried. But ten days later, at the behest ofthen-acting Governor Cellucci, the Boardreconvened and voted to raise the cut-off point,prompting the resignation of Interim CommissionerFrank Haydu.

""I think it was highly political. The debatewas filled with finger-pointing," Murphy says."Candidates were trying to outdo themselves withteacher-bashing."

Acting Commissioner David Driscoll hasattempted to guide the department through the lastturbulent few months, vigorously defending theteacher tests.

"The focus of the criticism of the tests ismisguided. The issue is about the answers to tests[which] clearly indicate poor basic literaryskills," he said in a press release.

Teachers' Tribulations

But the intertwining of education and politicshas resulted in a paradox.

As Massachusetts weeds out supposedlyunqualified candidates, it faces a dire teachershortage, as a majority of the teacher populationis reaching retirement age.

"How in the hell do we attract more people intothe profession?...I don't think anybody has ananswer," says Mark Roosevelt '78, a co-author ofthe 1993 Massachusetts Education Reform Act.

Haney agrees. "If you don't have an oversupplyof teachers, it doesn't matter even if you have avalid test," he says.

Meanwhile, teachers question whether the testadequately assesses their knowledge, and whetherNES is a qualified administrator.

Malec, for example, says that in her test, thetape recorder used in the dictation portion wasgarbled, the wait to take the test lasted over anhour and the questions were ambiguous.

She says she is worried that these problems,though small, are representative of NES's biggerproblems.

"If they can't fix these little things, how doyou deal with the big ones?" she asks.

Cheryl T. Haynes, who has been teaching atCambridge's Martin Luther King Jr. School for 24years, thinks there is another way to testteachers.

Haynes says that an in-school evaluation likethe King School uses, where the principalevaluates the teachers in a class-room setting, ismore relevant.

For Katherine Kelley, president of theMassachusetts Federation of Teachers, the teachertests have become a political tool.

"[Teachers] are constantly castrated for notdoing their jobs when they're spending hours uponhours doing extra work," she says. "They've gottenno credit and they've gotten constant bashing."

Kelley says the recent debate has brought themorale of teachers to what she says is the lowestin thirty years.

Malec seems to exemplify this.

Although Malec, despite her tribulations,passed the teacher exam, she is looking to otherprofessions as a 27-year-old mother of twochildren.

"It's frustrating because it's really affectingpeople's career choices. This is really affectingpeople's lives. It's really a shame," she says.

"I find it hard to get excited about a jobwhere you are certain to take home work everynight, you can only go to the bathroom atspecified times of the day, and everyone (parents,politicians, etc.) complains that you have it easyor make too much money," she said in an e-mailmessage.CrimsonSarah E. HenricksonTEST PREP At the Martin Luther KingJr. school on Putnam Ave., teacher CHERYL T.HAYNES (rear) oversees her eighth graders.