(Don Harder / Flickr)
Chicago Public Schools’ internal watchdog is recommending the school district overhaul its Northwest Evaluation Association (NWEA) testing processes after finding a “concerning level” of possible “gaming and cheating techniques,” including longer-than-average test durations and high numbers of pauses.
In a new report submitted to the Chicago Board of Education, CPS Inspector General Nicholas Schuler said a series of data analyses conducted by his office found “unusual patterns” in the application of the untimed, computer-based test known as the NWEA Measures of Academic Progress (MAP) Growth test.
“These analyses, combined with interviews of students and teachers at some schools with unusual results, indicated that proper test administration procedures were repeatedly violated during the Spring 2018 NWEAs,” Schuler wrote in the memo. “This occurred in a minority of cases, but enough to be worrisome and to warrant action.”
In response to these findings, CPS said its current test durations are high in comparison to the NWEA averages, but noted that Schuler’s memo found no correlation between test duration and academic growth, meaning the longer test times didn’t necessarily benefit students, adding that “it does not call into question the accomplishments of our students and school communities.”
“The untimed format of the NWEA assessment requires additional guidance, training, and auditing, and we are carrying out a thorough plan of action to better support our schools,” CPS spokeswoman Emily Bolton said in a statement Friday.
NWEA tests are taken by students in second through eighth grades and their results have a host of applications within CPS, including student grade promotions, selective enrollment admissions, principal evaluations, school SQRP ratings and curriculum decisions made at a district level.
(Chicago Public Schools Office of Inspector General)
According to the report, one out of every four CPS NWEA tests taken during the spring 2018 session – almost 83,000 tests in total – took twice the national average duration to complete. Some 24,000 students took three times the national average, and nearly 7,500 students took four or more times that average to finish the test.
“As a result,” Schuler said, “in some CPS schools and grades, a maximum 53-question test that the average student nationally completed in roughly an hour turned into a multi-day and even a week-long event.”
Schuler noted in his report that these extended durations were largely clustered in just a handful of schools. For instance, about 20% of the students whose completion time was three times the national norm took the test in 14 out of a possible 500 schools.
Schuler said CPS testing times have also worsened over time, and that “if no action is taken, CPS durations could continue to move farther and farther from national norms.”
Part of the extended testing duration resulted from pauses – in some cases, dozens upon dozens of pauses – which allowed students additional time and gave them a chance at receiving potentially easier questions.
During the spring 2018 session, more than 12,000 CPS tests paused five or more times, but Schuler found some tests were paused 20, 30, 50 or even 60 times, with the heaviest rates coming among seventh- and eighth-graders who, again, were often clustered in the same schools.
By pausing the test, the student’s current question is replaced with “a new question of similar difficulty,” according to the report.
“However, if questions are paused because students don’t know the answer, students ‘have a 50 percent chance of getting a new question that might be more favorable,’ NWEA told the (Office of Inspector General),” Schuler wrote in the memo. “Consecutive pauses of the same question number increase those odds ‘considerably,’ NWEA said.”
(Chicago Public Schools Office of Inspector General)
Schuler’s office conducted interviews with 20 students and 10 teachers, some of whom outlined how they would get around anwering certain questions.
The report states:
“At another school, a Diverse Learner said his proctor would pause his sixth-grade Reading test up to three times in a row to net him new questions before finally requiring him to answer the question.
“At this same school, a former CPS general education student said two of the proctors in his fourth-grade Math testing room told students who were stumped on a question to raise their hands so the proctors could pause the test and resume it with a new question. Students merely had to say, ‘Can I get a new question?’ when they didn’t know an answer. If a student still didn’t know the answer the adults would give clues, such as nodding their heads ’yes’ or ’no,’ the student said. However, on the Reading test, which featured different proctors, students were not allowed to ask for new questions, the student said.”
In some cases, test proctors would pause a student’s test when they didn’t know the answer in order to give them a new question, according to the report. Schuler determined this was “an attempt to game the test.” Students could also let difficult questions “time out” by waiting for 25 minutes. Proctors could then resume the student’s test for them, bringing up a different question.
The NWEA told Schuler that there is “no reason” for a student or proctor to let a test time out, and doing so can compromise the assessment’s validity.
Schuler’s office officially made no finding about district malfeasance, stating that the “unusual patterns” may be due to cheating or permissible gaming, but could’ve also been caused by other, more benign reasons.
The report made several corrective recommendations – which have been accepted by CPS – to try and improve the district’s testing process, including:
– Reducing duration averages by setting a “reasonable” time limit for general education students
– Limiting pauses and develop instructions for proctors
– Recording the proctors of each test so they may be audited
– Establishing penalties for cheating
– Hiring a test security expert
“Moving forward,” Bolton said, “we will be working with parents, community members, educators, and school leaders to evaluate the full assessment landscape in our schools and determine if opportunities exist to reduce testing time while ensuring educators and parents have access to actionable academic data.”
Schuler, who has served as inspector general since 2014, turned in his resignation to Mayor Lori Lightfoot last month amid an investigation into allegations that he led a “toxic workplace.” His last day on the job is next week.
In an emailed statement, he said the release of his report Friday “took us by surprise,” as he had expected to disclose those findings at next week’s board meeting. He also criticized the district for failing to redact the names of certain schools, as his office typically does in its reports.
“It was always our position and understanding that schools would not be identified publicly because no findings of impropriety were made about individual schools,” Schuler said. “We urge the public not to jump to conclusions about individual schools based on our report.”
Contact Matt Masterson: @ByMattMasterson | [email protected] | (773) 509-5431
