Board makes assessment and accountability system easier to understand

By Matthew Tungate
matthew.tungate@education.ky.gov

The Kentucky Board of Education simplified how test scores will be used to calculate how well schools are doing in the new assessment and accountability system during its February meeting.

As in the models the board saw in its last two meetings, schools and districts would be accountable for five areas based on the results of still-to-be-designed tests: student achievement; closing achievement gaps; showing growth/progress; college- and career-readiness (middle and high schools); and graduation rates (high schools).

However, how the scores would be calculated in the achievement, gap, growth and readiness areas in the newest version of the plan changed.

“This is a fairly major change from what you have seen in October and December,” Commissioner Terry Holliday said.

He told the board that the changes make the new model more statistically valid and easier to understand.

“I don’t think we can argue with statistically valid,” board member Brigitte Ramsey replied.

The newest proposal calls for schools to receive one point for each percentage of students scoring proficient or distinguished in each content area and half a point for each percentage of students scoring apprentice, just as the previous models had. However, at their last board meeting, members agreed that districts should be rewarded for students scoring distinguished and penalized for students scoring novice.

So the newest proposal calls for bonus points: schools can gain half a point for each percentage of students scoring distinguished and lose half a point for each percentage of students scoring novice. However, the proposal says, “If the novice performance completely offsets the distinguished bonus, no points are added to or subtracted from the achievement calculation.”

The biggest change comes in how gap, growth and college and career readiness are calculated. The previous versions of the assessment and accountability model required each school to have a baseline number and a goal, and divided the difference over time to set the expected annual improvement.

However, using real numbers for schools showed that using that model caused some schools to get no credit in those three areas because of minor differences in students’ scores from year to year, Associate Commissioner Ken Draut said.

“The simulation didn’t work very well at all with that model,” he said.

So results from each of the three will be used differently to calculate a new raw score.

The gap score is based on the percentage of students who score proficient and distinguished. Like previous proposals, the newest model only counts students’ scores once even if they fit into multiple demographic categories that are considered “gap” groups, such as those based on race or ethnicity, income or disability.

As in the previous models, the proposed growth calculation is designed to measure a student’s growth in learning each year as compared to the student’s academic peers. So, higher-performing students’ scores are compared against other high-scoring students, while lower-performing students’ scores are compared against other lower-scoring students’ scores.

Under the newest proposal, points are awarded for the percentage of students who grow at typical or higher levels relative to their peers. A regulation related to the new assessment and accountability system says, “Typical yearly growth shall be at least the 40th student growth percentile or higher.”

For high schools, readiness for college or career will be calculated by dividing the number of high school graduates that have successfully met one of the indicators of readiness by the total number of graduates. The indicators of readiness include:

• the Kentucky Council on Postsecondary Education’s Systemwide Benchmarks on the ACT in reading, English and mathematics
• the Kentucky Council on Postsecondary Education’s College Placement Test Benchmarks
• the Career Measures as defined by the Kentucky Department of Education

For middle schools, a college-readiness percentage will be calculated by averaging the percentage of students who meet the ACT EXPLORE benchmarks for reading, English and mathematics.

Holliday said establishing college readiness “is pretty well established.”

However, the president and director of the Kentucky Association of Career and Technical Education (KACTE) presented recommendations to the board that would change how career readiness is calculated.

The association recommends defining career readiness as:

• the level of preparation attained by students in core academic communication and mathematics skills, which will allow students to function and excel in the classroom, at the workplace or in routine daily activities
• employability skills, such as critical thinking and responsibility, which are essential in any career area
• technical and job-specific skills related to a specific career pathway that offers life-sustaining wages and opportunities for advancement

It also suggested measures of readiness in each area.

President Ken Talley told the board that KACTE believes that high school students should use a portfolio that includes:

• grade-point average in academic and technical courses
• attendance
• leadership activities such as participation in Career and Technical Education Student Organizations (CTSOs)
• skill demonstrations through CTSO events
• work-based learning of all types (accompanied by external evaluation)
• examples of work accomplished

He said much of the information could be compiled in the Individual Learning Plan, but acknowledged that using the portfolio would be more work.

Holliday reminded the board that Senate Bill 1 (SB 1), which requires the Kentucky Department of Education to develop a new assessment and accountability system to be in place by the 2011-12 school year, does not mention career readiness.

“You will get some pushback from folks who say, ‘We passed Senate Bill 1 to focus on college readiness,” he said. “’We didn’t pass Senate Bill 1 to focus on career readiness.’”

Vice chair Dorrie Combs said the board values career readiness as much as college readiness.

“We’re a broad-shouldered group who are ready to take any flak that may come our way,” she said.

The board did not change the way the graduation rate will be used in high school, as a school’s graduation rate will be used as the raw score.

Calculating the score

The board also changed some of the weights that each of the five categories will receive. The proposed weights are:

Grade Range	Achievement	Gap	Growth	Readiness for College/Career	Graduation Rate	Total
Elementary	30%	30%	40%	N/A	N/A	100%
Middle	28%	28%	28%	16%	N/A	100%
High	20%	20%	20%	20%	20%	100%

Middle school weights had originally been 30 percent in achievement, gap and growth and 10 percent in college readiness. High school weights had been 25 percent across the board with an undetermined weight in college- and career-readiness.

So schools will multiply their raw score in each category by the percentage each category carries for their overall score.

Based on their combined scores in those five areas, schools and districts would be classified as “Distinguished,” “Proficient,” “Needs Improvement” or “Persistently Low-Achieving,” depending on where the board sets the cut-off score for each area. “Persistently Low-Achieving” is defined by state law and the federal government.

However, the board has yet to establish the cut-off score for each classification, and Draut said it likely won’t be able to do so until after scores are received from testing in the spring of 2012.

Without real data, the board could set the standards and then realize that they were too easy or too difficult.

“If we said the proficient box was at 90, and in our first data run everybody is at 92, every single school is proficient,” he said. “You wouldn’t want to build a system like that. Or the opposite: you wouldn’t want to set 90 and find out that everybody’s at 20 and realize that we’ve set a goal that nobody can reach.”

Ex officio board member Robert King, president of the Council on Postsecondary Education, disagreed with that idea.

King said the board should resist the notion that they should lower the requirements for each category if too many schools score below the cut score.

“I think the measure has to be, in a sense, almost an absolute,” he said. “Are we measuring what these students know?”

The board will continue its discussion and likely vote on a proposed model at its April meeting.

Discussion on recognition and assistance

The board also discussed recognition, assistance and consequences for schools and districts, based on their performance and standings in the proposed accountability model.

Associate Commissioner Larry Stinson presented the board with some ideas about how to recognize schools districts that exceed still-to-be-determined expectations overall and in individual areas. Ideas ranged from the tried-and-true, like flags and certificates, to innovations such as freedom from some regulations and allowing some schools to create their own accountability models.

Stinson also said the board will have to decide when and how much assistance to give schools that don’t meet expectations. For instance, schools not identified as persistently low-achieving but still in the bottom 20 percent of state scores may have to rewrite their school plan or partner with a school with higher performance.

Holliday said the board will eventually have to decide if there will be any “teeth” to the consequences.

“We can’t keep just labeling folks and saying, ‘Now go out and do good things,’” he said. “We’ve got to figure out which ones we can give assistance to and what type of assistance within the capacity we have here in the department.”

Holliday said the recognition and assistance won’t be used until the fall of 2012 and only for the part of the assessment and accountability system that focuses on student testing.

The overall accountability system incorporates not only student testing but all aspects of school and district work and includes measures of principal and teacher effectiveness, and school and district support, including a working conditions survey and program reviews.

Program reviews have been written for three areas: arts and humanities, practical living and career studies and writing. Rather than testing students to see what they have learned, schools gather evidence about how they integrated the subjects across their curricula and whether they provided students with opportunities to learn.

In December the board discussed adding a fourth program review in world languages, separating it from arts and humanities. Several educators and parents from the Frankfort Independent school district appeared before the board to request that world languages be separated from arts and humanities and given its own program review.

At the February meeting, Associate Commissioner Felicia Cumings-Smith presented a survey of 1,101 teachers, administrators, parents, community members and students that showed 76 percent of respondents believe world languages should have its own program review.

Holliday told the board that SB 1 mandates the three program reviews that have been written and “they’re going to be ready.”

However, he does not believe that a world languages program review can be done next year.

“We’re going to need a year just to receive input from the field (on a world language program review),” he said.

The commissioner said the board has mentioned numerous programs that could be part of a program review, and he suggested the board prioritize a list.

Combs said she understands the departments’ restraints but thinks world languages need to be emphasized.

“We keep putting this on the back burner,” she said.

The board agreed to discuss implementation dates of future program reviews and ways to implement a world languages program review at its April meeting.

In other business, the board:

• voted to support Education Commissioner Terry Holliday’s call to continue as planned with the implementation of mandates from 2009’s SB 1
• approved district facility plans for the Bath County, Hazard Independent and Fulton Independent school districts
• approved 2010-2011 local district tax rates levied
• gave final approval to 702 KAR 7:065, the regulation related to high school interscholastic athletics
• approved the appointment of J. Gary Mudd of Louisville to the Kentucky School for the Blind Advisory Board
• approved the appointment of Hardin County teacher Vasco Perry to the State Textbook Commission

The Kentucky Board of Education’s next regular meeting is scheduled for April 13 in Frankfort.