Post by Moses on Apr 24, 2005 12:33:33 GMT -5
If you really are talented, intelligent, and/or have an understanding that there are variables, and non-linearality, you will fail these tests, and you will be held back, and not graduate, and become a drop -out. So much for these tests promoting technical talent in US:
None of the Above[/size]
TWO years ago, fifth graders taking Texas's annual standardized science test faced this multiple-choice question: "Which two planets are closest to Earth?" The four choices were "Mercury and Saturn," "Mars and Jupiter," "Mercury and Venus" and "Venus and Mars."
Simple, right? The Texas Education Agency thought so; every fifth grader should know that Venus and Mars orbit on either side of Earth's orbit (remember the mnemonic "My Very Excellent Mother Just Sent Us Nine Pizzas"?). "Venus and Mars," therefore, would have been a good pick.
But wait, said Mark Loewe, a Dallas physicist who was curious about what students are expected to know and so took the test. The question asked which planets - not which planets' orbits - were closest to Earth. So the correct answer depends on when the question is asked.
"Mercury, which orbits closest to the Sun, is closest to Earth most often," Dr. Loewe said, and sure enough, during that test week in spring 2003, Mercury and Mars were the planets closest to Earth, according to the National Aeronautics and Space Administration Web site. That pair was not among the possible answers.
So is the question valid? Perhaps, since the problem was written for a typical 10-year-old, not someone with Dr. Loewe's understanding of science. On the other hand, the problem ignores the physical world woven into the question, and that might trip up brighter fifth graders.
Beware the perils of ambiguity. It is a mantra that is increasingly pertinent to tests in mathematics and science. The two fields might seem immune from imprecision. But in mathematics, for example, today's tests assess more than a student's ability to do "naked computation," as Cathy Seeley, president of the National Council of Teachers of Mathematics, puts it. In many places, calculators have rendered meaningless the testing of basic computational tasks. Instead, more questions test students' comprehension in real-world contexts. A triangle is a corner garden bed. A rectangular object intersected by a line is a juice box, with a straw. A sloped line on a graph represents a year's worth of payments to the power company.
With these scenarios come variables, and mathematicians and scientists from British Columbia to Boston spend much time picking apart the questions, particularly in online discussion groups. If students are asked how many seeds can be planted in the surface area of a triangular garden, do you put seeds in the corners where there isn't room for plants to take root? What about relevant considerations like seasonality of utility bills or position of the planets? Multiple-choice questions, with no place to show your work and thinking, make such realities more vexing.
"To the lay eye, it may appear that I am being picky, criticizing the minutest detail of the exam," wrote James A. Middleton, a mathematics education professor at Arizona State University, in an online critique of his state's high school exams. "I am being picky. Any first-semester student of psychometrics (the statistical study of test design, administration and analysis) could tell you that if a test is to provide reliable and valid data, its items must be designed well, reflect the standards of the content, and clearly allow students who understand the content to demonstrate that understanding."
Professor Middleton decided that a quarter of the questions he analyzed had mistakes in content or context (he has just completed an analysis of recently released questions and says there is improvement).
"It's an increasingly severe problem," says Walter M. Haney, a senior researcher at Boston College's Center for the Study of Testing, Evaluation and Educational Policy.
The process by which questions are vetted is long and costly. "On most of the tests that are created today, the people who write them and the people who review them do a conscientious and good job," says Gregory Cizek, an education professor at the University of North Carolina at Chapel Hill, who has been an elementary school teacher and a test writer for ACT Inc. But, he adds, "stuff always slips through."
Part of the challenge is writing appropriate questions for a particular grade level while not misleading a student who happens to know more. A 10th grader with a sibling in 12th grade may know some higher-level math; a 12th grader taking a physics course at a local college or online may look at a question differently than another student in the same grade.
Consider, for example, another question critiqued by Dr. Loewe on the Texas Assessment of Knowledge and Skills in 2003. Students in 11th grade were asked to calculate how much force a frog would exert against a river bank while leaping off. Dr. Loewe, who has co-written a textbook on quantum mechanics, says that when he worked out the problem, he included the frog's gravitational weight (its force during rest, which he determined by using the formula for acceleration due to gravity, which was provided at the beginning of the test). But the answer key made clear that the question writer did not expect students to consider that.
Dr. Loewe found several other problems and informed the Texas Education Agency. "Texans are ill-served by such incompetence or dishonesty," he wrote. The agency responded by hiring university professors to review exam questions, says Victoria Young, director of instructional coordination in the agency's testing office. The professors review tests that relate to their areas of expertise but are also familiar with what is taught in high school.
Nonetheless, Ms. Young disputes Dr. Loewe's specific complaints. Asked whether the planets question might have led students to Dr. Loewe's answer, she responded, "Not a realistic viewpoint, in my opinion." And of the leaping frog, she says that physics educators have told her that "only if you brought a very advanced level of college physics to the table would you know enough to know that the answer could be arrived at differently."
Not so, says Dr. Haney of Boston College. Students should have known to take the frog's gravitational weight into account, and so argued a few Texans in letters to The Fort Worth Star-Telegram, which reported on the dispute. Dr. Haney says one reason for the problem is that "a lot of the people who may be writing the math and science questions may not have a deep understanding of the math and science that they are trying to test."
In most cases, the people who write the questions are or have been teachers. Often, they are paid to attend summer workshops led by companies that have contracted with the states to develop the tests.
Marilyn Rindfuss, national senior mathematics consultant at Harcourt Assessment, which creates standardized tests for dozens of states, emphasizes a process she calls "tightening up." Ms. Rindfuss asks every question writer to go through a checklist that includes such questions as "Is the fictional information realistic? (no 75-pound housecats)" and "Is there one, and only one, clearly correct answer?" She also rejects questions that ask test takers to extrapolate patterns, because some people see a pattern that a question writer does not, leading the scorer to start "counting answers wrong that are, in fact, correct."
(To get a sense of such patterns, try this question, which has been kicking around New York City high schools for decades: What comes next in this sequence: 28, 23, 18, 14, __? Read on to find the answer.)
(continued next)
None of the Above[/size]
TWO years ago, fifth graders taking Texas's annual standardized science test faced this multiple-choice question: "Which two planets are closest to Earth?" The four choices were "Mercury and Saturn," "Mars and Jupiter," "Mercury and Venus" and "Venus and Mars."
Simple, right? The Texas Education Agency thought so; every fifth grader should know that Venus and Mars orbit on either side of Earth's orbit (remember the mnemonic "My Very Excellent Mother Just Sent Us Nine Pizzas"?). "Venus and Mars," therefore, would have been a good pick.
But wait, said Mark Loewe, a Dallas physicist who was curious about what students are expected to know and so took the test. The question asked which planets - not which planets' orbits - were closest to Earth. So the correct answer depends on when the question is asked.
"Mercury, which orbits closest to the Sun, is closest to Earth most often," Dr. Loewe said, and sure enough, during that test week in spring 2003, Mercury and Mars were the planets closest to Earth, according to the National Aeronautics and Space Administration Web site. That pair was not among the possible answers.
So is the question valid? Perhaps, since the problem was written for a typical 10-year-old, not someone with Dr. Loewe's understanding of science. On the other hand, the problem ignores the physical world woven into the question, and that might trip up brighter fifth graders.
Beware the perils of ambiguity. It is a mantra that is increasingly pertinent to tests in mathematics and science. The two fields might seem immune from imprecision. But in mathematics, for example, today's tests assess more than a student's ability to do "naked computation," as Cathy Seeley, president of the National Council of Teachers of Mathematics, puts it. In many places, calculators have rendered meaningless the testing of basic computational tasks. Instead, more questions test students' comprehension in real-world contexts. A triangle is a corner garden bed. A rectangular object intersected by a line is a juice box, with a straw. A sloped line on a graph represents a year's worth of payments to the power company.
With these scenarios come variables, and mathematicians and scientists from British Columbia to Boston spend much time picking apart the questions, particularly in online discussion groups. If students are asked how many seeds can be planted in the surface area of a triangular garden, do you put seeds in the corners where there isn't room for plants to take root? What about relevant considerations like seasonality of utility bills or position of the planets? Multiple-choice questions, with no place to show your work and thinking, make such realities more vexing.
"To the lay eye, it may appear that I am being picky, criticizing the minutest detail of the exam," wrote James A. Middleton, a mathematics education professor at Arizona State University, in an online critique of his state's high school exams. "I am being picky. Any first-semester student of psychometrics (the statistical study of test design, administration and analysis) could tell you that if a test is to provide reliable and valid data, its items must be designed well, reflect the standards of the content, and clearly allow students who understand the content to demonstrate that understanding."
Professor Middleton decided that a quarter of the questions he analyzed had mistakes in content or context (he has just completed an analysis of recently released questions and says there is improvement).
"It's an increasingly severe problem," says Walter M. Haney, a senior researcher at Boston College's Center for the Study of Testing, Evaluation and Educational Policy.
The process by which questions are vetted is long and costly. "On most of the tests that are created today, the people who write them and the people who review them do a conscientious and good job," says Gregory Cizek, an education professor at the University of North Carolina at Chapel Hill, who has been an elementary school teacher and a test writer for ACT Inc. But, he adds, "stuff always slips through."
Part of the challenge is writing appropriate questions for a particular grade level while not misleading a student who happens to know more. A 10th grader with a sibling in 12th grade may know some higher-level math; a 12th grader taking a physics course at a local college or online may look at a question differently than another student in the same grade.
Consider, for example, another question critiqued by Dr. Loewe on the Texas Assessment of Knowledge and Skills in 2003. Students in 11th grade were asked to calculate how much force a frog would exert against a river bank while leaping off. Dr. Loewe, who has co-written a textbook on quantum mechanics, says that when he worked out the problem, he included the frog's gravitational weight (its force during rest, which he determined by using the formula for acceleration due to gravity, which was provided at the beginning of the test). But the answer key made clear that the question writer did not expect students to consider that.
Dr. Loewe found several other problems and informed the Texas Education Agency. "Texans are ill-served by such incompetence or dishonesty," he wrote. The agency responded by hiring university professors to review exam questions, says Victoria Young, director of instructional coordination in the agency's testing office. The professors review tests that relate to their areas of expertise but are also familiar with what is taught in high school.
Nonetheless, Ms. Young disputes Dr. Loewe's specific complaints. Asked whether the planets question might have led students to Dr. Loewe's answer, she responded, "Not a realistic viewpoint, in my opinion." And of the leaping frog, she says that physics educators have told her that "only if you brought a very advanced level of college physics to the table would you know enough to know that the answer could be arrived at differently."
Not so, says Dr. Haney of Boston College. Students should have known to take the frog's gravitational weight into account, and so argued a few Texans in letters to The Fort Worth Star-Telegram, which reported on the dispute. Dr. Haney says one reason for the problem is that "a lot of the people who may be writing the math and science questions may not have a deep understanding of the math and science that they are trying to test."
In most cases, the people who write the questions are or have been teachers. Often, they are paid to attend summer workshops led by companies that have contracted with the states to develop the tests.
Marilyn Rindfuss, national senior mathematics consultant at Harcourt Assessment, which creates standardized tests for dozens of states, emphasizes a process she calls "tightening up." Ms. Rindfuss asks every question writer to go through a checklist that includes such questions as "Is the fictional information realistic? (no 75-pound housecats)" and "Is there one, and only one, clearly correct answer?" She also rejects questions that ask test takers to extrapolate patterns, because some people see a pattern that a question writer does not, leading the scorer to start "counting answers wrong that are, in fact, correct."
(To get a sense of such patterns, try this question, which has been kicking around New York City high schools for decades: What comes next in this sequence: 28, 23, 18, 14, __? Read on to find the answer.)
(continued next)