Effectiveness of the KEMTP Test in Predicting Final Grades
Paul Eakin, Carl Eberhart and Steve Newman
Paul Eakin is Professor of Mathematics at the University of Kentucky (UK), website director of the Kentucky Early Mathematics Testing Program (KEMTP), and principal investigator for the Appalachian Mathematics and Science Partnership sponsored by the National Science Foundation.
Carl Eberhart is Professor of Mathematics at the University of Kentucky, website director of the KEMTP, and Director of Undergraduate Studies in Mathematics at the University of Kentucky.
Steve Newman is Professor of Mathematics at Northern Kentucky University, program director of the KEMTP, a member of the Kentucky Department of Education’s Algebra II Task Force, and a member of the national Content Expert/Employer Panel in mathematics for the American Diploma Project.
I. Overview
The Kentucky Early Mathematics Testing Program (KEMTP) was created when House Bill 178, written and sponsored by Representative Jon Draud (R, Crestview Hills), was unanimously passed by the 2000 General Assembly of the Commonwealth of Kentucky and signed into law by Governor Paul E. Patton. The program offers a voluntary, online mathematics test intended primarily for high school sophomores and juniors in Kentucky. The test provides students an assessment of their mathematical preparation for college early enough so that they can improve their mathematical preparation while still in high school and thereby avoid placement into remedial courses in college and increase their chances of success in college-level mathematics and science courses. The program is funded by the Kentucky Council on Postsecondary Education and is a partnership between Northern Kentucky University (NKU), which administers the program, and the University of Kentucky (UK), which develops and maintains the website. The program began testing in the spring of 2001 and has been testing every semester since then.
The University of Kentucky mathematics department began giving the KEMTP test to all students in their elementary calculus course, a course required of all business and economics majors, in the fall of 2001. Students take the test the first day of class. At the end of the semester, the scores they got on the test are compared with their final grades. The correlation between these test scores and final grades is remarkably good. Although a positive correlation between basic algebra and geometry skills at the beginning of a calculus course and final grades in the course would be expected, the correlation was so strong and so consistent over time that the UK mathematics professors were impressed and surprised. The department also gave the test in a similar way to all students in their spring, 2003, sections of college algebra, a course required of all students seeking a bachelor’s degree, and to all students in their spring, 2002, sections of first-semester calculus, a course designed for students majoring in mathematics, science, engineering and computer science. The correlation between the test scores and final grades in these two courses is also remarkably good.
II. Background Information about the KEMTP Test
1. Test Content
The KEMTP test covers only topics taught in Algebra I, Geometry and Algebra II in Kentucky high schools. Thus the test covers standard right triangle trigonometry, but not circular trigonometry that is generally covered in a precalculus or trigonometry course. The test does not cover exponential and logarithmic functions because these topics are generally covered in courses beyond Algebra II, even though they are sometimes introduced earlier. Most of the topics covered on the KEMTP test are in Kentucky’s Grade 9 to 11 Core Content for Mathematics Assessment. Some topics, such as solutions of quadratic equations, are covered on the KEMTP test but are not in the Core Content, apparently because these topics are generally covered in Algebra II, a course not required of all students. However, if the topics contained in the Algebra II guidelines developed by a Kentucky Department of Education task force are combined with topics in the Core Content, then every question on the KEMTP test involves one or more topics in these two documents.
2. Test Construction
A test construction committee, consisting of roughly six high school mathematics teachers, six college mathematics teachers and a representative from the Kentucky Department of Education (KDE), assembles each June to construct the KEMTP test for the following year. The committee has met for the last two years and will meet again this year. A spirited discussion of testing issues and test questions takes place in a congenial atmosphere of cooperation during the meeting, and following the meeting via e-mail, until the unanimously agreed upon test has been developed.
The committee examines the mathematics portion of the American College Test (ACT) that is required for admission to Kentucky public colleges and universities, various in-house mathematics placement tests developed by Kentucky colleges and universities, the state assessment test called the Kentucky Core Content Test, and early mathematics placement tests given in other states. The committee is ever mindful of the test development directions given it by the General Assembly in House Bill 178:
The computer website-testing program shall develop appropriate tests to determine the level of mathematics knowledge of high school students in relation to standards of placement tests given at the community and technical colleges and undergraduate public universities.
3. Test Acceptance
High school mathematics teachers throughout Kentucky have accepted the KEMTP test as a reasonable and fair evaluation of their students’ mathematical preparation for postsecondary education. Many favorable comments have been received both anecdotally and in writing through the KEMTP end–of-year survey sent to all the high schools in the state. Some teachers have expressed concern that their students are being tested on material not yet covered, but such concern has never been a criticism of the test. The Kentucky college mathematics community has also embraced the test, as reflected by the willingness of Kentucky colleges and universities to participate actively in support of the testing program. The University of Kentucky, in particular, has led the way in using the test as a yardstick to evaluate the mathematical preparation of their students. Cheryl King, Vice President for Adult Education at the CPE, is urging the adult education community throughout Kentucky to use the KEMTP test so that adult learners can determine their mathematical preparation for postsecondary education.
The use of the KEMTP test has spread beyond Kentucky. The University of Oklahoma (OU) is in the process of developing an online test modeled on the KEMTP. In the spring of 2002, OU officials requested and received copies of the previous KEMTP tests. They examined the tests in detail to determine whether the tests covered the topics they wanted their college algebra students to know. They examined the Oklahoma High School Mathematics Standards and found that the KEMTP tests were closely aligned with these standards. In the end, they constructed a 30-question test using 28 KEMTP test questions and gave this test on paper to some of their college algebra students in the fall. In the spring of 2002, they visited with KEMTP directors in Lexington and decided to run their program through the KEMTP website next year and use the KEMTP test. Eastern Washington University officials are interested in starting a testing program in the Cheney, Washington, area and they too will run their program through the KEMTP website using the KEMTP test.
Several high schools in Tennessee have tested their students through the KEMTP website using the KEMTP test over the past several semesters. These schools are now participating in the Appalachian Mathematics and Science Partnership (AMSP), a $22 million project sponsored by the National Science Foundation (NSF) to improve mathematics and science education in the Appalachian region. UK professor Paul Eakin is the principal investigator for the AMSP and website director for the KEMTP. Much of what is being done in the AMSP grew out of work done in developing the KEMTP. Thus it is to be expected that the KEMTP test will used by more and more high schools nationwide as the AMSP becomes better known.
III. Effectiveness of the KEMTP Test in Predicting Final Grades in Elementary Calculus
1. Testing Environment and Procedures at UK
The elementary calculus course (MA 123) at UK provides an excellent, controlled environment to evaluate the efficacy of the KEMTP test. A large number of students take this course since it is required for all business and economics majors. All examinations in the course, including the final examination, are the same for all students. Each of the instructors teaching a section of the course grades a single question on a given examination for all students in all sections of the course to insure uniformity of grading.
A portion of the KEMTP test (15 questions out of the 30 KEMTP test questions in the fall of 2001, 12 out of 25 in the fall of 2002 and in the spring of 2003) is given to all students on the first day of class. The questions are randomly selected from the KEMTP test so that there are at least three different tests given to students depending on the time when the test is administered. Thus nearly all of the KEMTP questions are likely to be used on at least one of the tests. The course instructors grade the test and put aside the test scores until final grades in the course have been given. These test scores are then compared to the final grade.
2. Fall 2001: Test Results Compared with Final Grades
A total of 929 students in elementary calculus in the fall of 2001 took a 15-question portion of the KEMTP test as described in section 1 above. A comparison of their scores and final grades is given in Table 1. The number 4 in row 2, column 2 is the number of students who got a KEMTP test score of 15 and got an A in the course. The other numbers in the middle six columns of the table are interpreted in a similar way. The percentages in Table 1 and Table 2 are calculated so that, for example, Percent (A, B, C) is the percentage of As, Bs, and Cs out of the total number of As, Bs, Cs, Ds, Fs, and Ws.
TABLE 1
|
KEMTP Test Score |
A |
B |
C |
D |
F |
W |
Percent (A,B,C) |
|
15 |
4 |
0 |
0 |
0 |
0 |
2 |
67 |
|
14 |
25 |
2 |
1 |
3 |
2 |
3 |
78 |
|
13 |
22 |
15 |
11 |
5 |
3 |
14 |
69 |
|
12 |
21 |
15 |
19 |
7 |
7 |
16 |
65 |
|
11 |
26 |
21 |
16 |
7 |
7 |
24 |
62 |
|
10 |
14 |
18 |
20 |
8 |
9 |
20 |
58 |
|
9 |
16 |
20 |
21 |
13 |
20 |
30 |
48 |
|
8 |
11 |
14 |
21 |
11 |
16 |
37 |
42 |
|
7 |
3 |
10 |
17 |
14 |
14 |
40 |
31 |
|
6 |
4 |
10 |
17 |
10 |
12 |
19 |
44 |
|
5 |
5 |
4 |
12 |
5 |
8 |
26 |
35 |
|
4 |
3 |
2 |
6 |
4 |
4 |
14 |
33 |
|
3 |
1 |
1 |
5 |
1 |
6 |
10 |
29 |
|
2 |
1 |
1 |
3 |
3 |
2 |
2 |
42 |
|
1 |
1 |
0 |
3 |
0 |
4 |
4 |
33 |
|
0 |
1 |
0 |
0 |
0 |
0 |
0 |
100 |
TABLE 2
KEMTP Test Score |
Percent (A) |
Percent (A,B) |
Percent (A,B,C) |
Number |
|
Greater than 70% |
33 |
51 |
66 |
298 |
|
Between 50% and 70% |
13 |
29 |
49 |
319 |
|
Less than 50% |
6 |
15 |
35 |
312 |
Table 2 summarizes the data in Table 1 in a concise way by dividing the students into three approximately equal groups according to their KEMTP test score; specifically, those 298 students whose scores (11, 12, 13, 14 or 15) were greater than 70%, those 319 students whose scores (8, 9 or 10) were between 50% and 70%, and those 312 students whose scores (0 to 7) were less than 50%. Of those students whose scores were greater than 70%, 33% got an A in the course, 51% got an A or B, and 66% got an A, B or C. Of those students whose scores were between 50% and 70%, 13% got an A in the course, 29% got an A or B, and 49% got an A, B or C. Of those students whose scores were less than 50%, 6% got an A in the course, 15% got an A or B, and 35% got an A, B or C.
3. Fall 2002: Test Results Compared with Final Grades
A total of 1318 students in elementary calculus in the fall of 2002 took a 12-question portion of the KEMTP test as described in section 1 above, although only 789 of these scores were usable. The results are shown in Table 3 and Table 4. The column headed by R represents students who withdrew before the deadline to have a W appear on the transcript. The percentages in Table 3 and Table 4 are calculated so that, for example, Percent (A, B, C) is the percentage of As, Bs, and Cs out of the total number of As, Bs, Cs, Ds, Fs, Ws and Rs.
TABLE 3
|
KEMTP Test Score |
A |
B |
C |
D |
F |
W |
R |
Percent (A,B,C) |
|
12 |
9 |
4 |
2 |
3 |
2 |
2 |
0 |
68 |
|
11 |
30 |
14 |
8 |
6 |
1 |
8 |
4 |
73 |
|
10 |
26 |
17 |
14 |
11 |
2 |
16 |
6 |
57 |
|
9 |
22 |
20 |
14 |
5 |
10 |
22 |
15 |
52 |
|
8 |
9 |
12 |
20 |
12 |
13 |
28 |
16 |
37 |
|
7 |
4 |
17 |
18 |
16 |
13 |
34 |
18 |
26 |
|
6 |
3 |
9 |
11 |
18 |
16 |
37 |
12 |
22 |
|
5 |
3 |
4 |
5 |
8 |
8 |
20 |
17 |
18 |
|
4 |
2 |
2 |
5 |
2 |
4 |
21 |
12 |
19 |
|
3 |
0 |
0 |
1 |
3 |
4 |
15 |
8 |
3 |
|
2 |
0 |
0 |
1 |
0 |
3 |
6 |
4 |
7 |
|
1 |
0 |
0 |
0 |
0 |
1 |
1 |
0 |
0 |
|
0 |
0 |
0 |
0 |
0 |
0 |
0 |
0 |
0 |
TABLE 4
KEMTP Test Score |
Percent (A) |
Percent (A,B) |
Percent (A,B,C) |
Number |
|
Greater than 70% |
30 |
48 |
61 |
293 |
|
Between 50% and 70% |
5 |
16 |
31 |
336 |
|
Less than 50% |
3 |
7 |
14 |
160 |
Table 4 summarizes the data in Table 3 in a concise way by dividing the students into three approximately equal groups according to their KEMTP test score; specifically, those 293 students whose scores (9, 10, 11, or 12) were greater than 70%, those 336 students whose scores (6, 7 or 8) were between 50% and 70%, and those 160 students whose scores (0 to 5) were less than 50%. Of those students whose scores were greater than 70%, 30% got an A in the course, 48% got an A or B, and 61% got an A, B or C. Of those students whose scores were between 50% and 70%, 5% got an A in the course, 16% got an A or B, and 31% got an A, B or C. Of those students whose scores were less than 50%, 3% got an A in the course, 7% got an A or B, and 14% got an A, B or C.
4. Spring 2003: Test Results Compared with Final Grades
A test consisting of 12 questions from the KEMTP test was given to all students in elementary calculus in the spring 2003 as described in section 1, with 405 useable test scores recorded. The results are shown in Table 5 and Table 6, and are to be interpreted in exactly the same way as in Table 3 and Table 4, respectively, in section 3.
TABLE 5
|
KEMTP Test Score |
A |
B |
C |
D |
F |
W |
R |
Percent (A,B,C) |
|
12 |
2 |
2 |
0 |
0 |
1 |
0 |
0 |
80 |
|
11 |
7 |
4 |
4 |
0 |
2 |
5 |
1 |
65 |
|
10 |
10 |
6 |
2 |
5 |
2 |
3 |
0 |
64 |
|
9 |
16 |
5 |
9 |
4 |
4 |
5 |
2 |
67 |
|
8 |
15 |
9 |
8 |
8 |
5 |
12 |
3 |
53 |
|
7 |
9 |
8 |
16 |
8 |
6 |
13 |
9 |
48 |
|
6 |
10 |
11 |
12 |
10 |
13 |
11 |
10 |
43 |
|
5 |
4 |
2 |
12 |
9 |
6 |
10 |
3 |
39 |
|
4 |
0 |
1 |
6 |
3 |
3 |
8 |
2 |
30 |
|
3 |
0 |
2 |
6 |
4 |
0 |
6 |
2 |
40 |
|
2 |
1 |
0 |
0 |
2 |
2 |
2 |
1 |
13 |
|
1 |
0 |
0 |
0 |
0 |
0 |
0 |
1 |
0 |
|
0 |
0 |
0 |
0 |
0 |
0 |
0 |
0 |
0 |
TABLE 6
KEMTP Test Score |
Percent (A) |
Percent (A,B) |
Percent (A,B,C) |
Number |
|
Greater than 70% |
35 |
51 |
66 |
101 |
|
Between 50% and 70% |
17 |
30 |
48 |
206 |
|
Less than 50% |
5 |
10 |
35 |
98 |
IV. Effectiveness of the KEMTP Test in Predicting Final Grades in College Algebra and Calculus
1. The KEMTP Test in College Algebra and Calculus.
The KEMTP test is also an effective predictor of final grades in courses as diverse as college algebra and calculus. All students seeking a bachelor’s degree at UK must take college algebra (MA 109) unless their ACT mathematics scores and other placement information permits them to take a more advanced mathematics course. Calculus (MA 113) is a first course in calculus designed for students majoring in mathematics, science, engineering and computer science. As such, it is a more rigorous course than elementary calculus even though it carries a lower number (MA 113 as compared with MA 123).
2. Test Results Compared with Final Grades in College Algebra.
A test consisting of 12 questions from the KEMTP test was given to all college algebra students in the spring of 2003 as described in section III.1, with 644 useable test scores recorded. The results are shown in Table 7 and Table 8.
TABLE 7
|
KEMTP Test Score |
A |
B |
C |
D |
F |
W |
R |
Percent (A,B,C) |
|
12 |
0 |
0 |
0 |
0 |
0 |
0 |
1 |
0 |
|
11 |
1 |
0 |
0 |
0 |
0 |
0 |
0 |
100 |
|
10 |
3 |
1 |
0 |
0 |
0 |
2 |
0 |
67 |
|
9 |
4 |
4 |
2 |
0 |
0 |
3 |
0 |
78 |
|
8 |
7 |
5 |
2 |
2 |
1 |
0 |
0 |
82 |
|
7 |
12 |
12 |
13 |
7 |
4 |
4 |
1 |
70 |
|
6 |
20 |
20 |
18 |
9 |
13 |
5 |
5 |
64 |
|
5 |
14 |
24 |
22 |
20 |
22 |
12 |
7 |
50 |
|
4 |
15 |
29 |
22 |
16 |
15 |
20 |
4 |
56 |
|
3 |
4 |
19 |
18 |
18 |
30 |
19 |
9 |
35 |
|
2 |
3 |
9 |
18 |
10 |
12 |
11 |
5 |
44 |
|
1 |
1 |
1 |
7 |
7 |
5 |
6 |
2 |
31 |
|
0 |
1 |
0 |
2 |
2 |
0 |
2 |
0 |
43 |
TABLE 8
KEMTP Test Score |
Percent (A) |
Percent (A,B) |
Percent (A,B,C) |
Number |
|
Greater than 70% |
38 |
62 |
71 |
21 |
|
Between 50% and 70% |
24 |
48 |
68 |
160 |
|
Between 30% and 49% |
12 |
34 |
52 |
242 |
|
Less than 30% |
4 |
17 |
38 |
221 |
Table 8 summarizes the data in Table 7 in a concise way by dividing the students into four groups according to their KEMTP test score; specifically, those 21 students whose scores (9, 10, 11, or 12) were greater than 70%, those 160 students whose scores (6, 7 or 8) were between 50% and 70%, those 242 students whose scores (4, 5) were between 30% and 49%, and those 221 students whose scores (0, 1, 2, or 3) were less than 30%. Of those students whose scores were greater than 70%, 38% got an A in the course, 62% got an A or B, and 71% got an A, B or C. Of those students whose scores were between 50% and 70%, 24% got an A in the course, 48% got an A or B, and 68% got an A, B or C. Of those students whose scores were between 30% and 49%, 12% got an A in the course, 34% got an A or B, and 52% got an A, B or C. Of those students whose scores were less than 30%, 4% got an A, 17% got an A or B, and 38% got an A, B or C.
3. Test Results Compared with Final Grades in Calculus.
A test consisting of 18 questions from the KEMTP test was given to all calculus students in the spring of 2002 as described in section III.1, with 156 useable test scores recorded. The results are shown in Table 9 and Table 10.
TABLE 9
|
KEMTP Test Score |
A |
B |
C |
D |
F |
W |
Percent (A,B,C) |
|
18 |
0 |
2 |
3 |
0 |
0 |
0 |
100 |
|
17 |
5 |
2 |
4 |
0 |
1 |
1 |
85 |
|
16 |
4 |
6 |
4 |
2 |
1 |
2 |
74 |
|
15 |
4 |
7 |
5 |
3 |
0 |
0 |
84 |
|
14 |
6 |
7 |
6 |
3 |
3 |
4 |
66 |
|
13 |
2 |
3 |
2 |
2 |
4 |
3 |
44 |
|
12 |
1 |
3 |
4 |
1 |
6 |
5 |
40 |
|
11 |
0 |
2 |
1 |
4 |
3 |
1 |
27 |
|
10 |
0 |
0 |
1 |
1 |
4 |
0 |
17 |
|
9 |
0 |
1 |
0 |
0 |
4 |
1 |
17 |
|
8 |
0 |
0 |
2 |
1 |
0 |
2 |
40 |
|
7 |
0 |
0 |
0 |
0 |
3 |
3 |
0 |
|
6 |
0 |
0 |
0 |
1 |
0 |
0 |
0 |
|
5 |
0 |
0 |
0 |
0 |
0 |
0 |
0 |
|
4 |
0 |
0 |
0 |
0 |
0 |
0 |
0 |
|
3 |
0 |
0 |
0 |
0 |
0 |
0 |
0 |
|
2 |
0 |
0 |
0 |
0 |
0 |
0 |
0 |
|
1 |
0 |
0 |
0 |
0 |
0 |
0 |
0 |
|
0 |
0 |
0 |
0 |
0 |
0 |
0 |
0 |
TABLE 10
KEMTP Test Score |
Percent (A) |
Percent (A,B) |
Percent (A,B,C) |
Number |
|
Greater than 80% |
23 |
54 |
82 |
56 |
|
Between 65% and 79% |
14 |
34 |
52 |
65 |
|
Less than 65% |
0 |
9 |
20 |
35 |
Table 10 summarizes the data in Table 9 in a concise way by dividing the students into three groups according to their KEMTP test score; specifically, those 56 students whose scores (15, 16, 17 or 18) were greater than 80%, those 65 students whose scores (12, 13 or 14) were between 65% and 80%, and those 35 students whose scores (0 to 11) were less than 65%. Of those students whose scores were greater than 80%, 23% got an A in the course, 54% got an A or B, and 82% got an A, B or C. Of those students whose scores were between 65% and 79%, 14% got an A in the course, 34% got an A or B, and 52% got an A, B or C. Of those students whose scores were less than 50%, 0% got an A in the course, 9% got an A or B, and 20% got an A, B or C.
V. Advice Given High School Students Based on their KEMTP Scores
Table 11 gives a brief summary of the advice given to high school students about their current level of mathematical preparation for college based on their KEMTP scores. Students are encouraged to take more mathematics in high school, especially during their senior year, in order to improve their mathematical preparation for college and to insure that their mathematical skills do not decline through disuse during their senior year.
TABLE 11
|
KEMTP Test Score |
Percent Correct |
Advice Given |
|
17-25 |
68%-100% |
Well prepared for entry-level college mathematics course |
|
13-16 |
52%-64% |
Possible need for a remedial course in college |
|
9-12 |
36%-48% |
Probable need for one, possibly two, remedial courses |
|
0-8 |
0%-32% |
Probable need for at least two remedial courses |
In the past, the advice given students was based on similar advice given by early mathematics testing programs in other states, Ohio and North Carolina in particular, whose programs had been operating successfully for many years, together with the best professional judgment of the KEMTP test designers.
The data from UK provides powerful new evidence of the validity of the KEMTP test and the accuracy of the advice given students. The advice given students who are deemed well prepared for college mathematics is especially compelling. High school teachers will be able to show these students that their solid mathematical preparation for college will significantly improve their chances for success in college mathematics courses. These students will be motivated to work all the harder when given such strongly encouraging feedback.
VI. Placement and Advising Issues at UK Involving the KEMTP
The validity of the placement criteria used at UK, based primarily on ACT mathematics scores for incoming freshman students, is confirmed by the KEMTP test scores. Students in college algebra are less prepared in mathematics than students in elementary calculus, and those in elementary calculus are less prepared than those in calculus. For example, 65% (a score of 13 or more out of 18) of all calculus students scored above 70%, 33% (combining the numbers found in Tables 2, 4 and 6) of all elementary calculus students scored above 70%, and only 3% of all college algebra students scored above 70%.
The mathematics department at UK is using this data to improve advising and placement procedures and to increase the success rate in their courses by encouraging students to stay in the courses in which they are properly placed. Too many students who could succeed in a course based on the KEMTP scores are withdrawing. The success rate in the courses as shown in Tables 1 through 10 would of course go up dramatically if those who did not complete the course were not included.
Paul Eakin
Department of Mathematics
University of Kentucky
Lexington, KY 40506
Carl Eberhart
Department of Mathematics
University of Kentucky
Lexington, KY 40506
Steve Newman
Department of Mathematics
Northern Kentucky University
Highland Heights, KY 41099