News and events

Testing by Objective vs. Testing “Classic”
by Steven B. Just

Most corporate exams are given as criterion-referenced tests. Based on a set of learning objectives derived from the learning system, the test developer creates questions that test each of the learning objectives ensuring (when properly executed) full domain coverage of the learning materials. Any given learning objective may be represented by one or more questions, though typically one to three questions are written per objective. These tests have a single passing score set either arbitrarily (“90% seems about right”) or correctly (via the Angoff method).

Beginning with Version 9.0, in addition to “classic” criterion-referenced testing, Pedagogue users have an alternate way to measure subject mastery: Testing by Objective. In an objectives-based test questions are grouped by learning objective as they are in a “classic” test. However, mastery is achieved not by obtaining a single passing score but by mastering all test objectives (though you can still report to the student his/her “test score”). What does it mean to master all objectives? It means that the student must obtain not one (as in classical testing) but many passing scores (one for each objective). Can the passing scores for each objective be different? Yes, each objective can have its own passing score. What if the student doesn’t master all objectives? Then on the retest the test developer has a choice: Test the student again on all the objectives (i.e. give the entire test again) or only test the student on the objectives he/she failed to master on the first try. This process can repeat itself until the student eventually achieves mastery of all objectives (though, as on a classical test you may want to limit the number of tries).

Each method of testing has its advantages and disadvantages. Let’s look at some of them:

• Let’s assume you have a 100 question “classic” test where the passing score is 90%. Let’s further assume that those 100 questions are drawn evenly from 10 objectives. This means it is possible (though perhaps not likely) for a student to get ALL of the questions wrong in one of the objectives and still pass the test! It is certainly possible, and perhaps even likely, that a student scoring at or just about passing for the test as a whole will have some serious weaknesses on one or more objectives.
• A typical learning system will often have dozens, if not hundreds, of objectives. In order to keep a final, summative exam to a reasonable length the test creator cannot possibly test on all objectives, and even then most objectives will of necessity be tested with only one or two questions. So, if you are testing by objective, there’s a good chance that good old-fashioned guessing will enable some students to pass some of those objectives. (I addressed the issue of too many objectives, too few questions per objective in this article: A Critical Look at Learning Objectives and Assessments.)
• Testing by objective facilitates adaptive learning. Students need only re-study those materials on which they failed to achieve mastery. While it is true that on a “classic” test you can report score by objective, students will be retested on all the materials so rather than focusing their time on their areas of weakness only they must spend at least some time reviewing material they have already mastered.
• Senior management often prefers “classical” testing because it provides them with a single metric (the passing score) with which they can assure minimum competency.

Bottom Line? There are strengths and weaknesses to both ways of testing. I often recommend using a blended approach: Use testing by objective for formative module-based tests and testing “classic” for the final, summative exam. This enables you to leverage the unique strengths of both types of testing: (1) to facilitate an adaptive learning strategy for knowledge acquisition and (2) to create a single minimum competency score for high stakes summative exams.