| Testing
by Objective vs. Testing “Classic”
by Steven B. Just
Most corporate exams are given as criterion-referenced
tests. Based on a set of learning objectives derived from
the learning system, the test creator develops questions that
test each of the learning objectives ensuring (when properly
executed) full domain coverage of the learning materials.
Any given learning objective may be represented by one or
more questions, though typically one to three questions are
written per objective. These tests have a single passing score
set either arbitrarily (“90% seems about right”)
or correctly (via the Angoff method).
Beginning with Version 9.0, in addition to “classic”
criterion-referenced testing, Pedagogue users have an alternate
way to measure subject mastery: Testing by Objective. In an
objectives-based test questions are grouped by learning objective
as they are in a “classic” test. However, mastery
is achieved not by obtaining a single passing score but by
mastering all test objectives (though you can still report
to the student his/her “test score”). What does
it mean to master all objectives? It means that the student
must obtain not one (as in classical testing) but many passing
scores (one for each objective). Can the passing scores for
each objective be different? Yes, each objective can have
its own passing score. What if the student doesn’t master
all objectives? Then on the retest the test developer has
a choice: Test the student again on all the objectives (i.e.
give the entire test again) or only test the student on the
objectives he/she failed to master on the first try. This
process can repeat itself until the student eventually achieves
mastery of all objectives (though, as on a classical test
you may want to limit the number of tries).
Each method of testing has its advantages and
disadvantages. Let’s look at some of them:
- Let’s assume you
have a 100 question “classic” test where the
passing score is 90%. Let’s further assume that those
100 questions are drawn evenly from 10 objectives. This
means it is possible (though perhaps not likely) for a student
to get ALL of the questions wrong in one of the objectives
and still pass the test! It is certainly possible, and perhaps
even likely, that a student scoring at or just about passing
for the test as a whole will have some serious weaknesses
on one or more objectives.
- A typical learning
system will often have dozens, if not hundreds, of objectives.
In order to keep a final, summative exam to a reasonable
length the test creator cannot possibly test on all objectives,
and even then most objectives will of necessity be tested
with only one or two questions. So, if you are testing by
objective, there’s a good chance that good old-fashioned
guessing will enable some students to pass some of those
objectives. (I addressed the issue of too many objectives,
too few questions per objective in this article: A
Critical Look at Learning Objectives and Assessments.)
- Testing by objective
facilitates adaptive learning. Students need only re-study
those materials on which they failed to achieve mastery.
While it is true that on a “classic” test you
can report score by objective, students will be retested
on all the materials so rather than focusing their time
on their areas of weakness only they must spend at least
some time reviewing material they have already mastered.
- Senior management often
prefers “classical” testing because it provides
them with a single metric (the passing score) with which
they can assure minimum competency.
Bottom Line? There are strengths and weaknesses
to both ways of testing. I often recommend using a blended
approach: Use testing by objective for formative module-based
tests and testing “classic” for the final, summative
exam. This enables you to leverage the unique strengths of
both types of testing: (1) to
facilitate an adaptive learning strategy for knowledge acquisition
and (2) to create a single minimum
competency score for high stakes summative exams.
|
|
