|
|
|
News and events
Testing by Objective
vs. Testing “Classic”
by Steven B. Just
Most corporate exams are given as criterion-referenced
tests. Based on a set of learning objectives derived from
the learning system, the test developer creates questions
that test each of the learning objectives ensuring (when properly
executed) full domain coverage of the learning materials.
Any given learning objective may be represented by one or
more questions, though typically one to three questions are
written per objective. These tests have a single passing score
set either arbitrarily (“90% seems about right”)
or correctly (via the Angoff method).
Beginning with Version 9.0, in addition to
“classic” criterion-referenced testing, Pedagogue
users have an alternate way to measure subject mastery: Testing
by Objective. In an objectives-based test questions are grouped
by learning objective as they are in a “classic”
test. However, mastery is achieved not by obtaining a single
passing score but by mastering all test objectives (though
you can still report to the student his/her “test score”).
What does it mean to master all objectives? It means that
the student must obtain not one (as in classical testing)
but many passing scores (one for each objective). Can the
passing scores for each objective be different? Yes, each
objective can have its own passing score. What if the student
doesn’t master all objectives? Then on the retest the
test developer has a choice: Test the student again on all
the objectives (i.e. give the entire test again) or only test
the student on the objectives he/she failed to master on the
first try. This process can repeat itself until the student
eventually achieves mastery of all objectives (though, as
on a classical test you may want to limit the number of tries).
Each method of testing has its advantages and
disadvantages. Let’s look at some of them:
- Let’s assume you
have a 100 question “classic” test where the
passing score is 90%. Let’s further assume that those
100 questions are drawn evenly from 10 objectives. This
means it is possible (though perhaps not likely) for a student
to get ALL of the questions wrong in one of the objectives
and still pass the test! It is certainly possible, and perhaps
even likely, that a student scoring at or just about passing
for the test as a whole will have some serious weaknesses
on one or more objectives.
- A typical learning
system will often have dozens, if not hundreds, of objectives.
In order to keep a final, summative exam to a reasonable
length the test creator cannot possibly test on all objectives,
and even then most objectives will of necessity be tested
with only one or two questions. So, if you are testing by
objective, there’s a good chance that good old-fashioned
guessing will enable some students to pass some of those
objectives. (I addressed the issue of too many objectives,
too few questions per objective in this article: A
Critical Look at Learning Objectives and Assessments.)
- Testing by objective
facilitates adaptive learning. Students need only re-study
those materials on which they failed to achieve mastery.
While it is true that on a “classic” test you
can report score by objective, students will be retested
on all the materials so rather than focusing their time
on their areas of weakness only they must spend at least
some time reviewing material they have already mastered.
- Senior management often
prefers “classical” testing because it provides
them with a single metric (the passing score) with which
they can assure minimum competency.
Bottom Line? There are strengths and weaknesses
to both ways of testing. I often recommend using a blended
approach: Use testing by objective for formative module-based
tests and testing “classic” for the final, summative
exam. This enables you to leverage the unique strengths of
both types of testing: (1)
to facilitate an adaptive learning strategy for knowledge
acquisition and (2)
to create a single minimum competency score for high stakes
summative exams. |