[Tlhingan-hol] Certification Test Woes

[d'Armond Speers, Ph.D.]

I have tried to be consistent, fair and permissive in grading of KLCP
tests.  The goal is not to make someone discouraged by grading
hyper-critically, and to allow room for creativity (language is not a 1:1
thing).

For each question in the test bank there is an explanation of the purpose
of the question, what is being tested by the question.  These are printed
in the answer keys with the expected answer(s), which the proctor uses when
grading.  When I'm grading tests, I am specifically looking for mistakes
that pertain to the purpose of the question.  If someone answers the
question correctly but is not strictly adhering to the vocabulary or
grammar of their test level, I do not count that as an error.  (I would not
count off if someone used {-'e'} in an answer in the level 1 test.)  If
there are vocabulary or grammatical errors that do not pertain directly to
the purpose of the question I count off a single point rather than marking
the whole question wrong.  If they completely miss the point of the
question, or the number of errors exceeds the available points per question
(5 points per question for levels 1 & 2), then I just count the question
wrong.  (If someone is making that many errors per question, then they
really aren't at that level yet.)

Whenever I distribute tests to someone else to administer and grade, I
provide these instructions.  With multiple test administrators over a
period of many years, it's easy to see how these guidelines may not be
followed consistently.  I have not personally attended a qep'a' or qepHom
in several years, or administered tests, though I do still regularly
generate new tests from the test banks for qep'a'mey and qepHommey.  It
sounds like we may need to get some written guidelines for test
administrators and strive for greater consistency in grading.

Creation of the KLCP was a group effort.  I designed the structure with
Lawrence (3 levels with 3 pins, because of the cultural importance of the
numeral 3).  There are 100 questions for each level in the test bank (plus
the reading comprehension questions for Level 3), and generating tests
means selecting questions from the test bank at random.  Level 1 is a
sub-set of TKD; level 2 is all of TKD; and level 3 is open-season on all
available materials (mainly including additional materials from KGT).  We
produced written guidelines for each level, describing the vocabulary and
grammar that was in scope.  Questions were written to be distributed evenly
across the topics for each level.  Each question in the test bank was
reviewed by multiple highly-skilled speakers when we were creating the
program, following the written guidelines for the level.  I have all of
this in a MS Access database, which allows me to manage the randomization
and generation of tests and keys quickly and easily.  We put a ton of
thought and effort into it, to make it as fair, correct and relevant as we
could.  And ultimately, it is intended to encourage people to achieve
certification; it is supposed to be a positive experience.

All of that being said, we now have the benefit of many years of
experience, which we didn't have when we were creating the program.  And if
it's not achieving its goals of encouraging learning and rewarding
progress, then we should be open to that feedback and willing to make
adjustments.  The challenge is that the more significant the adjustments
(severely limiting the vocabulary for level 1, for example), the more
effort will be involved in re-developing the test bank.  If we make Level 1
too simple, then the gap between Level 1 and Level 2 becomes huge.  And to
be honest, I would not be able to do this re-development work, so someone
else would have to take it over completely.  Taking out the questions about
the suffix number would be easier, and I agree that it's probably too
pedantic for the level 1 test.  These questions would have to be replaced
with something else, maintaining balance across the topics for the level.
 I'm not sure how many questions there are like this in Level 1, but if
anyone has interest in writing some new replacement questions, I'd be happy
to work with them on improving the test bank.

Another thing to consider is that if we simplify level 1, then people who
worked hard to accomplish that achievement might feel slighted because we
lowered the bar.  Not sure if that's a real thing, it just occurred to me.

Sorry for the wall of text, just lots ideas bouncing around.

-- Holtej
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kli.org/pipermail/tlhingan-hol/attachments/20140330/55af4426/attachment.html>