[Tlhingan-hol] Certification Test Woes

[Robyn Stewart]

I second most of Elizabeth's comments and want to apologize for the subject
line of my e-mail possibly implying that there is a problem in the tests
themselves. I know a lot of work went into them by the right people and I am
proud of the level of commitment that they show of our bronze pach holders.
Thank you d'Armond for writing and supporting the test. I'm not sure if it
was in list e-mail or privately that I said recently that while at first I
wanted to see the test made easier, when I looked at the requirements I
changed my mind, and now simply want to see beginners better prepared so
that more of them take it and more takers pass it.

 

While the suffix number question has been unexpected and tripped people up
in the past, it's not that hard to teach to. I'm sure I won't be the first
person to tell people this:
Remember the sample sentence in TKD, "Due to your apparent minor errors"?
Write it like this:

QaghHommeyHeylIjmo'
             1       2       3    4   5

Now you know the numbers for all the noun suffixes. Similar sentences (just
one would be kind of ridiculous) can be constructed to fix the sequence for
the verb suffixes, and what was once a stumbling block becomes a free point.



I would like to see simpler levels to reward the achievement of people who
are above the pointing and grunting level without the level required for the
claws. I always worry that the people who come to qep'a' knowing nothing and
work really hard for three days, making amazing progress, go away not
knowing the extent of their achievement because they "failed the lowest
test."  I'm going to make sure that those who want to and have advance
preparation time can pass the test, and see that it is made clear at qep'a'
that while this is the lowest level test, it represents a significant
understanding, and not one that can be gained in a long weekend. 

 

I've never heard anyone really complain about the tests, just watched them
be disappointed, and had them admit they were blindsided by the suffix
number question. I would have to stop and count on my fingers to answer
them, myself, but I wouldn't argue that they had no place on the tests.

 

- Qov

 

From: d'Armond Speers, Ph.D. [mailto:speersd at georgetown.edu] 
Sent: March 30, 2014 15:48
To: tlhIngan-Hol List
Subject: Re: [Tlhingan-hol] Certification Test Woes

 

I have tried to be consistent, fair and permissive in grading of KLCP tests.
The goal is not to make someone discouraged by grading hyper-critically, and
to allow room for creativity (language is not a 1:1 thing).

 

For each question in the test bank there is an explanation of the purpose of
the question, what is being tested by the question.  These are printed in
the answer keys with the expected answer(s), which the proctor uses when
grading.  When I'm grading tests, I am specifically looking for mistakes
that pertain to the purpose of the question.  If someone answers the
question correctly but is not strictly adhering to the vocabulary or grammar
of their test level, I do not count that as an error.  (I would not count
off if someone used {-'e'} in an answer in the level 1 test.)  If there are
vocabulary or grammatical errors that do not pertain directly to the purpose
of the question I count off a single point rather than marking the whole
question wrong.  If they completely miss the point of the question, or the
number of errors exceeds the available points per question (5 points per
question for levels 1 & 2), then I just count the question wrong.  (If
someone is making that many errors per question, then they really aren't at
that level yet.)

 

Whenever I distribute tests to someone else to administer and grade, I
provide these instructions.  With multiple test administrators over a period
of many years, it's easy to see how these guidelines may not be followed
consistently.  I have not personally attended a qep'a' or qepHom in several
years, or administered tests, though I do still regularly generate new tests
from the test banks for qep'a'mey and qepHommey.  It sounds like we may need
to get some written guidelines for test administrators and strive for
greater consistency in grading.

 

Creation of the KLCP was a group effort.  I designed the structure with
Lawrence (3 levels with 3 pins, because of the cultural importance of the
numeral 3).  There are 100 questions for each level in the test bank (plus
the reading comprehension questions for Level 3), and generating tests means
selecting questions from the test bank at random.  Level 1 is a sub-set of
TKD; level 2 is all of TKD; and level 3 is open-season on all available
materials (mainly including additional materials from KGT).  We produced
written guidelines for each level, describing the vocabulary and grammar
that was in scope.  Questions were written to be distributed evenly across
the topics for each level.  Each question in the test bank was reviewed by
multiple highly-skilled speakers when we were creating the program,
following the written guidelines for the level.  I have all of this in a MS
Access database, which allows me to manage the randomization and generation
of tests and keys quickly and easily.  We put a ton of thought and effort
into it, to make it as fair, correct and relevant as we could.  And
ultimately, it is intended to encourage people to achieve certification; it
is supposed to be a positive experience.

 

All of that being said, we now have the benefit of many years of experience,
which we didn't have when we were creating the program.  And if it's not
achieving its goals of encouraging learning and rewarding progress, then we
should be open to that feedback and willing to make adjustments.  The
challenge is that the more significant the adjustments (severely limiting
the vocabulary for level 1, for example), the more effort will be involved
in re-developing the test bank.  If we make Level 1 too simple, then the gap
between Level 1 and Level 2 becomes huge.  And to be honest, I would not be
able to do this re-development work, so someone else would have to take it
over completely.  Taking out the questions about the suffix number would be
easier, and I agree that it's probably too pedantic for the level 1 test.
These questions would have to be replaced with something else, maintaining
balance across the topics for the level.  I'm not sure how many questions
there are like this in Level 1, but if anyone has interest in writing some
new replacement questions, I'd be happy to work with them on improving the
test bank.

 

Another thing to consider is that if we simplify level 1, then people who
worked hard to accomplish that achievement might feel slighted because we
lowered the bar.  Not sure if that's a real thing, it just occurred to me.

 

Sorry for the wall of text, just lots ideas bouncing around.

 

-- Holtej

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kli.org/pipermail/tlhingan-hol/attachments/20140330/c1e0f7e5/attachment.html>