W2 Cognitive Task Analysis

by Clayton

The purpose of cognitive task analysis is to determine what knowledge and skill is needed to perform a given task. Its significance for designing an educational gamelet is twofold:

Suppose playing the gamelet requires knowing certain things T and having certain skills S.

(1) If the student/player doesn't know T or have S, they won't be able to play the game.

(2) Playing the game will strengthen the student's grasp of T and S, which could be an educational goal for the game.

Here, to strengthen T or S means that T can be retrieved more quickly and reliably, and that S can be applied more quickly and reliably.

Declarative and procedural knowledge

Note: The treatment of the psychology of learning that we will base our methods on is adapted from the work of John Anderson and colleagues at Carnegie-Mellon University. While almost nothing about psychology is uncontroversial, in my opinion Anderson's work provides the best-supported foundation for analyzing learning processes. It is used by Anderson and others as the design basis for the most successful interactive tutoring systems, and for analyzing some gamelike learning experiences.

Please note that the treatment I'm giving here is a selective simplification of the ideas of Anderson and colleagues. You'll have a chance to sample the real stuff in the reading for this assignment.

It appears that knowledge of facts and possession of skills are embodied in different mental mechanisms. Psychologists call facts, like "the product of 2 and 3 is 6", declarative knowledge. Skills, like being able to recognize a multiplication problem and retrieve or calculate the answer, are called procedural knowledge.

Informally, declarative information can be listed as propositions (subject and predicate) in English, ignoring philosophical issues like whether the proposition that "Walter Scott wrote Ivanhoe" and that "The author of Waverly wrote Ivanhoe" are the same or not (technical treatments can't ignore such matters, and we may not always be able to, but let's cross that bridge if we come to it.)

Procedural knowledge is listed as production rules, each of which has a condition and an action. In a fully technical treatment productions are specified in a formal language and can be executed in a simulation environment. We will be much less formal in our treatment, allowing conditions to refer to things the student can perceive, or to propositions that the student knows, or that are generated by productions. Actions can include physical actions, like pressing a button.

Technically, a proposition that is part of the student's declarative knowledge can't be tested directly by the condition of a production: it must first be retrieved and placed in working memory. Whether this happens or not depends on the strength of the proposition in long-term memory.

Propositions can refer to internal entities called goals. Something like subroutining in procedural knowledge is accomplished by having one production create a proposition describing a goal, which then triggers other productions that test for goal propositions and act on them. Technically goals are processed on a stack; I think we can leave that idea informal, at least for now. Similarly, there are some technicalities regarding communication among goals that I think we can overlook for now.

An example: multiplying two digits, written with an * in between

P1: if I see two digits d1 and d2 separated by * then post goals to multiply d1 and d2, and to put down the answer

[notice that productions can have variables in the condition, whose bindings are available in the action of that production, only.]

P2: if the goal is to multiply d1 and d2, and the product of d1 and d2 is p, then assert that the answer is p

[The variables d1 and d2 in P2 have nothing to do with d1 and d2 in P1: they pick up their binding by being matched against the proposition about the goal ]

[The proposition "the product of d1 and d2 is p" will have to be retrieved from long-term memory. This will only happen if the student knows the relevant number fact, and the strength of their knowledge is enough for the retrieval to work.]

P3: if the goal is to put down the answer then post goals to find where to write the answer and to write the answer there

P4: if the goal is to find a place to write the answer, and I see an = sign with a blank b next to it, then post the goal to write the answer in b

P5: if the goal is to write the answer in b, and b is blank, then write the answer in b

Some things to notice in the example

Productions are not ordered (I've numbered them only for convenience in referring to them). A production can execute whenever its condition is satisfied. The probability that it will execute, and what happens if more than one production could execute, is determined by the strength of the productions. Strength depends on how often the production has been executed in the past, and how well it has worked and won't concern us yet.

There would be many, many different collections of productions that would do the job of these. Finding out what productions a student is actually using takes detailed psychology, based on things like eye movements, details of execution times, and response to variations in problems. Some of the pioneering work of this kind was done using subtraction, where it was found that children exhibit fascinating bugs that can be interpreted as defects in their productions.

Serious work on developing high-quality tutoring systems includes doing these analyses, but we won't try it. Instead, we'll be satisfied with a much cruder inventory. In this example, all we might try to extract from the productions would be that the student will have to be able to recognize the layout and format of the problem (nobody's born knowing that), including recognizing that the required operation is multiplication (and not, say, addition), where the digits are that have to be multiplied, and where the answer should be written. They have to be able to retrieve the relevant number facts, and they have to be able to write the answer.

To illustrate that things at this level really matter, it's the case that success rates on arithmetic problems written horizontally are different from success rates for the same problems written vertically, on standardized tests.

Another observation, related to this standardized test example, that will shape our work quite a bit: the best indications are that the procedural knowledge used in solving one problem is useful in solving another problem only if some of the productions can be applied without change. So we won't want to think of productions as standing for some vague and squishy "understanding", as in "Pat understands how to add". As the test example shows, it happens all the time that Pat can do addition problems presented horizontally but not vertically, or vice versa. That's because the productions needed for the two kinds of problems are not identical, and you can have one and not the other.

This runs counter to our strong intuitions about how knowledge works: we tend to think that when we know something our knowledge is much more general than it really is. But this is just one of the many cases in which our intuitive, "common sense" ideas of ourselves are simply wrong.

Assignment

1. Spend two hours reading Anderson et al at http://act-r.psy.cmu.edu/papers/526/FSQUERY.pdf . Don't expect to understand all of it, but keep at it and get as much from it as you can.

Here are some suggestions for what aspects are most relevant to our aims:

You should feel that there is a powerful psychological theory presented here that is capable to explaining a good deal about human learning even in complex practical situations. The discussion "Putting it All Together", starting on p1045, is probably the best place to get this. There's a lot of detail here that needn't concern you, but try to get the drift

Try to get a sense of the structure conveyed in Figure 1. Here and elsewhere feel free to ignore the aspect of the description that links the mental functions described to brain structures.

Try to relate the discussion in the notes above to the paper. Key places to focus on are "The Goal Module", p1041, "The Declarative Memory Module", p 1042, and "Procedural Memory", p1044. In all of these sections, look especially at the beginning of the discussion, and feel free to ignore the quantitative details.

Turn in a brief (<=10 sentence) summary of what you got from the paper. Append to your summary 2 good questions for class discussion ("good" means that discussing them would be useful given the goals of this course.)

2. Here is a description of a gamelet called "tic tac toe products" (we'll abbreviate it TTTP). I haven't been able to determine who invented it. I learned about it from Andee Rubin, who works on educational software and related issues at TERC, a thinktank in Cambridge, MA. It's a two-player game that works like this..

There's a 6x6 grid, labeled with numbers, like this, with the numbers from 1-9 written in a row underneath:

1	2	3	4	5	6
7	8	9	10	12	14
15	16	18	20	21	24
25	27	28	30	32	35
36	40	42	45	48	49
54	56	63	64	72	81

1 2 3 4 5 6 7 8 9

There are two markers that can be used to pick out two numbers in the row (paper clips are often used.)

The first player puts one of the markers on any desired number in the row. The second player then places the other marker on any desired number in the row, and then puts an O in the square of the grid that contains the product of the two marked numbers (note that both markers can be on the same number.)

The first player then must move one of the markers to any new number, and then puts an X in the square of the grid that contains the product of the two marked numbers.

The second player does the same thing, marking a square with O. Neither player can move the markers in such a way that the product formed has already been used.

The winner is the first player to get 4 of their markers in a row, vertically, horizontally, or diagonally. If a player cannot move the game is a draw.

Do a cognitive task analysis of this game. List the facts a player must know (you don't have to list them all separately if they form an obvious collection that you can describe more economically, as for example "the sum of 2 and 3 is 5, the sum of 3 and 2 is 5, and other addition facts for the digits." List a collection of productions adequate to play the game, in the informal style of the example above.

Your aim should be to address the two applications of cognitive task analysis mentioned at the top of these notes: What does somebody need to know and be able to do to play this game? and What knowledge and skill would be strengthened by playing this game?

You can decide what level of play you want to assume in your analysis. That is, as you'll soon see in playing the game yourself, there is a strategic aspect of play beyond the mechanics, and you'll need to make some assumptions about the strategy the player is using to complete your inventory of required knowledge and skill. I'd suggest analyzing a primitive level of play first, and then trying to analyze a more sophisticated strategy.

You should spend about four hours on this part of the assignment. If you prefer, you can divide your time between doing an analysis of a simple strategy for TTTP and doing an analysis of some other gamelet that interests you.

How to submit your work: Put a text document called W2 in your folder on the Yahoo group site. Don't forget to put your weekly progress report there, too.