Computing and Cognition
Talk presented at the Greeno Fest, New York, May 28, 2006 [Note 0]
Clayton Lewis
Department of Computer Science
Institute of Cognitive Science
Coleman Institute for Cognitive Disabilities
Rehabilitation Engineering Research Center for Advancing Cognitive
Technologies
University of Colorado
clayton dot lewis at colorado dot edu
In honor of Professor James G. Greeno, with warmest
gratitude.
When I turn a new corner in my research, I often find Jim Greeno. Jim
has covered a lot of ground in his career. But he has had a consistent
focus on the most important problems. This keeps him out in front of
many of the rest of us, as we bob and weave through the intellectual
landscape. That's why we keep encountering him and his work.
It's been fun for me, and I hope will be fun for Jim, and maybe even
for the rest of you, to review some of these encounters. Because of my
own trajectory through life, the narrative will tend to highlight the
relationship between Jim's ideas and computation.
The first time I encountered Jim was already a surprise.
I was working on the design of a comprehensible computing system at IBM
Watson Research Center, and it had become obvious that none of us had
the least idea what that would mean. We'd start our meetings determined
to talk about comprehensibility, and in five minutes we'd be deep in
discussion about optimizing compilers, something we did know something
about. John Gould at Watson pointed out that cognitive psychology might
be useful. Michigan accepted me, but the question was, who there could
help understand the kind of complex cognition that's involved in
computer use?
Jim sent me a paper, I think Mayer and Greeno 1972 [Note 1] about
learning probability concepts. Intriguingly, they found that the exact
wording of questions had a big influence on people's ability to answer.
How could that be? This seemed like a really interesting problem. The
work also addressed a domain of realistic complexity. I signed up, and
just missed Rich Mayer on his way out of the Perry Building.
Jim's lab had a computer, unusual (and very expensive) then (this is
1973 we're talking about). Dave Kieras, who wishes he could
be
here, had written an operating system [Note
2] for this thing, I believe he
wrote it using a stylus and clay tablets. This system could
make
it run a bunch of subject terminals. It was a great setup for
collecting data from people pressing buttons.
But Jim was already moving on from this kind of data. Newell and
Simon [Note 3], and protocols,
were opening the way to new insights into
thinking. Jim was collecting protocols from students solving math
problems. He made sure his students got a good grounding in these
techniques.
Jim may himself be unaware of the impact this work of his has had in
the field of computer science. When I returned to IBM Watson, and was
charged with developing a research and development strategy for more
usable systems, thinking aloud protocols seemed the most promising
tool. The tool paid off, in the form of a new approach to the
evaluation of user interfaces. The method often let designers
determine not just that users were having trouble
with a system, but why, which is what's needed
for design improvement. Today, thinking
aloud protocols are in everyday use in software development
laboratories around the world.
A key component of usability of computing systems is learnability, and
a key component of learnability is generalizability. At Colorado, my
students and I explored the role of causal analysis in generalization,
a line of inquiry that took us back to the work of Max Wertheimer. Here
too we were following Jim, as this comparison of pictures from my 1988
paper on causal analysis and generalization, and Jim's 1968 book on
theoretical psychology shows [Note 4].
Another line of research on comprehensible computing systems led to
work on programming languages for kids as a knowledge medium. In our
Science Theater/Teatro de Ciencias project grade school children
created animated models of the processes and mechanisms they were
studying in science. In the picture below the colorful disks are
frictions, part of the explanation of lightning created by a trio of
fifth graders.
This work, despite its somewhat idiosyncratic
origin, turned out to follow on lines Jim and his students had
developed in their work on pedagogical uses of computers, as this
picture from Roschelle's Envisionment Machine suggests [Note 5].
At around this time Jim was playing a more and more prominent role in
the development of the situative perspective. While Lucy Suchman and
others were active in developing this view and its implications for
computer systems, I have to confess that I did not get on board.
Indeed, as the issues sharpened and blossomed into controversy, I felt
some personal conflict. I had had the privilege of studying with John
Anderson as well as with Jim, Jim's, and my office mate had been
another controversialist, Lynne Reder. So I had strong intellectual
allegiances on both sides; I'm sure others in this room felt the same.
Incidentally, remembering this controversy brings to mind an
observation about Jim that I know all of us have made: what an
outstandingly NICE person he is. Not many controversies as sharp as the
situationist-cognitivist one close with a joint declaration [Note 6] that
combines sweetness and light, mutual respect, and continued principled
divergence the way that one did. It reflects enormous credit on all the
participants.
Anyway, I certainly wasn't prepared for the next corner around which
I've found Jim, in my recent work on technology for people with
cognitive disabilities. Unexpectedly, situationivity is right there!
The context is this. To develop supports for people with cognitive
disabilities, at least that are more than palliative, we need to
understand the mental mechanisms underlying the disabilities. But this
understanding is elusive.
Our understanding of intelligence in typical people foreshadows the
difficulty. As Ian Deary's review makes clear, cognitive
reductionist approaches to intelligence, starting with Robert
Sternberg's pioneering work, have been disappointing:
Differential
psychologists'
intercourse with cognitive psychology have
been somewhat fruitful in producing interesting results, but not yet
understandings. Individual differences researchers have, with touching
and pious hope, brought back cognitive tasks and processes like
medieval pilgrims amassing the relics of holy persons encased in sealed
caskets. Opening the casket rarely has revealed the relics expected,
and often none at all. Nothing daunted, we have worshipped
the
casket. (Deary [
Note 7])
Where cognitive impairment is concerned the situation is even worse.
Common causes, such as Down Syndrome, can involve literally hundreds of
developmental abnormalities in the brain, some of them quite sweeping.
The challenge is, not just to understand cognitive impairment, but also
to understand how the brain manages to function as well as it does
under these conditions.
Nevertheless, I have followed mainstream thinking in assuming that
cognitive disabilities would have to be
understood, ultimately, in
terms of the structures of individual people's brains, that is, in
cognitivist terms. I had the beginnings of an argument with Graham
Button, the ethnomethodologist, over just this point, after
hearing a talk in which he disparaged the standing of psychology as a
science.
We
cannot emphasize strongly enough the extent to which our arguments
have been designed, not to advance any alternative
'theory of
mind', nor any alternative 'philosophical
psychology' in opposition
to those positions which prevail in contemporary discussions,
but
rather to undermine the very idea that there are genuinely scientific
problems in this domain for which theoretical solutions could
legitimately be sought. (Button et al.,
Computers,
Minds and
Conduct [
Note 8])
In
denying the scientific legitimacy of minds, beliefs, and the whole
intellectual stock of psychology, how could someone like Button have
any story to tell about cognitive disabilities?
My reductio was brought up short by the book, The
Social Construction of Cognitive Disability, by Mark Rapley.
Among many examples, Rapley shows how things said by someone assumed to
have a cognitive disability are interpreted so as to confirm the
diagnosis, while the same things, interpreted without the presumption
of disability, don't suggest disability. Here's
Rapley's manifesto:
Intellectual
disability is usually
thought of as a form of internal,
individual affliction, little different from diabetes, paralysis, or
chronic illness. This study, the first book-length application of
discursive psychology to intellectual disability, shows that what we
usually understand to be an individual problem is actually an
interactional, or social, product [
Note
9].
At about the same time, obituary notices for Urie Bronfenbrenner
referred to his father's experience at an institution for
children
with cognitive disabilities. This quotation points up the power of the
situation to create deficiency:
…[T]he
New York City
courts would commit to our
institution...perfectly normal children. Before he could unwind the
necessary red tape to have them released, it would be too late. After a
few weeks... their scores on the intelligence test... proved them
mentally deficient.
--Urie Bronfenbrenner
(1979)
The Ecology of Human Development
[
Note 10]
One more example: not very long ago children with Down syndrome were
denied reading instruction on the grounds that they could not learn to
read. So, of course, they couldn't learn to read. NOT, as we now know,
because of their attributes as individuals, but because of the pattern
of social interactions they were forced into, or out of.
I am unwilling to follow Button in ruling out any significant role for
individual psychology in cognitive disability. But these examples,
together with the lack of progress so far of cognitivism in this arena,
have drawn my attention anew to situationism, that is, to Jim's ideas.
I'll return to this theme shortly.
The latest turn I want to describe may seem to have nothing to do with
psychology.
Many of us in computer science are confronting a failure to convey the
intellectual interest and importance of our field to the public,
including prospective students. During the dot boom, enrollments in
computer science were overflowing, and the last thing we worried about
was attracting students. As a discipline, we neglected our public face.
We allowed our field to be identified simply with programming.
Introductory courses in computer science, my own included, degenerated
into skills courses. These courses provide hardly a glimpse of the big
ideas in CS, even as these ideas are transforming the world.
So, what are those big ideas?
The best account (in my opinion) identifies computer science as the
science of representations.
This picture shows computer science in the trunk of a tree of
representations.
The representations near the top of tree are particular to the
specifics of various application domains, like biology, for example.
The representations near the bottom are those that link to physical
implementations, whether electronic, photonic, or molecular. The
province of computer science is those representations that are in the
middle, connected above to those tied to applications, and below to
those tied to implementations. Mathematics, and in fact
telecommunications, share the trunk of the tree, because these
disciplines, too, study representations with these same two forms of
independence.
But what are representations?
Besides the chance to work with Jim and John Anderson another prized
legacy for the Michiganders of my generation was an introduction to the
theory of measurement, from Dave Krantz. This theory provides a
framework for understanding representations that I can sketch for you
quickly.
This picture explains how the measurement of length works.
You can stick two rods together, and measure the resulting rod, and
get 5 cm. Or, you can measure the rods separately, getting 3
cm and 2 cm,
and ADD these numbers, getting 5 cm. If you didn't get the
same answer
either way, the measurement system wouldn't be representing length.
This picture shows the perhaps surprising fact that addition
is not uniquely required.
There's a perfectly good system for measuring length in which you
MULTIPLY the individual lengths instead of adding them. If you don't
believe it, the picture shows the kind of ruler you have to use in this
system, at the bottom.
I'll leave it as a problem for the audience whether you can
make subtraction work [Note 11].
This kind of diagram, cleaned up a little,
is central to the branch of mathematics called category theory. In that
theory the focus is on morphisms, shown as arrows in the diagrams.
Morphisms are mappings, or functions, that preserve the structure of
objects that participate in them. Preserving structure is defined
exactly in terms of these diagrams, where two orders of operation,
operate first and then map, or map first and then operate, must give
the same result. This framework is a natural generalization of the
theory of measurement, as was recognized 25 years ago
by Halford and
Wilson in the context of developmental theory [Note
12].
Jim's ideas come into this picture in two ways. Here is a diagram of
Jim's showing the mappings connecting the real world with a
variety of mental representations [Note
13].
Then here are two diagrams from an
exposition by F. William Lawvere, one of the pioneers of category
theory [Note 14].
While the diagrams don't fully convey the thinking, it's closely
related.
Because of this connection, I believe category theory offers a well
developed intellectual setting for some of Jim's ideas. But there is
another, deeper connection.
Category theory was developed by Eilenberg and Mac Lane in the
1940's
[Note 15] as a tool for
managing work in algebraic topology, a field few of us
spend much time thinking about, I venture. But in the 1960's Lawvere,
a student of Eilenberg's, astonished the mathematical world (or that
part that was paying attention, anyway) by demonstrating that category
theory could be used to represent all of set theory with no
reference
to the elements of sets [Note 16]. All
of the necessary apparatus, including
for example quantifiers, can be expressed entirely in terms of
morphisms of objects and their relationships.
Gradually, the recognition has spread that category theory provides a
radically new foundation for logic and the rest of mathematics. To
mention just two examples, category theory naturally gives rise to a
wide variety of unfamiliar logics, in which there are multiple,
qualitatively distinct truth values [Note
17]. Less exotically, category
theory enables a far simpler alternative [Note
18] to Abraham Robinson's nonstandard analysis [Note 19] as the basis for a
treatment of ordinary
calculus in which the infinitesimals that Leibniz used, and that are
still represented in our dy/dx notation, are proper mathematical
objects.
But why do we care?
Consider these quotations from Jim, in the midst of the
situationist controversy.
The
situative perspective adopts a
different primary focus of analysis.
Situativity focuses primarily at the level of interactive systems that
include individuals as participants, interacting with each other and
with material and representational systems (Greeno 1997 [
Note 20]).
...[R]eferential
meanings are
characterized as relations between
situations, rather than as properties of symbolic expressions. In a
conversational interaction, the meaning of the utterance is considered,
not as a property of the utterance itself, but rather as a relation,
called a refers-to relation, between the situation in which someone
makes the utterance and a situation to which the utterance is
interpreted as referring (Greeno 1998 [
Note 21]).
Now consider this one from Barry Mazur in a recent
essay
exploring category theory and its treatment of the natural numbers [Note 22]. The
parallel is striking.
Since
the "compromise" we sketched
above has "mathematical objects
determined by the network of relationships they enjoy with with all the
other objects of their species," perhaps we can go to extremes within
this compromise, by taking the following further step: subjugate the
role of the (it) mathematical object to the role of its network of
relationships-- or, a further extreme-- simply (it)replace the
mathematical object by this network.
The lesson of category theory is that mathematical entities are best
thought of strictly in terms of their relationships to one another, and
not in terms of their individual identity or internal structure. In
fact, thinking of entities as bundles of morphisms gives immediate
uniqueness results for structures like the natural numbers [Note 23]:
The
beauty of this result is that
it has the following decidedly
structuralist, or Wittgensteinian language-game, interpretation: (it)
an object X of a category C is determined (always, up to canonical
isomorphism...) by the network of relationships that the object X has
with all other objects in C.
In contrast, trying to think about what individual numbers
"really are" yields incoherence, not uniqueness.
This point was made in 1965 by Paul Benacerraf in his paper,
"What
numbers could not be," [Note 24]
in which he showed that the existence of many
different, equally adequate, ways to represent numbers as sets means
that numbers cannot be any of these sets. Our
knowledge of numbers is
not of things but of relationships.
Related points have been made over the years by critics of various
accounts of knowledge representation.
Searle's Chinese Room critique of artificial intelligence, to which
Jim has referred in his work [Note 25],
as well as arguments within the
philosophy of mind, such as Putnam's Twin Earth construction [Note 26], also
come to bear.
Efforts to pin down the "meanings" of symbols, or networks,
or
other representations, in isolation, have run into the sand.
As Jim has argued, these problems with attempts to represent meaning in
isolated structures are central inspirations for situationism. Category
theory provides a clear framework not only for understanding these
problems, but also, potentially, for solving them. The approach it
suggests, as also suggested by Jim's diagram that we looked at earlier,
is to focus on identifying and characterizing the mappings among
individual minds, and between individual minds and the world.
The intellectual quest to which Jim has provided such inspiring
leadership has seemed to some critics to carry us away from the clarity
of mathematical and computational models into a tangle of mere
metaphors, vague and wishful. Jim has himself vigorously disputed this,
making clear the continued role he sees for such edged tools as
computational modeling.
But critics do still call for a conceptually clearer framework for this
work. Perhaps this new math, in the good sense, can provide it.
The mathematics concerned takes some getting used to. At
first it
can be as unfamiliar as the world of quantum physics. It lacks the
homely furniture of sets, just as the quantum world lacks the familiar
behaviors of ordinary medium-sized things. But like quantum physics, it
may reveal to us a reality behind our supposed reality. The revealed
reality, the reality of set theory without elements, of relationships
without things, accords remarkably with the situationist picture Jim
and his colleagues are developing for us.
What could be more exciting?
Notes
[Note 0] Many thanks to
Valerie Shalin for organizing the fest.
[Note 1] Mayer, R.E. and
Greeno, J.G. (1972) Structural differences between
learning outcomes produced by different instructional methods.
Journal of Educational Psychology,
63(2), 165.173.
[Note 2] Kieras, D. (1973) A
general experiment programming system for the IBM
1800. Behavioral Research Methods and Instrumentation,
5(2),
235-239.
[Note 3] Newell, A. and Simon,
H.A. (1972) Human Problem Solving. Englewood
Cliffs, NJ: Prentice-Hall.
[Note 4] Greeno, J.G. ( 1968) Elementary
Theoretical Psychology. Reading, MA: Addision-Wesley, fig.
5, p 223.
Lewis, C.H. (1988) Why and how to learn why: Analysis-based
generalization of procedures. Cognitive Science
12, 211-256, figs 9a, 9b, p 252.
[Note 5] Teasley, S. D.,
& Roschelle, J. (1993). Constructing a joint problem space: The
computer as a tool for sharing knowledge. In S. P. Lajoie & S.
J. Derry (Eds.), Computers as Cognitive Tools.
Hillsdale, NJ: Erlbaum, 229-258, fig 1.
[Note 6] Anderson, J.R.,
Greeno, J.G., Reder, L.M., and Simon, H.A. (2000) Perspectives on
learning, thinking, and activity. Educational Researcher,
Vol. 29, No. 4, pp. 11-13
[Note 7] Deary, I. (2000) Looking
Down on Human Intelligence: From Psychometrics to the Brain.
Oxford: Oxford University Press, p 181.
[Note 8] Button, G., Coulter,
J., Lee, J.R.E., and Sharrock, W. (1995) Computers, Minds
and Conduct. Cambridge: Polity Press, p 211.
[Note 9] Rapley, M. (2004) The
Social Construction of Intellectual Disability. Cambridge:
Cambridge University Press, p i.
[Note 10] Bronfenbrenner, U.
(1979) The Ecology of Human Development : Experiments by
Nature and Design. Cambridge, MA: Harvard University Press.
[Note 11] You can't.
Concatenation of rods, in the relevant interpretation here, is
commutative, but subtraction isn't.
[Note 12] Halford, G.S.,
& Wilson, W.H. (1980). A category theory approach to
cognitive development. Cognitive Psychology, 12,
356-411.
[Note 13] Greeno, J.G. (1989)
Situations, mental models, and generative knowledge. In D. Klahr and K.
Kotofsky (Eds.) Complex Information Procession: The Impact
of Herbert A. Simon. Hillsdale, NJ: Erlbaum, 285-318. fig.
11.5, p 307.
[Note 14] Lawvere, F.W. and
Schanuel, S.H. (1997) Conceptual Mathematics: A First
Introduction to Categories. Cambridge: Cambridge University
Press, p 84.
[Note 15] Eilenberg, S.
and and Mac Lane, S. (1945) General theory of natural
equivalences. Transactions of the American Mathematical
Society, 58, 231-294.
Category theory has an interesting history, with slow uptake outside
the upper reaches of mathematics, and in some respects there as well.
But recently two relatively elementary treatments, suitable for
undergraduates, have appeared, by Lawvere and coauthors, cited here.
The paper by Mazur, cited below, is also reasonably accessible. While
not very accessible on technical material, the book by David Corfield
[Corfield, D. (2003) Towards a Philosophy of Real Mathematics.
Cambridge: Cambridge University Press) gives a very interesting
discussion of the status and potential of category theory, and much
else of interest to cognitive scientists.
The World Wide Web seems to be providing a way around the traditional
reticence of many mathematicians in revealing what they are thinking,
rather than just the usually rather sterile distillation of their
thoughts, and in the related matter of communicating with
non-initiates. Corfield maintains an interesting blog;
the paper by Mazur cited below is available from his homepage.
Lawvere's
homepage includes some interesting historical commentary on
the material considered in this talk.
[Note 16] Lawvere, F.W. (1964)
An elementary theory of the category of sets. Proc Natl Acad
Sci U S A. 52(6):1506-11. See also [Note 15].
[Note 17] Lawvere, F.W. and
Rosebrugh, R. (2003) Sets for Mathematics.
Cambridge: Cambridge University Press.
[Note 18] Bell, J.L. (1998) A
Primer of Infinitesimal Analysis. Cambridge: Cambridge
University Press.
[Note 19] Robinson, A. (1974) Non-Standard
Analysis. Amsterdam: North Holland.
[Note 20] Greeno, J.G. (1997)
Response: On claims that answer the wrong questions. Educational
Researcher, 26(1) 5-17, p 7.
[Note 21] Greeno, J.G., and
Middle School Mathematics Through Applications Project Group (1998) The
situativity of knowing, learning, and research. American Psychologist,
53(1), 5-26, p 9.
[Note 22] Mazur, B. (2006)
When is one thing equal to some other thing? Ms downloaded from http://www.math.harvard.edu/~mazur/,
p 6.
[Note 23] Mazur op cit p 18.
In discussion after the talk, Professor Greeno expressed interest in
Mazur's treatment of the role of different representations in
highlighting some aspects of a subject while muting others. Here are
quotations that bring this out:
The
easiest way of comparing the
Peano axioms with the Peano category as modes of defining natural
Numbers , is to ask what each of these formats
*shines a
spotlight on?
*keeps in the shadows?
and
*keeps in the dark?
(p 14)
The
lights are dimmed on mathematical
objects and beamed rather on the corresponding
functors, that is, on the network of relationships entailed by the
objects (p 20).
With
the other lights low, the
mathematical concepts shine out in this new beam, as pinned down by the
web of relations they have with all the other objects of their species.
What has receded is are set theoretic language and logical apparatus.
What is now fully incorporated, center stage under bright lights, is
the curious class of
objects of the category, the template
for the various manners in which a mathematical object of interest
might be presented to us (p 23).
[Note 24] Benacerraf, P.
(1965) What numbers could not be. Philosophical Review,
74, 47-73.
I was led to this paper by the discussion in Makkai, M. ( 1999)
Structuralism in mathematics. In R. Jackendoff, P. Bloom, and K. Wynn
(Eds.) Language, Logic, and Concepts: Essays in Memory of
John Macnamara. Cambridg, MA: MIT Press, 43-66.
There is a discussion of the relationship between Benacerraf's paper
and Lawvere's on the category of sets [Note
16] in the introduction to a reprint of Lawvere's paper by
Colin McLarty:
Lawvere, F.W. (2005) An elementary theory of the category of sets (long
version) with commentary. Reprints in Theory and
Applications of Categories, 11, 1-35. Available at http://tac.mta.ca/tac/reprints/index.html.
[Note 25] Searle, J. R. (1980)
Minds, brains, and programs. Behavioral and Brain Sciences
3 (3), 417-457. See Greeno, op cit. 1989 [Note
13].
[Note 26] Putnam, H.
(1975/1985) The meaning of 'meaning'. In Philosophical
Papers, Vol. 2: Mind, Language and Reality. Cambridge:
Cambridge University Press.