Abstract
Two experiments examined the utility of restructured help information as
an aid to new users learning the UNIX commands trn and tin. In both
experiments, the help information was reorganized to reflect the conceptual
model or mental organization inferred from users response to questions, rather
than the dictionary-like organization favored by programmers. A retention test
showed higher scores on questions about the program, and student ratings
indicated a higher opinion of the instruction they received, greater
self-confidence in their knowledge.
This paper concludes that the difficulties new users have with most computer
manuals is the lack of correspondence between the user's schemata or mental
model of the task, and the programmers conceptualization of the task upon which
the manual is usually based
Keywords: mental models, computer manuals, documentation, user training,
online help, educational applications.
Introduction
Any time spent with new users of any program results in a long list of
complaints about the manuals and help available. Despite the wide and growing
use of UNIX, probably no system can compete with UNIX on this dimension.
Systems that match closely the users conceptual structure are described as
intuitive and user friendly. It may be possible to do this same sort of thing
with the reference materials for a system.
Current cognitive theory [7] sees the user as active and inquiring, while
constructing meaning of experience. The more ordered--the better the order
matches one's own--the easier to learn and use, the less frustration. Kieras
and
Bovair [4] found evidence that encouraging users to create a mental model of
how a system works is more useful than simply giving listing of the procedures
to be followed. On-line help needs to provide examples relevant to the user's
questions [2]. The function of the mental model [1] has been found important
to understanding the context [3] and in problem solving environments [6].
From the programmer's viewpoint, it makes sense to arrange information in order
of command or option name. English Dictionaries are organized
orthographically. A thesaurus is organized by relationships among meanings. A
textbook is organized by ideas. A Thesaurus would be a better guide for
organizing a manual. The usual manual is more like a dictionary.
For example, the UNIX program trn has some 200 commands and 27 pages of manual.
It is difficult and confusing for a new user, although informative and helpful
to programmers. The help command within tin produces a page full of briefly
described commands in alphabetical order.
The purpose of the present study is to test the effect a method of
restructuring reference materials has on new UNIX users.
METHODOLOGY
Subjects. Fifty-one students participated in the first study
(learning trn) and 96 in the second (learning tin). About half were
experienced using e-mail and about half had no experience using e-mail. None
reported experience using newsgroups. All students were participating in
classes learning to use newsgroups.
Materials.The organization of the reference materials was
developed with the help of a group of eight informant students being introduced
to trn (or tin) who were asked to construct a list of things "to do" with the
program. For each topic they identified (program control commands, selecting a
newsgroup, selecting a thread, reading an article, responding to an article)
and for each program (tin, trn) a structure of how these users viewed the
program and what it did was developed. This structure reflected the schema or
conceptual model these new users had for the programs.
Procedure.The subjects were instructed and tested in small
groups of 3 to 10 in a computer lab setting. Student subjects were randomly
assigned to a dictionary group or conceptual group as they entered the
laboratory.
All students received a copy of the UNIX help listing for each mode of the
program they were studying. Students in the conceptual group also received a
copy of the help instructions as reorganized to reflect a mental model
previously developed from the responses and interviews of the informant
students.
Students received the instructional material at the beginning of a 2-hour
computer lab where they worked at individual terminals. A TA was available to
answer questions from individual students but he was asked to not comment on
the materials individual students received
Fifteen minutes before the end of the period, students were asked to complete a
test and questionnaire.
Measures.For each program, a ten-item achievement test was
prepared (Examples: How do you mark an article as unread? What options are
available for moving within an article being read?) to meaure knowledge.
Responses were scored on a 0-1 point scale where 1 represents a complete and
correct answer and zero represented no indication of understanding of the
question.
Students were also asked to rate the quality of the reference material they
received (How good was the manual?), the quality of the instruction they
received (How good was the instruction you received?), and their own confidence
in their knowledge of the program (How confident are you that you understand
this program?).
--------------------------------------------------------------------------------
Table 1
Mean scores for each condition
--------------------------------------------------------------------------------
N Know Refer Instr "Know Order ledge ence uction it"
--------------------------------------------------------------------------------
trn
dictionary 25 5.2 4.2 7.4 6.5
conceptual 26 7.1* 5.2 8.2* 7.5*
tin
dictionary 51 4.2 3.7 5.3 5.1
conceptual 45 6.2* 5.4* 7.1* 6.7*
--------------------------------------------------------------------------------
RESULTS
A MANOVA found an overall difference in the performance between the two
instructional conditions (F(1,141)=7.61, p< .01). Following the significant
MANOVA, individual t-tests (indicate by *) were use to test differences on the
individual performance variables with the alpha level set at .01/8.
DISCUSSION AND CONCLUSIONS
I have presented evidence of the usefulness of a simple technique for
finding the conceptual structure and mental model of user, and using that
information for the design of reference and instructional material.
Manual users, instructors, manual writers, need to better understand why the
communication of information about a program is difficult. Basically, the
communication needs to fit with the schema of the task being used by the
user.
These finding are in agreement with Neerincs and de Greef [5] who found that
learning from an 'unstructured' environment proved hard and learning from a
'structured' environment seemed relatively easy.
Why are computer manuals so bad? The answer is in the mind of the
beholder--the user of the manual--and the schema the user has for the task.
Most manuals are written by and for programmers and are structured somewhat
like a dictionary of commands. For that purpose they are fine. But the novice
and intermediate-level user does not have a schema of the task with concepts
and structure in common with the programmer. These users find the dictionary
of commands very difficult to use. They need a manual which better matches
their schema of the task, the perspective they bring to the task. Such models
begin with everyday naive concepts.
Acknowledgments
The author would like to acknowledge the support of the University of
California Committee on Research.
References
1. Elkerton, J. Online aiding for human-computer interface. in Helander,
M. (Ed.) Handbook of Human-Computer Interaction. Elsevier Science,
1988..
2. Kay, D. Using cognitive models of learning in instructional design. In
Proceedings of the Ninth Annual Conference for the Cognitive Science
Society (Seattle, WA, July 16-18), Lawrence Erlbaum Associates, Hillsdale,
NJ 1987, pp.315-324..
3. Kieras, D. E. What mental model should be taught: Choosing instructional
content for complex engineered systems. In J. Psotka, L. Massey, & S.
Mutter (Ed.) Intelligent tutoring systems: Lessons learned. Lawrence
Erlbaum Associates, Hillsdale, NJ, 1988.
4. Kieras, D. E., and Bovair, S., (1984) The role of a mental model in learning
to operate a device. Cognitive Science, 8, 255-273.
5. Neerincs, M. A. and de Greef, H. P. (1993) How to aid non-experts.
INTERCHI'93 Conference Proceedings, pages 165-171.
6. Sebrechts, M. M., Marsh, R. L., & Furstenberg, C. T. Integrative
modeling: Changes in mental models during learning. In S. P. Robertson,
Zachary, W. & Black, J. B. (Eds.). Cognition, computing, and
cooperation. Ablex Publishing Corporation, Norwood, NJ, 1990, 338-398.
7. Zimmerman, Barry J. Models of Self-Regulated Learning and Academic
Achievement. in Zimmerman, Barry J., & Schuck, Dale H. (Ed.)
Self-Regulated Learning and Academic Achievement.
Springer-Verlag: New York, 1989.
