Tomorrow's Teaching and Learning
The posting below is from the Stanford Report on-line edition of May
25, 2000. It looks at a new Internet grading service currently being
used in college-level logic courses, but which has the potential for
use in other courses across a wide range of disciplines.
UP NEXT: The Case for an Anthropological Pedagogy
Tomorrow's Teaching and Learning
----------------- 1,657 words ---------------
INTERNET GRADING SERVICE REDUCES TEDIUM FOR TEACHERS, STUDENTS
By Kathleen O'Toole
On-Line edition, May 25, 2000
After dinner when he needs a pick-me-up, John Etchemendy often sneaks
off to watch students submit their homework. Tapping into the
Internet from his home computer, the professor of philosophy and
author of logic textbooks and software reaches one of two Sun
workstations named Grade Grinder. He can watch as the wannabe
historians and lawyers taking logic from Professor John Justice at
Randolph-Macon Woman's College and the tech majors taking logic from
Professor Selmer Bringsjord at Rensselaer Polytechnic Institute (RPI)
submit answers to their problem sets.
These students and others taking introductory logic courses around
the country use the Internet to interact with Grade Grinder, a
robotic teaching assistant that doesn't give them answers to
problems, but gives them hints and reminders of principles they have
previously encountered. The robot's advice is personalized to address
the specific shortcomings of the last answer each student has
submitted, and it is delivered by e-mail in seconds. That compares to
the week or more that is typical of feedback from a human grader.
Grade Grinder is an Internet grading service that is provided with
purchase of a new textbook, Language Proof and Logic, and four pieces
of software. Etchemendy co-authored the package with the late Jon
Barwise of the University of Indiana and formerly of Stanford, and a
team of researchers at Stanford's Center for the Study of Language
and Information (CSLI) and Indiana's Visual Inference Laboratory.
Co-published last fall by CSLI Publications and a commercial textbook
house, Seven Bridges Press, the textbook/software package for
introductory logic is priced at $43.95, slightly less than most logic
textbooks alone. There is one catch: Because the purchaser is buying
lifetime tutoring help from Grade Grinder, each educational package
comes with a unique registration number, and the student cannot
resell that ID to another student. But the textbook covers more
ground than most introductory courses, so the student can continue to
receive tutoring from Grade Grinder years after taking a formal
"We are grading about half of the students' exercises live at a
central point, so I get to watch their progress, and that is one of
the most rewarding things," says Etchemendy, who has authored other
textbooks and software but without this interactive component. (He
also chaired the university's Commission on Technology in Teaching
and Learning, which funded proposals for developing learning
technologies on campus.) "First you'll see a student submit an answer
that is wildly incorrect, then get some feedback from Grade Grinder
and keep resubmitting until it's correct. As a textbook author, you
don't usually get that chance to see how students learn from it,
except your own."
When he finds a pattern of trouble, Etchemendy goes into Grade
Grinder's Java-language software and tinkers. For example, he noticed
a pattern last fall of students at California State
University-Northridge having difficulties with word problems that
took the form "neither . . . nor." Many of the students there
acquired English as their second language, and the grammatical
formulation confused them. Seeing the overall pattern allowed
Etchemendy to improve Grade Grinder's advice both to the students and
Etchemendy got into this business by accident, but it has turned out
to fit well with his research on reasoning systems that use multiple
forms of representation. He began building teaching software in the
1980s out of frustration with some of the mistakes students made in
logic courses. Formal logic requires translating English sentences
into a language that lacks the ambiguities and subtleties of natural
languages. "As an example," he says, "I use the old 'Saturday Night
Live' joke that goes, 'Every five minutes, a man is mugged in New
York City. We're going to interview him tonight.'"
The joke is based on what logicians call a "quantifier scope
ambiguity." "People don't even recognize that the English sentence
has this ambiguity because we understand it correctly in context. If
I said to you, 'Every five minutes, a man from the L.A. Times has
been calling,' you would immediately interpret the sentence
differently than you interpreted the one about the man being mugged.
But when students try to learn an unambiguous language, they have
problems because they lack this understanding of English ambiguities."
Grade Grinder uses sophisticated computer algorithms to check such
things as the logical equivalence of the student's sentence with
expected answers, and its truth or falsity in a large number of
contexts. It can check files created using programs packaged with the
textbook -- Tarski's World, Fitch and Boole. It performs this check
much faster and with fewer errors than even an expert human logician,
says Dave Barker-Plummer, a logician and senior research scientist at
Grade Grinder most likely never will replace a human instructor,
Barker-Plummer and Etchemendy say, but it can free instructors and
students of their most tedious teaching and learning tasks.
Instructors still grade about half of the homework. "This isn't
classical distance education, because we think that no amount of
technology can replace an instructor's interactions with students
when they are trying to understand deeper conceptual issues,"
The development team "stumbled on this minimally invasive approach,"
Etchemendy says, and "we also didn't see in advance that part of the
value of Grade Grinder would be allowing us to centrally analyze
About a dozen of the professors who used Language Proof and Logic
this year were polled by e-mail for this article. They gave it high
marks, some saying it was a "revolutionary" development in the use of
technology in the classroom. All who responded said the grading
service made teaching logic easier on them and learning it easier on
"The automated grader worked flawlessly and freed my TA to help with
substantive issues in logic, rather than mechanical checking of
proofs. It's like having another TA -- for free," said Bringsjord,
director of the Minds and Machines Lab at RPI. "I believe this is the
future. It's the start of tutoring agents that handle parts of
teaching traditionally done by humans. My students also loved it."
Justice of the Department of Philosophy at Randolph-Macon agreed,
adding, "With the Grade Grinder always available on the Internet, the
student can know within seconds if she is doing the work correctly.
What's more, she can correct her work before she asks the Grade
Grinder to forward a report to the instructor. This instant feedback
makes learning logic quicker and less frustrating for the student."
Not all the students in the classes of Professor Tom Burke at the
University of South Carolina were pleased, however. "Some students
hate the software precisely because one cannot indulge in shortcuts
or sloppiness that pencil-and-paper homework easily permits," he said.
Burke said his only frustration with the program was "handling the
massive amount of information I get via e-mail on students' progress
through the homework. I wish I could get this information in a form
that could be easily imported into a database."
The CSLI team is addressing that issue. "We're hoping to have web
access to all the data for a particular instructor's course in a
compact form by the end of summer," Etchemendy says.
FOILING CHEATERS, ADAPTING TO OTHER COURSES
Instructors in the past also have been concerned that automated
assessment systems might increase opportunities for student cheating,
Barker-Plummer says. To address that concern, all Grade Grinder
homework has a time stamp that makes it highly unlikely for students
to share their homework. "We've made it difficult enough that any
student who is savvy enough to circumvent the system will probably
find it easier and less time consuming to do the work."
The Grade Grinder software could be adapted for use by other courses,
Etchemendy and Barker-Plummer believe. "Any type of computer file
that a computer can do something sensible to and give useful feedback
could use the Grade Grinder framework," Etchemendy says. "All you
have to do is write a single grading module for that type of file.
Down the road, we could supply chemistry professors with a software
framework in which to plug in their chemistry module, for example."
The grader would be especially useful, Barker-Plummer says, in
scientific courses where the range of possible answers is so large
that it is difficult to tell if a student has found a correct one. "I
can imagine situations in chemistry where you are asked to write down
a formula for a molecule, and there may be hundreds of ways to do it."
The logic course package was sought after by commercial publishers,
but Etchemendy says he felt it would be "unfair if not immoral" to
turn over the rights to a product that was developed with university
resources. "Fortunately, CSLI has a press that publishes academic
books, and we were able to have them publish it, through an
arrangement with another press to do textbook marketing and
distribution." About $2 from the sale of each textbook/software
package is set aside to cover the expected operation and maintenance
costs of the grader.
Dikran Karagueuzian, who directs CSLI Publications, says he views
Language Proof and Logic as a breakthrough in educational technology.
An academic press, CSLI Publications publishes 35 to 40 titles a
year, many of them books in the cognitive sciences and all of which
are reviewed by experts in their fields before acceptance (see
http://csli-publications.stanford.edu/). Karagueuzian has rejected
other courseware proposals because referees found their quality
spotty, he says. "A lot of academic software is like a homemade
motorcycle. You can't take it on the freeway. The programs can be
used on the campus where they were born, but they can't pass the
"In this case, we have a package that doesn't annoy the best students
at the Harvards and Stanfords but which also doesn't alienate
community college students. The authors used lots of concrete
examples that don't go above the users' heads. The ideas sort of
unfurl, and the really nice thing is the instant feedback." SR