Criterion FAQ

Source: http://www.ets.org/criterion/faq/ Links to an external site.

Using the Criterion® Service in Teaching

How can the Criterion service help students?

Students get a response to their writing while it is fresh in their minds. They find out immediately how their work compares to a standard and what they should do to improve it. The Criterion® service also provides an environment for writing and revision that students can use independently, 24 hours a day. This environment, coupled with the opportunity for instant feedback, provides the directed writing practice that is so beneficial for students.

How many topics are available?

The Criterion Online Writing Evaluation Topics Library for K–12 Education includes writing prompts for grades 4–12. Criterion essay topics are constructed to elicit writing in various modes that include persuasive, expository, descriptive and narrative. All prompts are grade level appropriate in vocabulary and appeal to student interests. Each topic may be scored on either a 6-point or 4-point scale and the associated rubrics are shown with each prompt.

Currently, there are 61 College Level I topics appropriate for first-year writing courses, practice and placement; 64 College Level II topics appropriate for second-year writing courses and practice; 10 College Preparatory topics; 14 GRE® test topics; and 35 TOEFL® test topics.

Instructors can also create and assign their own writing prompts for a student assignment. Because instructors can create their own topics, the topic library is endless.

The Criterion service library of topics contains assignments representing the following writing genres: persuasive, informative, narrative, expository, issue and argumentative.

Where do Criterion service topics come from?

Criterion topics come from a number of sources, including ETS testing programs such as The Praxis Series™ assessments, the GRE and TOEFL tests, and client programs such as NAEP® and the English Placement Test designed for California State University. Criterion topics have been developed based on representative samples that are mode-specific and that utilize 6-point holistic scales based on widely accepted writing standards.

How does the Criterion service handle an unusual writing style?

The Criterion service looks for specific features of syntax, organization and vocabulary. If the essay under consideration is not sufficiently similar to those in its database of already-scored essays, the Criterion service posts a warning, called an Advisory, saying that it is unable to give an accurate score. Advisories usually result from essays that are too brief or those in which the vocabulary is unusual or the content is off-topic.

Will the use of the Criterion service stifle creative writing among students?

Not necessarily. The Criterion service is designed to be used for evaluating writing done under testing conditions — situations in which even the most creative writers concentrate on "playing it safe" with straightforward and competent writing.

Will the Criterion service catch cheating or plagiarism?

No. The Criterion service simply evaluates the essay. It is up to the institution to ensure that students are working independently and submitting their own work.

Instructors can opt to display a writer's sample for some topics on the "Create Assignment" screen. Students can then view the samples and refer to them while they write their own essays. The sample essays are in a read-only format and cannot be copied and pasted into another document.

What information does the Criterion service report to educators?

Educators have easy and secure access to each student's portfolio of essays, diagnostic reports and scores, as well as summary information on the performance of entire classes.

What information does the Criterion service report to students?

Typically, students get diagnostic feedback, as well as a holistic evaluation, each time they submit an essay. However, educators can block students from seeing their scores — and may choose to do so if they use the Criterion service for benchmarking.

Can instructors limit student feedback?

Yes. Instructors can elect to report all, some or none of the feedback analysis. When creating an assignment, instructors turn the score analysis feature on or off, as well as select which diagnostic feedback to report.

Can instructors limit access to assignments?

Yes, instructors can limit access when selecting assignment options. For example, the date and time an assignment is available are selected by instructors during setup. They can also limit how many times a student can write and revise an assignment.

Can instructors impose time limits on assignments?

Yes. Many assignments available from the Criterion service library of topics have time limits associated with them. When creating the assignment, instructors select whether to impose a time limit, or they can turn off the time-limit function to allow unlimited writing and revision time.

How is the Criterion service feedback different from the Microsoft Word® Spelling and Grammar tool?

The Microsoft® Word Spelling and Grammar tool can provide writers with a quick analysis of common errors.  However, the Criterion service, as an instructional tool used to improve writing, targets more precise feedback. Research shows that the spelling error detection and correction module in the Criterion service has better precision than the spelling error detection and correction module used in MS Word 2007. We continually strive through research and user input to improve the precision of all our feedback categories.

What is the Writer's Handbook?

The Writer's Handbook is an intuitive online tool that a student can access while reviewing diagnostic feedback. It explains every error or feature reported by defining it and providing examples of correct and incorrect use. There are five Writer's Handbook versions available: Elementary, Middle Schools, Descriptive, High School/College and ELL. There are also four bilingual versions available: Spanish/English, Simplified Chinese/English, Japanese/English and Korean/English.

 

Understanding the Technology

What is a Criterion score?

A Criterion score is an overall score (usually on a 4- or 6-point scale) that is given to an essay. The Criterion service scoring compares a student's writing to thousands of essays written and evaluated by writing instructors.

The essays used to build the scoring models have been scored by trained readers and were written by students under timed-testing conditions. The writers had no opportunity to revise, use a spell-checker or reflect on what they had written. So when students write on the Criterion service topics in a regular class, working under more relaxed conditions, instructors and students should recognize that students' scores may not precisely compare to those of the samples.

The Criterion score is a holistic score based on the traits of word choice, convention and fluency/organization. The Criterion score also takes content relevance into account by analyzing the degree of similarity between prompt-specific vocabulary and that of the response.

Does Criterion trait scores?

The trait scores are shown as Developing, Proficient and Advanced. These are based on a "normative range," where the majority (60 percent) of student scores falls. Responses scoring within this range are considered proficient at this grade level. Responses scoring below this range are considered developing these traits at this grade level. Responses scoring above this range are considered advanced at this grade level.

How does the Criterion service come up with its scores?

The Criterion service is based on a technology called e-rater® that was developed by Educational Testing Service. The e-rater scoring engine compares the new essay to samples of essays previously scored by readers, looking for similarities in sentence structure, organization and vocabulary. Essays earning high scores are those with characteristics most similar to the high-scoring essays in the sample group; essays earning low scores share characteristics with low-scoring essays in the sample group.

What is the technology used in the e-rater scoring?

The e-rater scoring engine is an application of Natural Language Processing (NLP), a field of computer technology that uses computational methods to analyze characteristics of text. Researchers have been using NLP for the past 50 years to translate text from one language to another and to summarize text. Internet search engines currently use NLP to retrieve information.

The e-rater scoring engine uses NLP to identify the features of the faculty-scored essays in its sample collection and store them — with their associated weights — in a database. When e-rater evaluates a new essay, it compares its features to those in the database in order to assign a score.

Because the e-rater scoring engine is not doing any actual reading, the validity of its scoring depends on the scoring of the sample essays from which the e-rater database is created.

Can students trick the Criterion service?

Yes. Since the e-rater engine cannot really understand English, it can be fooled by an illogical, but well-written, argument. Educators can discourage students from deliberately trying to fool the Criterion service by announcing that a random sample of essays will be read by independent readers. The Criterion service will also display an Advisory along with the e-rater score when an essay displays certain characteristics that warrant attention compared to other essays scored against the same topic.

Must students be connected to the Internet to use the Criterion service?

Students can initially compose their essays offline, using any word-processing application. However, they will ultimately need an Internet connection to be able to cut and paste their essays into the Criterion essay submission box so their work can be scored and analyzed. For assignments that are timed, essays should be composed online only to ensure accountability by all students and to accurately reflect their writing skills in this environment.

Can I import student identifiers from my data management system?

Yes, the Criterion service has import capabilities for administrators at several levels. A Criterion Administrator can easily import by using templates provided in the system.

Details are provided in both the HELP text and the Criterion® User Manual and Administrator Supplement.

Can I save my data?

Yes, the Criterion service has export features that easily allow users to create export files, and an archive portfolios feature that can be used to create export files in a comma-delimited format (.csv) that can be opened by most text editors and spreadsheet programs. Detailed instructions for both features are provided in the Criterion® User Manual and Administrator Supplement.