View All Pages

Criterion: What it Does and Does Not Do

Broad Overview of Potential Uses and Pitfalls

Caveat: I have not had a chance to use Criterion myself in a class and, by and large, do not teach classes where it features would prove useful. This is based on research and my analysis of the service in the context of my training faculty on using Canvas and my experience teaching writing focused courses as well as courses that have large writing components in hybrid and online formats.

For more detailed analysis see Criterion: Research On Using Criterion In Your Classes

Primary Benefits

The main benefits for using Criterion are to:

  • Ensure students do not submit an un-proofed draft of a writing assignment for instructor grading.
  • Provide rudimentary mechanical feedback on basic technical issues and grammar.
  • Provides pre-packaged essays topics that students can write on and get detailed comparative feedback and exemplar writing examples from ETS's massive database.
  • Faculty can create assignments and grade papers on the system.

Primary Issues

The main issues for using Criterion are:

  • Complexity of set-up and using the system for faculty.
  • Adding another system to use for students - for example using Canvas, Turnitin, and Criterion.
  • Duplication of features with Canvas that may cause confusion.
  • Does not address coherency, argument structure, or topic knowledge.
  • When grading in the system it timestamps when feedback is created.
  • Timed writing feature is "hackable" by students who can defeat controls.

Slideshow: Criterion Dos and Don'ts

Link Links to an external site.

Complexity versus Utility

As with any academic technology the primary issue is the utility and benefits it brings compared with the costs of complexity and actually using the system. Moreover, its ability to bring benefits that outweigh the current systems used plus the efforts in adapting instructional design and faculty workflow. The bottom line is will it reduce workload, improve workflow, and increase student outcomes compared to existing strategies. The answers to this are highly dependent on the context of the course and the teaching techniques and philosophy of the instructor.

Evaluating student writing is a highly complex, and frankly intimate, endeavor with many moving parts that must be balanced against each other. The goal of writing assignments can vary greatly.

The reality is faculty, like students, differentiate between writing and "non-writing" courses - the former being courses where a primary learning objective is evaluating writing. Faculty usually will not provide detailed writing feedback unless it is required because of the time-intensive nature of such work. Moreover, most faculty have little or no training in assessing writing.

Finally, is the technology a sop to try and make flawed systems appear to work correctly? For example placing large number of students in writing classes where any sort of detailed instructor feedback is logistically impossible or in place of grading that should be conducted by the instructor. Is it good enough to be "better than nothing," or is it worse than nothing because it provides the illusion of meaningful assessment?

Existing Alternatives to Criterion

The primary benefit of Criterion is eliminating the first, often low quality, draft submission and providing automated feedback to address basic technical errors. This saves instructor time and frustration. Outside of Criterion, this can be accomplished via Canvas Peer Review. Student peer reviews, providing students are instructed on how to conduct peer reviews and are provided a well-designed rubric will, in my experience, eliminate 80% of basic technical and grammar problems. Peer pressure enhances peer reviews. Moreover, the process itself helps to improve student's writing by providing positive and negative examples and exercises their critical eye in evaluating writing.

Further, the use of multiple submissions and revisions serves to refine work and secondary or tertiary submissions usually require less intensive grading and allow for instructors to shift their focus to more nuanced issues especially in areas that Criterion does not cover, such as coherency and argument structure. Moreover, placing equal value on initial and subsequent instructor graded submissions serves to focus students efforts. Simple series such as:

  1. Peer review submission and review of 2+ other submissions
  2. Instructor submission #1
  3. Instructor submission #2

Potential Criterion Scenarios

In my analysis, there are several potential uses of Criterion that may reach the benefits threshold for faculty use and improve outcomes.

  1. As Dr. Backer noted, first submissions using the system had the duel benefit of producing better quality submissions for the instructor and improving the confidence of students in their submitted work.  In any class that requires longer written assignments, Criterion could provide another layer of writing feedback that either would normally would not occur or to reduce the grading for base technical errors.
  2. Criterion would be particularly useful for large courses where there is little or no detailed writing feedback because of the student/instructor ratio. This is a "better than nothing" scenario.
  3. Using the pre-loaded writing assignments and built in essay comparison function of Criterion, instructors could design writing drills for students to practice their writing. Proof of these drill could be required to submit actual course assignments or for grades/points. This could also be used in a lab setting for directed or self-remediation. For example, in ACCESS.

Faculty Adoption

In my opinion, faculty would need to have a comfort level in using similar systems, such as Canvas or Tunritin, in order to use Criterion. Moreover, technical aptitude is only the initial barrier as use of the system would need to be integrated with existing course design and schedules. Finally, faculty would have to overcome ideological or philosophical objections to the use of this software.

I find it unlikely that many faculty would adopt Criterion on their own, the barriers are too large and many faculty have an innate distrust of these systems. If faculty hesitate to use an LMS, they won't use Criterion.

The solution would be for specific programs to debate and make a decision to use Criterion, likely in a specific course(s), pilot its use and integration into existing instructional design, and provide a template or plug-and-play guide as well as support from a course coordinator.

Critique

Criterion and similar systems, like all software, are ideological. That is, their functionality and design reflect the ideology and philosophy of those who created it. Criterion, as a product of ETS, has a mass production standardization philosophy in approaching education. Writing, unlike mathematics, does not fit well into such a system.

There is no doubt that Criterion, if used properly, could improve some aspects of writing and reduce certain types of grading for faculty. However, systems like these mask structural issues with how higher education is provisioned. Increasing class sizes can only be partially mitigated by technology. Nor can technology overcome the primary impediment to successful teaching - the lack of training in pedagogy and instructional design. Few of the issues that Criterion is designed to solve cannot be addressed by the proper and logical scaling of class sizes based on content and topic, training faculty in instructional design to increase efficiency and manage workflow, and bring in empirically proven effective teaching techniques.  In my opinion, rolling out a systematic training program for faculty would yield much better benefits than simply throwing new technology at them, which most will not use, that mask larger systemic problems.