Skip to Main Content

Open Educational Resources (OER)

OER Evaluation 

Comprehensive OER Evaluation Tool

Adapted for use in the higher education context, by ISKME, 2017, from Achieve.org’s Rubrics for Evaluating Open Education Objects (OER) Objects, Achieve: CC-BY 3.0 Unported. Users are encouraged to use or modify the rubrics below for their own purposes.

The following rubrics comprise an evaluation system for objects found within Open Educational Resources. An object could include images, activities, assignments, assessments, full courses, and more. For the purpose of this evaluation, any component that can exist as a stand-alone qualifies as an object. The rubrics in this packet can be applied across content areas and object types.

In general, the rubrics should be applied to the smallest meaningful unit. In some cases, this may be a single activity, learning experience, or instructional support material, while in others it might be a complete unit of study or set of support materials. If multiple units are included in an OER, the reviewer needs to determine if all units will be examined, if only those that deal with essential aspects of the curriculum are to be considered, or if it would be best to evaluate random units, looking at, for example, every third or fifth unit.

These rubrics are typically used to rate the potential, not actual, effectiveness of a particular object in a learning environment. Each rubric should be scored independently of the others using the following five scores that describe levels of potential quality, usefulness, or alignment to learning objectives:

  • 3: Superior
  • 2: Strong
  • 1: Limited
  • 0: Very Weak/None
  • N/A: Rubric Not Applicable

The OER evaluation tool is comprised of eight rubrics, including:

  • Rubric I. Degree of Alignment to Learning Objectives
  • Rubric II. Quality of Explanation of the Subject Matter
  • Rubric III. Utility of Materials Designed to Support Teaching
  • Rubric IV. Quality of Assessment
  • Rubric V. Quality of Technological Interactivity
  • Rubric VI. Quality of Instructional and Practice Exercises
  • Rubric VII. Opportunities for Deeper Learning
  • Rubric VIII. Assurance of Accessibility

Rubrics

Rubric I: Degree of Alignment to Learning Objectives

This rubric is applied to learning objects that have suggested alignments to learning objectives. It is used to rate the degree to which an individual object actually aligns to each proposed objective. Before the rubric can be applied, the assumption is that a user has proposed an alignment between the object being reviewed and the selected learning objective(s).

Scale Definition
3 - Superior

An object has superior alignment only if both of the following are true:

  • All of the content and performance expectations in the identified learning objective are completely addressed by the object.
  • The content and performance expectations of the identified learning objective(s) are the focus of the object. While some objects may cover a range of objectives that could potentially be aligned, for a superior alignment the content and performance expectations must not be a peripheral part of the object.
2 - Strong

An object has strong alignment for either one of two reasons:

  • Minor elements of the learning objective(s) are not addressed in the object.
  • The content and performance expectations of the learning objective(s) align to a minor part of the object.
1 - Limited An object has limited alignment if a significant part of the content or performance expectations of the identified learning objective(s) is not addressed in the object, as long as there is fidelity to the part it does cover.
0 - Very Weak

An object has very weak alignment for either one of two reasons:

  • The object does not match the intended learning objective(s).
  • The object matches only to minimally important aspects of the learning objective(s). These objects will not typically be useful for instruction of core concepts and performances covered by the learning objective(s).
N/A

This rubric does not apply for an object that has no suggested learning objectives for alignment.

For example, the rubric might not be applicable to a set of raw data.

Rubric II: Quality of Explanation of Subject Matter

This rubric is applied to objects designed to explain subject matter. It is used to rate how thoroughly the subject matter is explained or otherwise revealed in the object.

Scale

Definition

3 - Superior

An object is rated superior for explanation of subject matter only if all of the following are true:

  • The object provides comprehensive information so effectively that the target audience should be able to understand the subject matter.

  • The object connects important associated concepts within the subject matter. For example, a lesson designed to analyze how an author develops ideas across extended text would make connections among the various developmental steps and the various purposes the author has for the text.

  • The object does not need to be augmented with additional explanation or materials.

  • The main ideas of the subject matter addressed in the object are clearly identified for the learner.

2 - Strong

An object is rated strong for explanation of subject matter if it explains the subject matter in a way that makes skills, procedures, concepts, and/or information understandable. It falls short of superior in that it does not make connections among important associated concepts within the subject matter.

1 - Limited

An object is rated limited for explanation of subject matter if it explains the subject matter correctly but in a limited way. This cursory treatment of the content is not sufficiently developed for a first-time learner of the content. The explanations are not thorough and would likely serve as a review for most learners.

0 - Very Weak

An object is rated very weak or no value for explanation of subject matter if its explanations are confusing or contain errors. There is little likelihood that this object will contribute to understanding.

N/A

This rubric is not applicable (N/A) for an object that is not designed to explain subject matter, for example, a sheet of mathematical formulae or a map. It may be possible to apply the object in some way that aids a learner’s understanding, but that is beyond any obvious or described purpose of the object.

Rubric III: Utility of the Materials Designed to Support Teaching

This rubric is applied to objects designed to support instructors in planning or presenting subject matter. The primary user would be an instructor. This rubric evaluates the potential utility of an object at the intended grade level for the majority of instructors.

Scale

Definition

3 - Superior

An object is rated superior for the utility of materials designed to support teaching only if all of the following are true:

  • The object provides materials that are comprehensive and easy to understand and use.

  • The object includes suggestions for ways to use it with a variety of learners. These suggestions include materials such as “common error analysis tips” and “precursor skills and knowledge” that go beyond the basic elements.

  • All objects and all components are provided and function as intended and described. For example, materials lists are complete, and explanations make sense.

  • For larger objects like courses, materials facilitate the use of a mix of instructional approaches (lecture, group work, research projects, etc.).

2 - Strong

An object is rated strong for the utility of materials designed to support teaching if it offers materials that are comprehensive and easy to understand and use but falls short of “superior” for either one of two reasons:

  • Does not include suggestions for ways to use the materials with a variety of learners.

  • Some core components (e.g., instructions) are underdeveloped in the object.

1 - Limited

An object is rated limited for the utility of materials designed to support teaching if it includes a useful approach or idea to teach an important topic but falls short of “strong” for either one of two reasons:

  • The object is missing important elements (e.g. instructions for some parts are not included).

  • Important elements do not function as they are intended to (e.g. instructions are unclear or practice exercises are missing or inadequate). Instructors would need to supplement this object to use it effectively.

0 - Very Weak

An object is rated very weak or no value for the utility of materials designed to support teaching if it is confusing, contains errors, is missing important elements, or is for some other reason simply not useful, in spite of an intention to be used as a support for instructors in planning or preparation.

N/A

This rubric is not applicable (N/A) for an object that is not designed to support instructors in planning and/or presenting subject matter. It may be possible that an educator could find an application for such an object during a lesson, but that would not be the intended use.

Rubric IV: Quality Assessments

This rubric is applied to those objects designed to determine what a student knows before, during, or after a topic is taught. When many assessment items are included in one object, as is often the case, the rubric is applied to the entire set.

Scale

Definition

3 - Superior

An object is rated superior for the quality of its assessments only if all of the following are true:

  • All of the skills and knowledge assessed align clearly to the content and performance expectations intended, as stated or implied in the object.

  • Nothing is assessed that is not included in the scope of intended material unless it is differentiated as extension material.

  • The most important aspects of the expectations are targeted and are given appropriate weight/attention in the assessment.

  • The assessment modes used in the object require the student to demonstrate proficiency in the intended concept/skill.

  • The level of difficulty is a result of the complexity of the subject-area content and performance expectations and of the degree of cognitive demand, rather than a result of unrelated issues (e.g. overly complex vocabulary used in math word problems).

2 - Strong

An object is rated strong for the quality of its assessments if it assesses all of the content and performance expectations intended, but the assessment modes used do not consistently offer the student opportunities to demonstrate proficiency in the intended concept/skill.

1 - Limited

An object is rated limited for the quality of its assessments if it assesses some of the content or performance expectations intended, as stated or implicit in the object, but omits some important content or performance expectations and/or fails to offer the student opportunities to demonstrate proficiency in the intended content/skills.

0 - Very Weak

An object is rated very weak or no value for the quality of its assessments if its assessments contain significant errors, do not assess important content/skills, are written in a way that is confusing to students, or are unsound for other reasons.

N/A

This rubric is not applicable (N/A) for an object that is not designed to have an assessment component. Even if one might imagine ways an object could be used for assessment purposes, if it is not the intended purpose, not applicable is the appropriate score.

Rubric V: Quality of Teaching Interactivity

This rubric is applied to objects designed with a technology-based interactive component. It is used to rate the degree and quality of the interactivity of that component. “Interactivity” is used broadly to mean that the object responds to the user, in other words, it behaves differently based on what the user does. This is not a rating for technology in general, but for technological interactivity. The rubric does not apply to interaction between students, but rather to how the technology responds to the individual user.

Scale

Definition

3 - Superior

An object, or interactive component of an object, is rated superior for the quality of its technological interactivity only if all of the following are true:

  • The object is responsive to student input in a way that creates an individualized learning experience. This means the object adapts to the user based on what s/he does, or the object allows the user some flexibility or individual control during the learning experience.
  • The interactive element is purposeful and directly related to learning.
  • The object is well-designed and easy to use, encouraging learner use.
  • The object appears to function flawlessly on the intended platform.

2 - Strong

An object, or interactive component of an object, is rated strong for the quality of its technological interactivity if it has an interactive feature that is purposeful and directly related to learning, but does not provide an individualized learning experience. Similarly, to the superior objects, strong interactive objects must be well designed, easy-to-use, and function flawlessly on the intended platform. Some technological elements may not be directly related to the content but for a strong rating they must not detract from the learning experience. These kinds of interactive elements, including earning points or achieving levels for correct answers, might be designed to increase student motivation and to build content understanding by rewarding or entertaining the learner, and may extend the time the user engages with the content.

1 - Limited

An object, or interactive component of an object, is rated limited for the quality of its technological interactivity if its interactive element does not relate to the subject matter and may detract from the learning experience. These kinds of interactive elements may slightly increase motivation but do not provide strong support for understanding the subject matter addressed in the object. It is unlikely that this interactive feature will increase understanding or extend the time a user engages with the content.

0 - Very Weak

An object, or interactive component of an object, is rated very weak or no value for the quality of its technological interactivity if it has interactive features that are poorly conceived and/or executed. The interactive features might fail to operate as intended, distract the user, or unnecessarily take up user time.

N/A

This rubric is not applicable (N/A) for an object that does not have an interactive technological element. For example, the rubric does not apply if interaction with the object is limited to, for example, opening a user-selected PDF

Rubric VI: Quality of Instructional and Practices Exercises

This rubric is applied to objects that contain exercises designed to provide an opportunity to practice and strengthen specific skills and knowledge. The purpose of these exercises is to deepen understanding of subject matter and to routinize foundational skills and procedures. When concepts and skills are introduced, providing a sufficient number of exercises to support skill acquisition is critical. However when integrating skills in complex tasks, the number of exercise problems is less important than their richness. These types of practice opportunities may include as few as one or two instructional exercises designed to provide practice applying specific concepts and/or skills. Sets of practice exercises are treated as a single object, with the rubric applied to an entire group.

Scale

Definition

3 - Superior

An object is rated superior for the quality of its instructional and practice exercises only if all of the following are true:

  • The object offers more exercises than needed for the average student to facilitate mastery of the targeted skills, as stated or implied in the object. For complex tasks, one or two rich practice exercises may be considered more than enough.

  • The exercises are clearly written and supported by accurate answer keys or scoring guidelines as applicable.

  • There are a variety of exercise types and/or the exercises are available in a variety of formats, as appropriate to the targeted concepts and skills. For more complex practice exercises the formats used provide an opportunity for the learner to integrate a variety of skills.

2 - Strong

An object is rated strong for the quality of its instructional and practice exercises if it offers only a sufficient number of well-written exercises to facilitate mastery of targeted skills, which are supported by accurate answer keys or scoring guidelines, but there is little variety of exercise types or formats.

1 - Limited

An object is rated limited for the quality of its instructional and practice exercises if it has some, but too few exercises to facilitate mastery of the targeted skills, is without answer keys, and provides no variation in type or format.

0 - Very Weak

An object is rated very weak or no value for the quality of its instructional and practice exercises if the exercises provided do not facilitate mastery of the targeted skills, contain errors, or are unsound for other reasons.

N/A

This rubric is not applicable (N/A) to an object that does not include opportunities to practice targeted skill

Rubric VII: Opportunities for Deeper Learning

This rubric is applied to objects designed to engage learners in at least one of the following

deeper learning skills, which can be applied across all content areas:

  • Think critically and solve complex problems.

  • Work collaboratively.

  • Communicate effectively.

  • Learn how to learn.

  • Reason abstractly.

  • Construct viable arguments and critique the reasoning of others.

  • Apply discrete knowledge and skills to real-world situations.

  • Construct, use, or analyze models.

 

Scale

Definition

3 - Superior

An object is rated superior for its opportunities for deeper learning only if all of the following are true:

  • At least three of the deeper learning skills from the list identified in this rubric are required in the object.

  • The object offers a range of cognitive demand that is appropriate and supportive of the material.

  • Appropriate scaffolding and direction are provided.

2 - Strong

An object is rated strong for its opportunities for deeper learning if it includes one or two deeper learning skills identified in this rubric. For example, the object might involve a complex problem that requires abstract reasoning skills to reach a solution.

1 - Limited

An object is rated limited for its opportunities for deeper learning if it includes one deeper learning skill identified in the rubric but is missing clear guidance on how to tap into the various aspects of deeper learning. For example, an object might include a provision for learners to collaborate, but the process and product are unclear.

0 - Very Weak

An object is rated very weak for its opportunities for deeper learning if it appears to be designed to provide some of the deeper learning opportunities identified in this rubric, but it is not useful as it is presented. For example, the object might be based on poorly formulated problems and/or unclear directions, making it unlikely that this lesson or activity will lead to skills like critical thinking, abstract reasoning, constructing arguments, or modeling.

N/A

This rubric is not applicable (N/A) to an object that does not appear to be designed to provide the opportunity for deeper learning, even though one might imagine how it could be used to do so.

Rubric VIII: Assurance of Accessibility Standards

This rubric is used to assure materials are accessible to all students. Accessibility is critically important for all learners and should be considered in the design of all online materials. Identification of certain characteristics will assist in determining if materials will be fully accessible for all students. Assurance that materials are compliant with the standards, recommendations, and guidelines specified assists educators in the selection and use of accessible versions of materials that can be used with all students, including those with different kinds of challenges and assistive devices.

The Assurance of Accessibility Standards Rubric does not ask reviewers to make a judgment on the degree of object quality. Instead, it requests that a determination (yes/no) of characteristics be made that, together with assurance of specific accessibility standards, may determine the degree to which the materials are accessible. Only those who feel qualified to make judgments about an object’s accessibility should use this rubric.

 

Scale

Definition

Yes

The object displays the characteristic or complies with the standards, recommendations or guidelines.

No

The object does NOT display the characteristic or comply with the standards, recommendations or guidelines.

Comments

Comments on Rubric 8 Object may include notes that describe the reason materials do not comply with the standard, recommendations or guidelines or further description that may clarify the characteristics of the object.






 

Attribution

Terms of use:

The OER libguide by PGCC Library is licensed under a CC BY-NC-SA 4.0. It is attributed to UMGC, Butler County Community College, D'Arcy Hutchings and the original versions can be found at UMGC, BCCC, and Hutchings.