Introduction to PLOE
The Program Learning Outcome Evaluation (PLOE) advanced data set is designed to provide program-level (or similar) competency achievement data. The learning outcome objects within the learning environment must be created at multiple organizational levels within the Brightspace hierarchy.
This document discusses each of these object types, the level at which they should be created, and the methodology which should be used for their creation. PLOE only works with the Brightspace Competency tool and not the Brightspace Learning Outcomes tool.
Requirements
-
The Competency tool must be turned on.
-
The Learning Outcomes tool must be turned off. Admin Tools menu > click Organization Tools > Availability tab > locate Learning Outcomes in the list and disable it using the slider on the right.
-
Data Hub must be available
-
Give approved roles permission (Can execute PLOE) to see PLOE.
-
Organizational structure must be created to accommodate program level competencies (details below).
-
Program org unit type must be created.
-
Configuration variables must be set (d2l.Tools.AdvancedDataSets.ProgramOrgUnitTypeId(Org) to the program org unit type ID.
Program Level Organization Structure Considerations
The competency objects within the learning environment will need to be created at multiple organizational levels. This documentation will discuss each of these object types, the level at which they should be created and the methodology which should be used for their creation.
Program level org units will be required to be built within the organization structure to provide reporting of learning outcome achievement at that level and to provide segmentation of the achievements of learners between programs.
Program org units can be created manually from the Org Unit Editor from the Admin Tools. The org unit type for Program will need to be added in the Org Unit Type Editor (if not already existing) by adding an org unit with the name “Program” (or similar). Then individual org units of type “Program” can then be created in the org structure. This would be accomplished by using the Org Unit Editor and creating a child org unit under the main organization (“Org”) level of type “Program”.
The children of the program org unit will be the course templates of the course offerings offered within the program. To add the children, navigate to the new program org unit in the Org Unit Editor and select the option to Add a Child. The course templates can be entered by name or by org unit ID (or “ou number”). This ou number is displayed in the left-hand organization tree pane of the Org Unit Editor in parentheses following the org unit name.
Program Level Enrollment
Learners will need to be enrolled into the program org unit associated with the outcome structure in order for their achievements to be reported. This enrollment can happen using different methods. The easiest method is using the Bulk User Management tool from the Admin Tools menu. This tool accepts a CSV file specifying the enrollments to process. Other methods of creating program enrollments include the Brightspace application APIs (“Valence”) or with D2L Standard CSV (v2 or higher). An enrollment API route is available to enroll users into any org unit within the learning environment. Other options could also be available for your specific use case. D2L Standard CSV is often used as part of a system of record (SIS, HRIS or CRM) integration, but can also be used standalone if needed.
Regardless of the enrollment method selected, the learner enrollment into this org unit level is completely invisible to the learner and has no impact on their interactions with the learning environment.
Program Level Objects
Program level competencies need to be created using the Competency tool from the proper program organizational unit. These objects are represented in the illustrations by a blue tipped triangle 
. These competency objects should remain in draft status until the entire structure has been created & reviewed. Once everything has been created and reviewed, then all the competency objects will need to be updated to the approved status.
Competency objects created at the program level will represent overall program outcomes to be achieved by students by the completion of the program. These program outcome objects will have learning objective children from the course template level of each course. The competency objects created at this program level need to be made available (shared) to all its child descendants of type Course Template (see the competency object settings section below).
Review the illustration provided in Appendix B – Competency Structure Diagram for additional examples of construction methodology in the competency structure.
Competency Object Settings
When the competency objects are created, there are some selections on the properties of the competency that should be set.
Figure: Competency object settings.
The “Make competency and its children visible to users” must be checked for the competency and its children learning objectives to be visible to the instructors for linking of learning activities.
The “Allow re-evaluation of users who have achieved this competency” must be checked for the competency achievement to be re-calculated when changes are made to the learning activities or anywhere within the child/parent structure under this competency.
Use the “Make Competency Available To” section to publish or share the competency and its children to the appropriate org unit level. In general, competency objects should only be shared down one level within the organization structure. Competency objects created at the Program level will be shared to the Course Template level. A secondary competency object (called the “Bridge Competency”) will be created at the Course Template level and shared down to the Course Offering level (see the section on Course Template Level Objects below.) Only one Bridge Competency is needed per Course Template.
When selecting the Add Org Units button, a new window will appear. For ease of searching you can use the Show Search Options and filter for the specific org unit type that you wish to select. In this stage of the selection you should locate the current org level type where this competency object is being defined.
Figure: Add Org Units' search parameters.
Once the proper org unit type objects are displayed, select the current org unit (the one in which the current competency object is being defined) and then in the Options column select “All descendants of type: Course Template”.
Figure: Program Add Org Units all descendant of type.
Then select the Insert button to save your selection.
The “Achievement Criteria” field allows the achievement of this competency to be based on successful achievement of “ANY” or “ALL” in the children org units. The default value for this setting is “Any one org unit” and would typically be left on this setting allowing the achievement of this competency to take place from any of the children org units linking to it. A supplemental diagram below demonstrates the ‘ANY’ or ‘ALL’ configuration. If ‘ALL’ is chosen, enrollments into each of the courses were the competency is located will be required to achieve the competency. The diagram below demonstrates the differences between ‘ANY’ and ‘ALL’.
Figure: Competency "Any" Model versus Competency "All" Model.
Course Template Level
The use of consolidated course templates in the learning environment will provide the ability to aggregate the learning outcome achievement across multiple sections of a course and across multiple terms. This can be accomplished if the Course Templates are structured such that all sections of a course are children of one common course template for that course (including any master courses). To fully support this reporting capability, learning objective objects should be created at the course template level to represent the outcomes which will be achieved within the scope of the course regardless of the section (instructor, modality, term, etc.).
Course Template Level Objects
Course template level learning objectives need to be created using the Competency tool from the course template level org unit. These objects are represented in the illustrations by a yellow striped triangle 
. These objects should remain in draft status until the entire structure has been created & reviewed. One learning objective object should be created at this level for each student learning outcome (or knowledge indicator) which will be evaluated within the scope of the course curriculum. The parent of these learning objective objects should be their associated competency objects at the program level. A second parent of each learning objective will be the course level competency object (called the “Bridge Competency”) which will be shared to all of the course offerings (see below).
The bridge competency is created at the course template level. This learning object will be a competency and will be a parent to all of the learning objectives created for this course. This course competency object will be shared to all course offerings under the course template. This course competency provides visibility of the learning objectives at the course template level to all current and future course offerings created under this template. This competency object does not require any parents and will have only children as the learning objectives which are created at the same level.
*Only 1 bridge per course template is needed.
Figure: Course Template for Add Org Units all descendants of type.
Each learning objective object within the course template level will have at a minimum two parents. One parent will be the bridge competency created at the course template level. The second (and any other) parent(s) will be the competency objects shared down from the program level.
When creating learning objective objects, it is recommended that the Enforced Assessment Method setting be left as No. No is the default value for this setting. If this setting is set to Yes and one or more tools are selected, then this learning objective is only available for assessment within the selected tools.
NOTE: This setting cannot be changed. A new learning objective must be created and inserted into the competency structure if a different enforced assessment method is needed.
Figure: Learning Objective Enforced Assessment Method.
Also, when creating new learning objectives, the Ready for Evaluation setting should be set to Yes. The default value for this setting is No. Learning objective achievement is not tracked for learning objectives in the No status. There is no functional benefit to leaving this object set as a “No” status. So, best practice is to set this status to Yes as new objectives are created. This eliminates one update step later in the process (which would be to manually set the status on these objects to Yes).
Figure: Learning Objective Ready for Evaluation.
Course Offering Level Objects
Within the course offering level the assigned learning activities will be created as normal. These learning activity objects are identified in the illustrations with a red based triangle 
. Each activity which has been identified as a critical assessment activity (or one in which successful achievement is required to properly achieve an identified course performance or knowledge indicator) must be associated with the proper learning objective at the course template level. This association is defined on the Objectives tab of the learning activity within the learning environment. Be sure to check the box selecting the activity as required and define the proper achievement threshold.
NOTE: It is important that the rubric evaluation to be used for outcome achievement be associated on the Objectives tab and NOT defined on the Properties tab. The rubric association that can be made on the Properties tab is to support grading only. The rubric association on the Objectives tab will support both outcome achievement and grading.
If your institution uses a “Master Course” construct, then the critical assessment activities within the master course can be associated with the learning objectives. That master course can then be copied into each of the individual course shells for the various course offering sections. The association between the critical assessment activity and the learning objective will be copied into the new course shell as long as the master course is a child of the same template where the course shells are created.
NOTE: Learning Objectives at the course template level must be set as ready for evaluation before they are available to be associated with course offering level learning activities.
NOTE: All learning activities within a course offering which are linked to a learning objective are joined in an AND relationship which means that all activities under a learning objective (within a course offering) must be achieved in order for the parent learning objective to be achieved.
Assignment Activities
Assignment activities can be evaluated using a rubric assessment. A holistic or analytic rubric can be defined for this assessment. A holistic rubric provides a single element evaluation method (similar to a grade only evaluation). An analytic rubric provides a two-dimensional evaluation where multiple criteria can be evaluated at multiple achievement levels. If an analytic rubric is used for this assessment, then the outcome achievement can be associated with the overall rubric performance or with one individual criterion. An achievement threshold must be defined to designate what the success criteria are for satisfactory completion of the indicator being evaluated.
Multiple learning objectives can be associated with an individual learning activity if the activity supports achievement of more than one indicator.
When attaching a rubric to an assignment folder, DO NOT attach the rubric on the properties tab of the assignment unless the rubric is being used for grading only. Rubrics must be associated on the Objectives tab with the learning objective if outcomes achievement data is to be collected. Rubrics associated on the Objectives tab can be used for both grading and outcomes achievement data collection.
NOTE: Create the rubric to be used for evaluation before continuing to link the assignment activity to the learning objective.
To associate the learning objective, select the Objectives tab and then select the Associate Learning Objectives button.
Figure: Associate Learning Objectives.
Then select one or more learning objectives from the list of learning objectives which is displayed.
Figure: Select Learning Objective.
After the learning objective(s) is/are added to the activity, the parameters of the assessment for each learning objective must be defined. These parameters are defined under the Action menu (the drop-down arrow on the right side) next to each learning objective. Select Add Assessment from the action menu.
Figure: Add Assessment.
In the Assessment parameters window, select the rubric to be used for evaluation.
Figure: Define Assessment Parameters.
From the Criteria drop-down list, select whether the overall rubric score or an individual criterion evaluation level should be used to determine achievement of this learning objective.
Figure: Select rubric criterion or overall score.
Check the box which indicates that this activity is required. This checkbox must be selected to collect achievement data for this activity. If this box is not selected, then the outcome achievement data is not collected. An achievement threshold for this activity can only be defined once this checkbox is selected. The achievement threshold value is not displayed until the check box is selected.
Select the minimum level of acceptable achievement from the rubric evaluation levels. Any evaluation at this level of performance or higher will constitute achievement of this activity.
Figure: Select Achievement Threshold.
Quizzing Activities
Quizzes can be associated with the achievement of one or more learning objectives at the course template level. If the Quiz is automatically graded, then the achievement of the learning objective an also be automatically evaluated. Numeric evaluation is available for quiz activities. When the quiz is associated with the learning objective it must be marked as required and a minimum achievement score must be defined.
Quizzes can be associated with learning objectives in the following manners:
• The entire quiz score
• The score on a particular quiz section (including random question sections)
• The score on one or more selected quiz questions
To associate the learning objective, select the Objectives tab and then select the Associate Learning Objectives button.
Figure: Associate Questions or Sections.
In the windows which is displayed, select one or more learning objectives to be evaluated in this quiz. Select the Associate Learning objectives button to add the selected learning objectives to the quiz.
Next, select question, set of questions, section or entire quiz which should be associated with this learning objective.
Figure: Select Questions and/or Sections.
Select Associate Questions to save the selected questions and/or sections.
Then, select the drop-down action menu to define the parameters of the assessment. Select the Add Assessment option to define the parameters of the assessment.
Figure: Add Assessment.
Define the parameters of the assessment. Be sure to check the box to indicate that the activity is required which will display the achievement threshold and trigger collection of the assessment data.
Figure: Define Assessment Parameters.
The achievement perspective on a quiz reflects the percentage (or score) of correct questions out of the selected questions (or total quiz score – depending on how the assessment is configured). So, for example, in a quiz with 20 questions, if 3 questions are selected to be associated one learning objective, each question would be equivalent to a 33 1/3 percentage achievement assuming equal weighing of the questions. The questions are accumulated on a 100 points basis for the selected questions. So, if an achievement threshold is defined as greater than or equal to 75%, for example, a student would have to successfully answer all three questions to achieve a score on those questions which met or exceeded the threshold (2 correct questions would only be 66.67%).
Grade Book and Discussion Activities
Individual grade book items and discussion threads can also be associated with learning objectives from the course template level. When defining this association for a grade book item you can choose whether the evaluation method will be a numeric or a rubric assessment. With enhancements released in the May, 2018 CD Update, grade items with rubrics directly associated in the gradebook can no longer be tied to assessment activities. If you had previously associated rubrics to the grade item for an assessment activity, those rubric associations should be relocated to the individual assessment.
When defining the association for a discussion thread, a rubric evaluation can be used. Enhancements to the rubric evaluation process in discussions have been released in the May, 2018 CD Update. These enhancements now allow a rubric to be associated directly on the Objectives tab of the Discussion Forum. The rubric evaluation will automatically be displayed to the learner in the gradebook and in the User Progress tool.
Rubrics
It is important to consider rubrics in conjunction with the learning outcome assessment implementation as an integrated piece of the project. Rubrics are the assessment tool used to evaluate the successful achievement of the learning outcome and as such are an integral part of the assessment project.
Internal standards should be developed in the construction of analytic rubrics. All rubrics within the organization should be consistently structured either with the first column on the left being the highest evaluation level moving toward the lowest on the right or vice-a-versa. The performance reports will not be able to properly aggregate the achievement results if some of the rubrics are structured in one manner and some are in the other.
BEST PRACTICE: It is recommended that analytic rubrics within a program be constructed using the same number of evaluation levels and using the same terminology to represent those levels. From a student perspective, consistently across evaluation tools is extremely helpful and aids in their understanding of performance expectations.
The criteria being evaluated within the rubrics can vary based on the purpose and use of each rubric. Consistency again is recommended so if faculty within a program can collectively agree on the use of standardized rubrics where possible within a program it will not only aid student comprehension and achievement but also aid performance reporting.
BEST PRACTICE: It is recommended that rubrics be limited to between three and five criteria whenever possible. Increasingly complex rubrics with multiple criteria being evaluated result in increased difficulty for the students to successfully achieve all of the required criteria.
Activation
Once the entire structure has been created, reviewed and approved, all competency objects must be set to “Approved” status before the achievement results can be accumulated and reported.
Retrieving PLOE Data
To retrieve the PLOE advanced data set, you will need to provide the organizational unit code where the parent competency resides. For example, if the program competency is located at the program level 6945, that value will be inputted into the Org Unit ID filter in the PLOE parameter.
Figure: Program Learning Outcome Evaluation in Data Hub.
PLOE Advanced Data Set Configuration
Required Config Variable
d2l.Tools.AdvancedDataSets.ProgramOrgUnitTypeId (set the org unit type ID that represents a program for an org)
Required Permission
Advanced Data Sets > Can execute PLOE (Organization level)
Available Filters
• Org Unit (required)
Note: This is an org unit's unique identification number that appears in the org unit's web address. All org unit IDs are listed in the Org Unit Editor. All descendants of course offerings will be included.
• Achievement Start Date and Achievement End Date
Note: The date range you select includes records based on the achievement date. If start date and end dates are defined, only records within the range are returned. If a start or end date is specified, records with no achievement date are not returned. If no date range is selected, the data set includes all historical rubric assessments, but does not include those with a null date.
• Include all eligible learners that do not meet the competency outcomes criteria to date
Note: When this checkbox is enabled, your results include both learners who have not achieved the required competencies, and learners who have. The checkbox is enabled by default.
Returned Fields
Field Name
|
Description
|
Can be null?
|
| OrgUnitId |
The org unit for which we want to collect results |
No |
| OrgUnitName |
The name of the org unit |
No |
| OrgUnitCode |
The code of the org unit |
No |
| Program |
The program is a custom org unit type, with the filter parameter passed in. There should be one line per program. |
Yes |
| OutcomeId |
Identifier of the competency or learning outcome |
No |
| OutcomeType |
Type of outcome. Options are Competency or Learning Objective. |
No |
| OutcomeName |
Name of the outcome |
No |
| OutcomeStatus |
Status of the outcome
-
Competency status options are Draft, In Review, Approved, or Archived
-
Learning Objective options are Ready for evaluation or Not ready for evaluation
|
No |
| FirstName |
First name of the learner being assessed by the outcome |
No |
| LastName |
Last name of the learner being assessed by the outcome |
No |
| UserName |
Username of the learner being assessed by the outcome |
No
|
| OrgDefinedId |
Identifier of the learner defined by the org unit |
No |
| UserId |
Numeric identifier of the learner |
No |
| RoleName |
Name of the user's role in the org unit |
No |
| RoleId |
Identifier of the user's role in the org unit |
No |
| IsAchieved |
Indicates if the outcome is achieved.
-
1 = Achieved
-
0 = Not achieved
-
Null = Not assessed
|
Yes |
| AchievementDateTime |
Indicates the date and time the achievement was accomplished |
No
|
Figure: Appendix A: Outcome Structure Diagram.
Figure: Appendix B: Competency Structure Diagram.
Figure: Appendix C: Competency Relationship Notes.
Figure: Notes about PLOE.