Originally published 19-May-2021
In the June 2021 release (20.21.06), Brightspace Data Hub will include nine new Brightspace Data Sets related to the Learning Outcomes tool. These data sets will allow you to extract data related to the Learning Outcomes tool for use outside of the Brightspace platform (for example, in your Data Warehouse and/or Business Intelligence tools), or for use as data set inputs in Self-Serve Analytics to develop your own reports, if your organization has a Performance Plus package license. With this data, you can answer questions about individual activity assessments related to individual learning outcomes, for students as well as overall levels of achievement for learners.
Within this article, I am consistently referring to both the tool and the objects as Learning Outcomes. However, keep in mind that within your specific instance, that terminology might be set differently according to the value of the config variable d2l.Languages.Terminology.LearningOutcomes which allows you to choose the term with which your organization refers to the tool and these items.
Learning Outcomes data will be available from the June release forward. It will not be possible to retroactively capture data for actions that were already completed prior to the June CD update. There are a few exceptions: Achievement Scales and Program Outcomes contain data that is more static in nature, and Outcomes Details data spans all courses for all outcomes used in your organization. Historical data for these data sets will be available in the July 2021 release. Any changes in Learning Outcomes tool use after the June update will be captured in the daily differential data sets and the weekly full data set. (If your organization has chosen hourly differential data sets then those will also be updated accordingly.). The historical capturing will take place for the following data sets:
- Achievement Scales (Outcomes Scale Definition and Outcomes Scale Level Definition)
- Program Outcomes (Outcomes Program Details and Outcomes in Registries for program registries)
- Outcomes Details (for ASN Imported outcomes only)
The Learning Outcomes data sets will be organized into three groups:
- Learning outcomes items, registries, item/registry relationships and item’s associations with tool objects – the “what” of the learning outcome assessments
- Scales – the “how” of learning outcome assessments
- Demonstrations – the “results” of learner evaluated submissions within the learning outcome assessments
A full breakdown of each new data set will be available after the June release in the Brightspace Core Analytics – Administrator Guide. For consistency, most data sets include CreatedBy, CreatedDate, LastModifiedBy, LastModifiedDate and IsDeleted fields. In some data sets, some of these fields may be null. The context of those fields is discussed in this article and documented in the Administrator Guide.
1. Learning Outcomes
This group of data sets describes the Learning Outcomes objects themselves, their presence in registries, and their alignments to tools for evaluation.
There are five individual data sets in this grouping:
Outcome Details
This data set provides information about the Learning Outcomes objects. Outcome objects are associated with registries at the program level or at the course offering level. In the context of this data set, the CreatedBy and CreatedDate fields are NULL for outcomes which have been imported from ASN. Only locally authored outcomes will have a value in the IsDeleted field. This field value will be NULL for outcomes imported from ASN (the Achievement Standards Network).
When an outcome object is deleted from a course, this details data set will show a value in the IsDeleted field, however all secondary data set references to that object will not show an ‘Is Deleted’ field value unless the object context is specifically deleted. For example, if an outcome is aligned to an assessment activity, has not yet been evaluated and is deleted from the course intent list, this data set will show an IsDeleted value for that object. However, the Outcome Alignments to Tool Objects data setwill not show an IsDeleted value for the assessment activity to which the outcome was aligned. The deletion is implicitly inherited down to child activities, so ensure you check deletion at all levels.
Outcomes Program Details
This data set provides the basic details of each program defined in the Learning Outcomes tool including the Program ID, Name, Creation and Last Modification information. In the context of the Learning Outcomes tool, the creation and definition of a “Program” has no correlation to the org unit structure. All programs are created directly at the Organization level. Currently, all programs and program outcomes are visible and available to all course offerings. Future development will focus on expanding upon this program level functionality.
If a program is deleted (which is only possible when there have been no assessments recorded for any of the outcomes in the program), the record is hard deleted from this data set.
Outcomes Registry Owners
In the Learning Outcomes tool, we are introducing the concept of an ‘outcome registry’. The registry is the list of outcome objects which have been imported or created at either the program level or within the context of a specific course offering. Each course offering (org unit) can own one registry. This registry is often called an ‘Intent List’ in the documentation. It represents the outcomes which are intended to be taught or referenced within the scope of the course offering. A program is its own registry and all the outcomes imported or created into this program are contained within that registry.
This data set provides the details of each defined registry. Course-level registries are automatically created upon the first user login into the course offering org unit by any user, so the CreatedBy field is NULL . Program-level registries have the ID of the user who created the registry in the CreatedBy field.
Something to note in this data set is how deletions are recorded. If a program is deleted (which is only possible when there have been no assessments recorded for any of the outcomes in the program), the associated registry is hard deleted when the program details record is deleted. If a course offering is deleted (sent to the recycle bin), the IsDeleted flag is set to TRUE in the Organizational Units data set. However, the outcomes registry is a secondary data set to the org unit for org unit owned registries so the IsDeleted flag will NOT be represented directly here. Again, since the deletion is implicitly inherited down to child activities, ensure you check deletion at all levels.
Outcomes in Registries
This data set documents which outcomes are contained within which registries (intent lists). This data set can be used to generate a list of outcomes being used in one or more courses within the organization. This data set provides the connection between the registry ID and the outcome object IDs contained within that registry.
Outcomes Aligned to Tool Objects
This data set includes alignments between outcome objects and other assessment items. In this context, assessment items could be rubrics, assignments, discussions, quiz questions or portfolio evidence items. If the outcome is aligned to a rubric, then any assignment or discussion evaluations using that rubric will automatically inherit the alignment to the outcome. This inherited alignment is referred to as an ‘indirect alignment’, whereas specifically adding an outcome to an assignment or to a discussion is referred to as a ‘direct alignment’. Both types of alignments are represented in this data set.
Additional information will be available in the Brightspace Core Analytics - Administrator Guide which provides a list of the object types and object IDs which are referenced in this data set.
2. Learning Outcome Achievement Scales
A scale represents an evaluation framework for proficiency levels to be measured consistently across all assessment activities which are aligned to the same learning outcome. Achievement scales provide a range of achievement levels indicating a learner’s level of proficiency related to a learning outcome. The outcome provides a statement of knowledge, skill or ability that the learner must know to meet the educational outcome requirement whereas the achievement scale provides a measure of proficiency, either at an individual assessment level or aggregated across multiple assessment activities.
There are two individual data sets in this grouping:
Outcomes Scale Definition
This data set provides the name and ID of the scale, creation and maintenance information, and the default indicator flag.
Outcomes Scale Level Definition
This data set provides the details of the scale levels: level names, colors, ranges, etc.
3. Demonstrations
This grouping of data sets provides the details of individual assessment evaluations. This includes both the individual activity evaluations as well as the calculated overall level of achievement or the manually overridden level of achievement from Mastery View. The overall level of achievement is referred to as a ‘checkpoint’ in the data set terminology.
This grouping of data sets includes two individual data sets:
Outcomes Demonstrations
This data set includes both individual activity evaluations and the overall levels of achievement from Mastery View which are referred to as ‘checkpoints’ in these data sets.
For individual assessment evaluations, this data set only includes those evaluations for which feedback has been published (made visible) to the learner. If the evaluation is in draft status, it is not included in this demonstration data set and is also not included in the overall level of achievement calculation in Mastery View.
All checkpoints are included in this demonstrations data set regardless of whether they have been published (made visible) to the learner. In this data set, the IsPublished data field refers to the published (visibility) status of the checkpoint only. The published status of the checkpoint is a completely independent function from the published status of the individual activity evaluation feedback. The published status of the checkpoint can be set at any time. Individual checkpoints for learners for individual outcomes can be published individually or all checkpoints for all learners for all outcomes can be published in bulk from Mastery View at any time in the course.
When a manually overridden overall level of achievement is cleared, that data record is not deleted but the AchievedScaleLevelID field is set to NULL.
Outcomes Assessed Checkpoints
This data set provides the setting details of the calculation methods which had been determined in the Mastery View settings page for the course. These settings values relate to each of the possible calculation method options which are available in Mastery View. The calculated value or manually overridden value of the checkpoint is contained in the Outcomes Demonstrations data set.
4. Questions to be Answered
Our expectation is that the following types of questions could be answered using the Learning Outcomes data sets. These are only suggested questions for example purposes and not intended to limit the use or restrict the use of these data sets in any way.
- On how many assignment evaluations was the suggested level of achievement manually overridden by the evaluator?
- Which learners have or have not achieved X achievement scale level for a specified outcome on a specific assessment activity?
- Which learners have or have not achieved above/below X achievement scale level for a specified outcome on a specific assessment activity?
- Which learners have or have not an overall level of achievement of X for a selected outcome in the course?
- Which learners have or have not an overall level of achievement above/below X for a selected outcome in the course?
- When a manual override is created for a user, for an outcome, who recorded that manual override and when was it recorded?
- When a manual override is cleared for a user, for an outcome, who cleared the manual override and when was it cleared?
- When was a learning outcome aligned/removed from a specific activity and by whom?
- When was an achievement scale created, changed, or set to default and by whom?
- When was a Mastery View calculation method changed and by whom?
- When were overall levels of achievement published to/retracted from learner view and by whom?
The Road Ahead
These data sets will continue to be updated as we enrich the Learning Outcomes tool. We will be developing more functionality related to programs, the relationship between programs and courses, the association between program outcomes and course outcomes, and the data necessary to create curriculum maps. This is an important element of our overall program outcomes initiative. As that work progresses, you will see that data being surfaced in data sets within the Brightspace Data Hub, and in Insights dashboards.