Product News Product blogs

Outcomes Assessment: How-To

Outcomes Assessment: Trialing or Using

This guide describes how to enable the Learning Outcomes features of Brightspace Platform trial or production courses.

In Brightspace Platform December 2018/10.8.8 release (10.8.8), we introduced the first features for Learning Outcomes. The guide was updated for Brightspace Platform September 2019/20.19.9 to reflect the feature changes that have been added since this guide was originally written. It was further updated to reflect updates in January 2020/20.20.1 and February 2020/20.20.2 to introduce additional Rubrics and Learning Outcomes workflow and functional changes.

1.0 Setup and Configuration

1.1 Config Variables

Turn on Outcomes Assessment: The d2l.Tools.LearningOutcomes.ToolIntegrationEnabled configuration variable enables outcomes throughout the assessment tools. It is ON by default.

Choose your preferred terminology: The ?d2l.Languages.Terminology.LearningOutcomes configuration variable enables you to choose the outcomes terminology used in the Brightspace Learning Environment. Set a term that is close to the terminology used in your institution. Options include Standards, Expectations, Competencies, Objectives, or Outcomes. Standards is the default and will be the language used for the remainder of this guide.

Enable or disable suggested levels of achievement based on rubric assessment: The d2l.Tools.LearningOutcomes.LoaSuggestions configuration variable provides suggested assessments for standards in an activity when the standard is aligned to multiple criteria in a points-based rubric, and based on the points and percentage thresholds you have set up. By default, this configuration variable is Enabled. To turn off the suggestions, change the setting to Disabled.

Enable the new quiz builder experience: The d2l.Tools.Quizzes.NewQuizBuilderExperience configuration variable ensures that the new quiz builder is turned on so that you can align learning outcomes to quiz questions. Set the configuration variable to ON (Opt In) or ON for the org unit(s) where you want to use the new experience. This can be set for the entire org or individual org units.

1.2 Enable the Learning Outcomes Tool

You can enable the Learning Outcomes tool from the Organization Tools list on the Admin Tools menu.

On the Availability tab, find Learning Outcomes on the tool list and ensure it is turned on.
On the Org Defaults tab, set the default tool status for org units that you want to use the Learning Outcomes tool.

NOTE: If you are piloting Learning Outcomes in your organization, you can set the default to OFF, in which case individual pilot courses would need to select the tool on within their courses. If you are broadly adopting the tool across your organization, then you can set the default status to ON. Defaulting the tool on within a course means that the tool is visible on the Course Admin page, but does not automatically mean that the tool is being used within the course.

1.3 Permissions

Ensure that all roles that will set up and use Learning Outcomes has the appropriate permissions:

Learning Outcomes > Manage Programs
Learning Outcomes > Manage Outcomes. This allows a user to import standards into a course.
Content > Manage Content. This allows a user to import standards using the New Content and if granted at the org level, will allow a user to see the org-level Standards tool to set up achievement scales Experience.

To set these permissions:

On the Administrator home page, click Admin Tools.
Under Security, click Roles and Permissions.
Select a user role that will be using Learning Outcomes.
Under Filter by Tool do the following:
- Click Learning Outcomes and assign the Manage Programs and Manage Outcomes permissions.
- Click Content and assign the Manage Content permission.

Note: Instructor permissions are required in the each of the individual tools to align standards to individual assessments.

2.0 Create an achievement scale

The achievement scale defines a learner’s performance. Levels may have labels such as Beginning, Meeting Expectations, Exceeding Expectations, and Proficient. You likely already have an internal policy for these terms. In this document, we’ll use just two levels: Not yet and Got it.

Using a role that has the Manage Content permission at the org level, click Admin Tools.

Figure: The Admin Tools icon on the navbar.

Under Organization Related, click Standards. Note: The link name depends on the language term you have defined in d2l.Languages.Terminology.LearningOutcomes.

Figure: The list of Admin Tools with Standards highlighted.

On the Standards page, click Create Scale.

Figure: The Create Scale button on the Standards page.

On the New Scale page, enter a scale name and number of levels. Enter a color indicator and name for each level of achievement. If you want suggestions for standards assessment based on % of points achieved on points-based rubrics, enter percentage thresholds for each level.

Figure: The New Scale dialog box.

Click Save and Close.

On the Standards page, click the context menu of the scale you just created and click Set as Default.

Figure: The Set as Default option on the Standards page.

The default scale is applied to every orgunit in your organization.

Verify the scale and click Set as Default.

Note: Starting in February 2020/20.20.2, with the alignment of outcomes to quiz questions, achievement scale suggestions in quiz questions require the use of a percentage-based scale. If a non-percentage-based scale is your default, then the achievement scale levels must be manually set by the instructor for each student’s quiz attempt.

Next Steps: After your scale is ready to be used, add some standards to your course.

3.0 Create a Program and add standards

In the Standards tool, Programs provide the opportunity to create higher level learning outcomes which apply to multiple courses and can be imported into a course. Programs are optional to use; however, if you are using them in your organization, this is the point when you should add them.

At a high level, you create a program and then add standards that will apply to the whole program. Standards can be added from ASN (Achievements Standards Network), or you can manually create custom standards.

If you are not already in the Standards tool. Using a role that has the Manage Content permission at the org level, click Admin Tools. Under Organization Related, click Standards. Note: The link name depends on the language term you have defined in d2l.Languages.Terminology.LearningOutcomes.

Click Programs.

Figure: The second tab of the Standards tool, used for creating Program standards.

Click Create Program, then enter a program name.

Click Create.

Click the name of the Program you just created.

Do one of the following:

Click Add Standard to manually enter a program-level standard, then add a standard description and click Save.

Figure: The Add Standard button, used to add custom standards to the Program standards list.

Click Import Standards to import a standard from ASN. Select filters for Jurisdiction, Subject, Framework, and Education Level as appropriate. Select a standard from the Available Standards list, then click Import Selected Standards.

The Import Standard button, used to import standards from ASN to the Program standards list.

Figure: The Import Standards button, used to import standards from ASN to the Program standards list.

If you want to add a lower-level standard below the one you just added, delete the standard, or edit the standard you just added, click the More Actions button beside the Standard you added.

Figure: More Actions button, expanded to show Add lower-level standard, Delete standard, and Edit options.

Figure: A screenshot of a social media post Description automatically generated.

4.0 Import standards into your course

Standards can be imported using the New Content Experience (Lessons) or using Standards.

For instructions on how to import standards using the Standard tool, see 4.1 Import Standards using the New Content Experience.
For instructions on how to import standards using the New Content Experience (Lessons), see 4.2 Import Standards using the Standards Tool in Course Administration.

Note: If standards for your jurisdiction, subject, and framework are not in the Achievement Standards Network (ASN), select ones that seem most appropriate for testing. For Higher-Ed clients, the most applicable standards for your purposes might be under the Engineers Canada jurisdiction. If you want to bring your standards into the ASN, visit the Achievement Standards Network website.

4.1 Import standards using the New Content Experience

Prerequisite: To import Standards using the New Content Experience (Lessons), ensure that the Learning Outcomes tool is turned on in Organization Tools, and your role has the Manage Outcomes permission at the organization and course levels

In the New Content Experience (Lessons), click the Standards Coverage gauge.

Figure: The Standards Coverage gauge, available above the course content list in the New Content Experience.

On the Standards Coverage dialog, click Add or Remove Standards.

Do one of the following:

Click Add Standard to manually create a new standard, then add a standard description and click Save.
Click Import from Program to import a standard from the Program standards. Select the program and the standard from the available options, then click Import Selected Standards.
Click Import from Achievement Standards Network to import a standard from ASN. Select filters for Jurisdiction, Subject, Framework, and Education Level as appropriate. Select a standard from the Available Standards list, then click Import Selected Standards.

Figure: Buttons available to manually add a standard or import standards from either Program-level standards or the ASN.

If you want to add more standards beneath the one you just created, click the More Actions menu beside the standard and select Add Lower Level Standard.

Once you have added or imported the standards, they appear in a list in the Standards Coverage dialog.

Figure: List of standards imported or added to the course.

4.2 Import standards using the Standards tool in Course Administration

From within a course, using a role that has the Manage Content permission at the org level, click Course Admin on the course navbar.

Under Assessment, click Standards. Note: The link name depends on the language term you have defined in d2l.Languages.Terminology.LearningOutcomes.

Figure: The Assessment group in the Course Admin options, with Standards highlighted.

Do one of the following:

Click Add Standard to manually create a new standard, then add a standard description and click Save.
Click Import from Program to import a standard from the Program standards. Select the program and the standard from the available options, then click Import Selected Standards.
Click Import from Achievement Standards Network to import a standard from ASN. Select filters for Jurisdiction, Subject, Framework, and Education Level as appropriate. Select a standard from the Available Standards list, then click Import Selected Standards.

Figure: Displays buttons used to add a custom standard or import from the Program or ASN.

If you want to add more standards beneath the one you just created, click the More Actions menu beside the standard and select Add Lower Level Standard.

5.0 Align standards to assessments

The following table shows the tools in the Brightspace Learning Environment where you can use Rubrics and Learning Outcomes:

Tool	Can align Standards	Can assess Standards
Assignments	Yes	Yes
Discussions	Yes	Yes
Brightspace Portfolio	Yes	Yes
Rubrics	Yes – to rubric criteria	Yes – to rubric criteria
Quiz Questions	Yes	Yes
Quizzes	Scheduled for future release	Scheduled for future release
Grade Items	No	Yes – using a rubric aligned with standards

Note: Using the New Content Experience, you can share rubrics from higher organization levels.

5.1 Align standards to rubric criteria

Note: Aligning standards to rubric criteria is only available using the new Rubrics experience.

You can align standards to rubric criteria during the creation process, or you can edit an existing rubric to add alignment standards.

Do one of the following:

Create a new rubric and add criteria.
Edit an existing rubric that contains criteria.

On the criterion where you want to align a standard, click Standards.

Figure: The Standards button is available for each criterion on a rubric.

On the Browse Standards dialog, select the standards you want to align and click Add.

Figure: Browse Standards enables users to select Standards to align to rubric criteria.

The added standards appear below the criterion, tagged with the information about the standard. The rubrics changes are automatically saved.

To see the full text of the standard, hover over the standard tag.

Figure: The Standards appear for each Criterion line, and the full text appears as hover text.

Figure: The Standards Coverage dialog.

5.2 Map Achievement Scale to Rubric Criteria

In the following scenarios, achievement level markers automatically appear. You can manually adjust markers if appropriate:

The achievement scale is not mapped to percentage ranges and the rubric is text-only.

The achievement scale is not mapped to percentage ranges and the rubric is points-based or custom points-based

The achievement scale is mapped to percentage ranges and the rubric is text-only. In this case, the Manually map achievement levels to rubric levels is automatically selected under the Learning Outcomes options, but the option cannot be changes.

In the following scenario, achievement level markers automatically appear, but achievement suggestions are determined based on actual learner scores:

The achievement scale is mapped to percentage ranges and the rubric has points-based or custom points-based scoring. Alignment mapping is not necessary and achievement markers are not displayed. You can manually adjust the calculated achievement levels.

Note: In this case, Automatically map achievement levels to percentage scores is selected; however, you can change that to Manually map achievement levels to rubric levels. If you enable the Manually map achievement levels to rubric levels, the achievement markers appear.

Figure: The Learning Outcomes options for rubrics define whether you want to automatically map achievement levels.

5.3 Align standards to an assignment

Note: The Standards tab might be labeled as Competencies, Expectations, Objectives, or Outcomes, depending on the term you have specified in the d2l.Languages.Terminology.LearningOutcomes configuration variable.

Create a new assignment or edit an existing assignment.

Select the Standards tab and click Add Standards.

Figure: The Edit Assignment dialog box indicating the Add Standards button.

Select the check boxes for the learning outcomes you want to align to the current assignment and click Add.

Click Save and Close.

5.4 Standards Aligned from a Rubric

On the assignment, if you add the rubric to which you previously aligned standards, those standards also appear in this tab as being aligned with this activity.

On the course home page, click Assignments.

Click Edit Assignment from the context menu of the assignment where you want to attach a rubric.

In the Properties tab, click Add Rubric.

Select the check box for the rubric you want to attach to the assignment.

Click Add Selected.

Click Save and Close.

The Standards tab display is updated to show the standards which are aligned to the rubric and associated with this activity using the rubric in comparison to those which are directly aligned to the assignment. Notice that an X appears to the right of each standard which was aligned directly to the activity. Only those directly aligned standards can be removed from the activity (until the first student is evaluated). The standards which were incorporated through the rubric cannot be removed individually.

The Edit Assignment dialog box displays the aligned learning objectives. Only objectives directly related (not through the rubric) can be removed here.

Figure: The Edit Assignment dialog box displays the aligned learning objectives. Only objectives directly related (not through the rubric) can be removed here.

5.5 Align standards to a discussion topic

In your course, from the navbar, click Discussions.

Create or edit a discussion topic.

Click the Standards tab and click Add Standards.

Figure: Displays the Edit Topic dialog with a focus on adding learning outcomes to the discussion.

Select the check boxes for the learning outcomes you want to directly align to the discussion topic and click Add.

Note: If a rubric with aligned outcomes is attached to the discussion topic, then the standards associated with the rubric appear on this tab. Using the Add button directly aligns learning outcomes to the discussion topic. In the image related to step 3, above, there are already outcomes directly aligned to the discussion, and not related to the rubric.

Figure: Shows a text box for entering the name of your new Program.

Figure: Browse Standards enables users to select Standards to align to discussion topics.

Click Close.

Note: To assess the standards you just aligned to the discussion topic you must do one of the following:

Link the discussion to a grade item
Attach a rubric to the discussion, and evaluate against the rubric

If you don’t link the discussion to a grade item or evaluate with a rubric, the option to assess the standards does not appear.

5.6 Align standards to portfolio evidence

As part of their course work, a learner can upload any evidence to their Brightspace Portfolio. After the evidence has been uploaded, an instructor can associate standards with the evidence.

Using the Class Portfolios tab, select a learner’s name from the list.

Figure: The Portfolio interface, with the Class Portfolios tab highlighted.

Do one of the following:

If you have turned on the Evidence Approval Queues (in Portfolio Settings), you can select the learner from the Approve Evidence tab and then select the evidence.
Select the learner evidence you want to review

From the learner’s Portfolio evidence, under Student & Parent, click Add Standards.

Figure: The Student & Parent area of Portfolio, with the Add Standards button highlighted.

Select the check boxes for the learning outcomes you want to align to the Portfolio evidence and click Add.

The standards appear in the sidebar, ready to be assessed.

5.7 Align Standards to Quiz Questions

Standards can be aligned to individual questions or question pools. If you align a question pool to a standard, all questions in that pool are aligned in all incidences of the question inside and outside the question pool.

From the course navbar or Course Admin > Assessment, click Quizzes.

Click a quiz that already contains quiz questions.

Figure: The Manage Quizzes tab, where you can select a quiz to use to align standards to quiz questions.

Click Add/Edit Questions.

Figure: The Quiz Questions page lists questions. The Add/Edit questions button is highlighted.

Select one ore more questions or question pools to be aligned with standards, then click More Actions.

Note: The More Actions button does not appear until you have selected a question.

Click Align to Standards.

Select the check boxes for the learning outcomes you want to directly align to the quiz questions and click Add. The associated standards now appear under each of the selected questions.

Clear the check boxes for the previously selected quiz questions and repeat the process to align any additional quiz questions to appropriate standards.

When you have completed aligning questions to standards, click Back to Settings for <selected> quiz.

6.0 Assess learner work against a standard

After you have aligned standards to Rubrics, Assignments, Discussions, Brightspace Portfolio evidence, and Quizzes, you can assess standards in each of these tools.

6.1 Assessing assignment submissions against standards

On the assignment submission page, a learner can only see a read-only list of the standards which have been aligned to assignment. For standards which are aligned to rubric criteria, the suggested achievement level of each standard is based on the mapping of the achievement level scale.

After an assignment is submitted for assessment, navigate to the evaluation page for the submission.

In the assessment panel, click Standards to expand the collapsed section.

Review the suggested achievement levels, and if necessary, manually adjust the levels based on professional judgement and discretion.

Figure: Suggested achievement levels for an assignment submission.

After you publish the feedback, the learner can see the results of the assessment as well as the rest of the feedback on their assignment submission.

6.2 Assessing discussion topic responses against standards

Discussion topics must have points-based assessment or be associated with a grade item or rubric in order to have the assessment options available.

On the assessment screen for discussion topics, open the Topic Score or rubric dialog.

Under the Standards section of the dialog, select the appropriate level of achievement for the learner for each standard.

Figure: The Standards list and suggested level of achievement.

Click Publish to complete the assessment and make it available to the learner.

6.3 Using rubrics to generate suggestions for assessment of standards

For assessment suggestions to appear, ensure the following are true:

Your achievement scale for the standard uses percentage thresholds

The d2l.Tools.LearningOutcomes.LoaSuggestions configuration variable is turned on (to provide suggestions).

On the assignment submission page for the learner, open the rubric for assessment.

On the pop-up rubric assessment, you can scroll over the target icon to review which standards are aligned to each criterion.

Figure: Hover over the target icon to review the full standard text.

For each criteria row in the rubric, click to select the assessment level and click Add Feedback if you want to include additional feedback for each criteria row in the rubric. Changes are automatically saved.

Explaining the scenario:

In the scenario used in the previous task, Mathematics Specific Expectation 1.1 is aligned to Criterion 1 and Criterion 2. The learner has achieved 10 points (Criterion 1) and 3 points (Criterion 2) for a total of 13 out of a possible 20 points, or 65%. The achievement scale previously set up has a threshold between the 2 levels at 70%.

Figure: Percentage threshold settings.

On the assessment panel for the standards on this assignment, the suggested assessment of Not Yet appears for the Mathematics Specific Expectation 1.1. Mathematics Expectation 1.2 was only aligned to Criterion 1, which earned 100%, so the suggested assessment is Got it.

Figure: Standards evaluated based on the scale threshold.

Other standards that were aligned directly to the assignment and not to the rubric are still available to be assessed as usual, but do not show suggestions.

6.4 Assessing portfolio evidence against standards

On the screen to view a learner’s portfolio evidence, after the standards have been aligned, select the appropriate level of achievement:

Figure: Brightspace Portfolio user interface, with the Standards area highlighted.

6.5 Assess Quiz Submissions Against Standards

For automated standards suggestions to appear on quiz submissions, ensure the following are true:

A percentage scale must be set as the default scale

The quiz must be configured to allow attempts to be automatically graded upon completion. You can set this option on the Assessment tab of the quiz.

It’s possible to view assessments in quizzes in two places:

From the Grade Quiz page to see individual responses. Click an attempt to view the quiz details.

Figure: The Grade Quiz interface, indicating the quiz attempts taken by a learner.

From the Achievement Level of each standard on the quiz.

Figure: Evaluation from the Achievement level of each standard.

Figure: Click the appropriate column of the rubric to indicate the assessment level.

7.0 View learner achievement on standards

The Outcomes Progress page is a tab in User Progress that shows a learner’s assessments against standards aligned to activities.

From Class Progress, click the learner you want to view.

Click the Standards tab.

Figure: The Progress Summary page.

On the Standards Progress page, there is a list of all standards aligned to activities in the course. The bars reflect the most recent 6 assessments, in order of assessment date.

Figure: Standard Progress page showing attempts against each standard.

Click on any individual standard to view more specific details about assessments for that learner on that standard, including the full history of assessment across a timeline and a summary of feedback given on each activity.

Figure: Progress Details page shows the trends and evidence for each standard.

7.1 Multiple Quiz Attempts

If quiz questions have been aligned to standards, then the quiz submission appears as a demonstration of that standard and is reflected in the visualization on the Standards Progress Page for the learner.

Figure: Standards Progress page.

If the quiz permits multiple attempts, each attempt at the quiz is reflected for each standard. On the demonstration details page, each submission attempt is displayed in the evidence timeline scrolling down the page. To launch the details page, click the title of the standard on the Outcomes Progress page.

Each attempt has the achievement sale level designated in the evidence title. If different levels of achievement were obtained in each attempt, then the attempts appear as different colors in the assessment timeline. The colors reflect those defined on the achievement scale.

Figure: Standards progress showing multiple tries against a specific standard.

Figure: Under the Evaluation and Feedback section, you can select a rubric to use to evaluate a learner.

Figure: Questions are selected to be aligned with standards. The More Actions button is highlighted.