Written by: Jackie Guan, Product Manager, D2L.
Advanced Assessments help instructors and administrators to better manage courses with large class sizes and multiple evaluators, delegate and manage evaluator workflows, and reduce bias in marking.
With the August 2024/20.24.8 release, Advanced Assessments features are enabled by default for all users. A new org-level configuration variable, which turns Advanced Assessments on or off, will be added in a future release. For more information, refer to the The New Planned Configuration Variable for Advanced Assessments blog post.
If you are interested in controlling the rollout of Advanced Assessments, fill out this online form before your August update. In this case, the features will be hidden on your site, and you can use the future configuration variable to choose when to enable them.
Once Advanced Assessments features are enabled in your instance and assignments are created, we do not recommend disabling the features with the future configuration variable.
Advanced Assessments allows instructors to:
- Assign multiple evaluators to evaluate a single assignment submission when creating or editing assignments.
- Choose which evaluators can publish grades for the assignment from the Assignment tool by setting them as publishers.
- Match evaluators to learners.
- Choose from two multi-evaluator workflows: One Shared Evaluation (co-marking) or Multiple Individual Evaluations.
Advanced Assessment features for Assignments in production environments
In the May 2024/20.24.5 release, the Advanced Assessment features were released to test, development, and staging (non-production) environments.
As of the August 2024/20.24.8 release, all clients have access to Advanced Assessment features for Assignments in all environments of Brightspace.
The following section details new Advanced Assessment features that are included all production environments of Brightspace as of the August 2024/20.24.8 release.
Set up a delegation, co-marking, or multi-evaluator workflow
When creating a new assignment or editing an existing assignment, in the Evaluation & Feedback accordion of the Assignments tool, the number of evaluators who can give feedback in the course is displayed.
If an instructor clicks Edit Evaluators, an Evaluators dialog displays all evaluators with the Give Feedback role permission in the course. When you click on Edit Evaluators, the advanced assessment workflows begin.
Instructors can add evaluators to the assignment by selecting evaluators in the Evaluators list dialog.
If an instructor does not make any selections and click cancel, the advanced assessment workflows will not be available, and the instructor can continue creating the assignment, and it will be a non-advanced assessment assignment. All evaluators who have the Give Feedback permission will have access to this activity.
If the instructor click Save on this page, it will trigger the advanced assessment workflows.
After the evaluators are assigned, there are now four new capabilities in the Evaluation & Feedback accordion:
- Evaluators
- Publishers
- Matching learners to evaluators
- Multiple Evaluator Mode
Assign publishers
Publishers are users who can publish evaluations and make them visible to learners.
While creating or editing an assignment with assigned evaluators, instructors can assign publishers based on the options listed below.
- All evaluators can publish any evaluations
- Evaluators can only publish their evaluation: This means that only the evaluator assigned to evaluate a specific learner submission can publish the evaluation. Review the Advanced Evaluation dialog to learn which evaluator gets to publish which learner submission.
- Manually select publishers: This means one or more users from a list of users who have the Give Feedback permission can be selected, which allows them to be the publisher. A list of users with the Give Feedback permission appears, and then a selection can be made to decide who publishes the evaluation to the learner.
Depending on the option selected, a Publish button will appear on the evaluation screen for users who meet the criteria of that option.
Allocate evaluators to learners (Manage Matching)
Click the Manage Matching link to open the evaluator mapping dialog table. Options are available in the drop-down list to determine how the learners should be matched to the evaluators. The system can randomly match each learner with up to three evaluators. However, it is also possible to customize the evaluator and learner mapping to choose more specifically who evaluates who using the learner-evaluator mapping table and selecting or un-selecting the matching check boxes.
It is possible to clear all the mappings in the table at once by clicking Clear Table.
Evaluator mapping can also be assigned by groups and/or sections using the Groups/Sections drop-down menu. In this menu, the desired group or section can be selected to display those learners. Then those learners can be mapped to an evaluator.
Two options are available for evaluation modes that allow you to assign evaluators quickly:
- One shared evaluation: All evaluators contribute to the same shared evaluation of the learner’s submission. This is often referred to as co-marking. This option is best for instructors who collaborate in person on a learner’s evaluation. For a more detailed, separate evaluation, review the multiple separate evaluation option instead.
- Multiple separate evaluation: Each evaluator completes their own individual evaluation separately. Those independent evaluations must be aggregated by the publisher before being released to the learner. This is often referred to as an independent multi-marker.
Figure: You can map learners to evaluators in the Allocation Options section.
Updates to Online Grade in Turnitin integration
Figure: Instructors with Turnitin integrations and Advanced Assessment features enabled have the option to disable Online Grading tools.
Note: We understand that plagiarism detection is a key feature of TurnItIn and therefore we will be supporting the Similarity Report instead, to ensure instructors still have access to TurnItIn's plagiarism detection tools.
Important: Online Grading is turned off to reduce discrepancies with the multi-evaluator workflow, as Turnitin currently does not support multiple evaluators. Instructors can still access the Online Grading option called Feedback studio in Turnitin when they access the tool on Turnitin's page but any inputs in their tool will not be restricted by the evaluators defined in Advanced Assessment. Additionally, any grades or feedback will not auto-sync with Brightspaces Consistent Evaluation tool or Grade book.
For more information, refer to the Create an assignment (Customize Evaluation & Feedback settings) topic.
Grade with Quick Eval
Quick Eval Activities view
Quick Eval enables instructors to quickly evaluate all submissions and activities. As of the August 20.24.8 release, production environments of Brightspace display co-marking, delegation, and multi-evaluator evaluations in the Activities tab in Quick Eval. This helps instructors know what to evaluate next. They can follow the same Quick Eval workflow as with other activities.
Review submissions from a single page
Quick Eval offers evaluators a single location to view all learner submissions that are awaiting evaluations. Submissions made to Assignments, Quizzes, and Discussions are displayed in a sortable list and can be selected to take you directly to their evaluation page.
When the advanced options are set for the assignment, the evaluator will only see learner submissions for the assignments they have been assigned to evaluate in the Quick Eval Submissions view.
The counts in the Quick Eval Activities view correspond to all learners, except sections.
In Quick Eval, you can see submissions for all learners and counts for Saved Drafts, Ready for Review and Ready for Aggregate statuses of evaluations are summed under the Evaluated count if you are a publisher assigned to publish all learner submissions.
If the user is a publisher assigned to publish all learner submissions, then, in Quick Eval, the publisher will see submissions for all learners.
Figure: Use the Activities view to see the corresponding counts.
Single Shared Evaluations
Often, evaluators and publishers may not be the same person. This feature ensures that users using co-marking and delegation assignments have a way to communicate with publishers when their evaluation is ready for review. The feature reduces communication errors during evaluation. When an evaluator has completed their evaluation, they can set the evaluation to Ready for Review, which will signal to the publisher that the evaluation is ready and can move on to the next phase.
For more information, refer to the Grade with Quick Eval (Evaluate learner submissions) topic.
Synchronize multi-evaluator grades between Grade Book and Assignments
Instructors can synchronize grades and feedback from the Grade Book into Assignment Evaluation while using multi-evaluator evaluations.
When a group of instructors use delegation or co-marking to evaluate an assignment, entered grades synchronize with Grade Book. Also, when an instructor enters a grade for an assignment in Grade Book, the entered grade synchronizes assignment grades.
Inputs into grade book populate into the Aggregated tab for Assignment Evaluation when a user chooses to sync from the grade book.
Other features include the following:
- Any Aggregated tab that has a draft saved and published will be overridden by the Grade Book inputs when the Grade Book sync is started.
- Any evaluation not marked as ready to aggregate is switched to ready to aggregate when the Grade Book sync has been started.
- Any evaluation already marked as ready to aggregate will stay unchanged when the sync is started.
For more information, refer to the Synchronize multi-evaluator grades between Grade Book and Assignments topic.
Status indicators for multi-evaluator workflows
With the availability of a multi-evaluator workflow, we support multiple personas in the evaluation process who act or may be needed at different stages of the evaluation process.
Prior to the May 2024/20.24.5 release, the Users tab of the Assignments tool only displayed the status indicators Evaluate, Draft Saved, and Published to show instructors the assignment status.
Figure: The Users tab of the Assignments tool before the May 2024/20.24.5 release.
Figure: The Users tab of the Assignments tool after the May 2024/20.24.5 release. Note that an assignment displays additional information with hover text.
Figure: The Users tab of the Asignments tool, for observational assignments, after the May 2024/20.24.05 release.
This change helps to increase consistency with other Brightspace tools
Known Limitations
- Brightspace APIs are not available to set up these added options on the assignment.
- The Turnitin Online grader is not currently available with these options.
- Brightspace datasets are not available for these options.
- Group assignments do not have these advanced options.
- Advanced options cannot be set on an assignment created at the org level.
FAQs
Is there any new permissions?
The Give Feedback permission is used to populate both the evaluator list and publisher list.
Can this new capability be enabled for only specific courses?
No, this capability must be enabled for all org units.
Who can set up these advanced options for the assignment?
Anyone with the Add/Edit Assignment Submission Folders permission can set up these new options.
What happens when a new learner enrolls in the course?
Based on the allocation option set for the assignment, we map the new learner to an evaluator. We do not overwrite any existing learner-evaluator mapping.
Are cascading roles supported?
No, we expect users to be explicitly enrolled in the course.
What is the behavior when the course package is exported and imported?
The advanced options are not maintained when the course is exported and imported back.
