BrightSpace quiz question types inadequate

Mike.B.559

While I suppose that this forum "Instructional Design Best Practices" is intended for discussions about how best to use existing technologies to achieve our intended outcomes, I am in a quandary. Because, as it stands today, the quiz question types we have available to us in BrightSpace as well as the workflows required to create them, are inadequate to my needs. Unless I am creating straight multiple choice or written response type questions, I am stuck with outdated workflows that produce sub-standard UX. Cloze questions with more than one gap (something common in our language classes) are a nightmare to create. There are no drag-and-drop options or hot-spots. Recent changes to the New Quiz Creation Experience are a step in the right direction, but they do nothing to address the underlying inadequacies of the question types themselves. So, we can talk here in the "best practices" group about how to use the existing question types to achieve desired outcomes, but in my opinion we will just be spinning our wheels. What we need to see is significant innovation by D2L in the types of questions we can create and simplified workflows for doing so. This is not a new issue. PIE items calling for these changes date back a decade.

My particular area of interest is language education but I know there are other disciplines that are affected. Nursing programs and schools preparing students for the NCLEX, for instance, have been quite vocal in expressing their concerns. I have also heard from others in STEM fields who would like to see simpler and more flexible cloze question types as well as more flexible alternatives for math and chemistry.

Please feel free to weigh in with your thoughts on this situation.

Also, please consider supporting these related items on the PIE:

BrightSpace quizzes in need of critical updates (D9656) 

Create New Generation NCLEX Question types for Quizzes (D6377)

Ask STEM community what functions THEY need in arithmetic questions and implement them (D6865)

Keywords: language education, language learning, quiz question types, drag and drop, cloze, NCLEX, nursing

Stefanie.B.518

Hi @Mike.B.559
I'm very grateful to you for staying connected here in the Community to share your feedback and thoughts.
I've shared these sentiments with our dedicated Quizzing Teams for review.

I think you and the Community Team will also have a chance to chat more about an upcoming event to bring language educators together to learn more about how different departments are using online learning tools to support assessment and evaluation experiences across different languages and program needs.

Your feedback and advocacy for your students and colleagues is helping us to learn together from one another- grateful to you for keeping the conversation going here.

Marcia.B.5

I echo the sentiment that the quiz question formats are inadequate and need to be expanded and updated. Particularly the matching - if Zoom can offer matching and ranking question formats that are user-friendly as well as accessible, D2L should absolutely be able to do so. Every question format needs to be fully accessible and usable.

Alisa.C.43

I am also interested in an expansion of quiz questions particularly calculated questions with multiple variables and multiple questions based on one case study. The other option that would be great that I ran across today is a Survey question type that blends MC and a short answer as in: What have you eaten today for breakfast? A. Eggs B. a sandwich C. Fruit D. nothing E. Other (please specify) [insert text field]. I think more question types and options for building more complex questions for higher ed is important especially in the context of generative AI options.

Oli.M.837

I have encountered its limitations as well recently when our team developed a digital escape room. The quizzes forcing submission before receiving any kind of feedback, and inability to halt progress without selecting the correct answer led to requiring an awkward solution that was much more clunky (using release conditioned pages based on quiz success to indicate the players ability to progress).

Though what we wanted to use it for seems light, I do find it odd to make it so awkward to receive dynamic feedback. And the idea that the Brightspace Quiz functions have to be navigated to rather than being integrated into the content experience is just... not the best. (Though my understanding is that there may be some update to the UX coming up in that regard).

Mike.B.559

@Oli.M.837 Would love to hear more about this "…update to the UX".

Stefanie.B.518

Hi @Marcia.B.5 Your feedback on Quiz question types is important. I've shared these sentiments with our Product Team.
When you mention accessible, are you describing ease of use or an issue of accessibility in terms of access with one or more assistive devices? If it's the latter, we will definitely want to create a support case to review the details with our product and Accessibility Teams.

@Alisa.C.43 I can picture a quiz that allows you to create multiple questions on a single case study- what types of challenges are you running into there? There are some great thoughts here on that type of combination style question. You're likely already taking an approach where you add a multiselect followed by a short answer question. That's the approach I would take currently as well.

@Oli.M.837 In your escape room, is there an opportunity to use hints as a form of inline feedback prior to learner submission? I'm definitely interested in learning more about the idea of halting progress- is that intended as a game mechanic or an opportunity to insert learner reflection (I'm truly just curious to learn more in case I can provide a handy solution in either case). We recently had an Ontario K12 Educator contribute her Escape Room Walk Through to the Brightspace Learning Center. She uses a few different activities like mazes, and wordle which might be thought of as an assessment as learning type of formative assessment and offer some support towards offering different kinds of feedback interspersed through the escape room. Let me know if this is handy/if you'd like to share more about your Escape Room-I would genuinely LOVE to hear more!!

@Mike.B.559 I think Oli is describing some features slated for the New Content Experience, described in the Product Roadmap from August-October our Content Dev Teams will be working on a new display of assessable activities in the New Content Experience. The goal of this effort is to allow Educators to create, evaluate, review, and monitor assessment activities consistently within the New Content Experience without navigating to other tools. Learners would be able to participate in these activities without having to be linked to them. The new display of assessable activities will be consistent across roles, activity types, and activity progress.

Couldn't be more grateful to each of you for your thoughts and comments here- please keep them coming and I hope you'll forgive the delay in response there as so many of us were supporting or at our annual Fusion Conference last week.

Mike.B.559

Thanks @Stefanie.B.518 but…I'm still not clear. What is an "assessable activity"? Is it a quiz? An assignment? A discussion post? All of the above? Or is it a whole new category of activity?

Stefanie.B.518

Hi @Mike.B.559
Great question. I think the roadmap is using that language to describe activities that receive a grade item in Brightspace.

Plans to bring assignment, quiz, and discussions inline in the content view of the New Content Experience, instead of linking to them were described in greater detail at our recent Fusion Conference. You can see some of this from the Fusion on Demand Content in the Product Deep Dive Higher Education Video around 15:54 in the video.

Mike.B.559

Link does not work:

Stefanie.B.518

Hi @Mike.B.559 that's so weird!
Sorry for the trouble - I'm not sure what's going on there - is there any difference or change when you try selecting from here: https://www.d2l.com/fusion/

It's the same link but formatted differently

Mike.B.559

@Stefanie.B.518 Thanks…it's working now.

Mike.B.559

Thanks for the tip Stefanie. That video was useful.

Marcia.B.5

https://community.d2l.com/brightspace/profile/9/Stefanie.B.518 the matching and rank question formats are not compatible with screen readers. When I tested these question formats with two of my staff who use screen readers, the screen reader would not read all the elements of the questions. But when we looked at these same question formats on Zoom, the screen readers were able to properly read the questions. This is an accessibility issue, not an ease of use issue. I've told my instructors they may not use matching or rank questions because those formats are inaccessible to screen readers.

Stefanie.B.518

Hi @Marcia.B.5
I'm very grateful to you for sharing more information here.

If you haven't yet, creating a support case that includes information about the types of screen readers used (JAWS/NVDA + any other assistive technology?) can help a great deal in finding the best solution.

I've passed this information on to your Customer Success Team here at D2L and to our Accessibility and Quiz Team for review.

Carin.H.549

Hi. I just wanted to chime in and ask more questions. My experience as a screen reader user is that matching questions, although nerve-wracking and cognitively taxing, can be done. It does take a lot of memorizing and double and triple checking to make sure you chose the correct numbers, but I have never encountered the scenario where the match options cannot be read. Which screen reader are you using? I recently completed some matching questions with JAWS and NVDA. Although Voiceover has been tested previously, I can't say that I've completed a matching question recently with Voiceover, although I certainly am willing to give it a try. But with JAWS and NVDA, I could read the possible answers.

I had some questions. You mentioned "completing a matching question with Zoom". Do you mean Zoom as in the conferencing software, or are you referring to Zoom on the mac, as in screen magnification? A small part of me has been a little sad that that video conferencing software chose the name Zoom because I was worried about confusions just like this.

Also, can you make a video of the screen-reader failing to read the options? I would like to understand the workflow that causes this. If it helps, you can't read the possible match options while in forms or focus mode if we're talking about JAWS or NVDA, or while quick nav is off in Voiceover parlance. You have to be able to browse the text above the dropdowns, then move past all the dropdowns and then read the text below where it says something like "1 is blue, 2 is red, 3 is green, 4 is yellow" for example.

In my perfect world, each dropdown would have all the options instead of a group of numbers. So the question would say "sky", and as you moved through the dropdown,

you would hear, "blue, red, green, yellow" and just be able to make your selection. But I have been told that for technical reasons that are beyond me, this would be very difficult to do. Part of the reason is we have no idea the length of the answers someone might generate. The user might have to match a term with a definition, for example. If you cram that definition inside the dropdown, the result might not be good for anyone. I had thought about some kind of truncated version, but what happens if all the beginnings of the matches are identical? How can you guarantee that there will be something meaningful in each of the answers? It is a difficult problem to solve, but we know it is a problem.

Have you seen a case of matching questions that would work well? It would be nice to see some fresh inspiration.

Looking forward to hearing more.

Marcia.B.5

Hi @Stephanie.B.518 and @Carin.H.549 - I'm sorry, but I am not usually in Community and just saw these posts today.

I am not a screen reader user - my colleagues are. My recollection is that when they tried to read the questions using JAWS it skipped the potential responses and only read one column of items instead of both. I will try to see if we can schedule a time to do some testing and recording, but honestly, we don't really have time for that.

The Zoom conferencing platform has added matching questions that my colleagues are able to use easily and without any issues. I would recommend your development team explore that format to see how they are setup and try to replicate in Brightspace.

Stefanie.B.518

@Marcia.B.5 apologies for the delayed response here-I know our Accessibility Team did have a chance to connect earlier in this thread, and I was able to confirm that they reviewed the quizzing experience based on your feedback.

For yourself, your colleagues, and others that may be following the thread as well- there's a wonderful group that meets monthly via Community called the Accessibility Interest Group. They have a dedicated discussion area and often bring in wonderful speakers to share about best practices, new research and assistive technologies.

I know they would be happy to have you join, and it can be another way to share feedback/connect with other users of assistive technology fairly quickly.

Grateful to you for sharing back with us and being a part of this Community space!

Mike.B.559

Hi all, I thought I would cross-post this item here since we all seem interested in improving question types on the platform. I am currently working on fill-in-the-blanks (cloze) questions in an online quiz and am running into major roadblocks. Any insights would be welcome.