What does it mean if there's no differential dataset generated on a certain date and how to handle?

Question

For a specific project, I need to have all content user progress differential datasets in a certain period. (We want to count the number of page vistis per day per module in a certain course so if we get the ‘last visit date’ from the ContentUserProgress.csv, daily, we should be able to get latest visits for each user for each page and this way get to the total sum of visits for a certain page each day).

When I check the past two weeks in the data hub, there seems to be one day where no (March 9) dataset is available for download. What does this mean and why does this happen? It is 99% impossible there wasn't any user behaviour affecting the data on this day.. Can this one still be generated or would I be able to find the information in a different way?

Aasim.Y.704 · Answer

Hey @Hugh.Gleaves7274,

Thanks for providing this feedback! I will connect this to our Product Experts for review.

Hoping to help continue to enhance your experience with Brightspace.

Hugh.Gleaves7274 · Answer

@Aasim Yac947 (@Donna.Schipper2598) - Hi Aasim. I've been seeing strong evidence that sometimes when a differential is not generated the following differential does not always contain the data that it should. We have evidence of this where an update was made that appears in the next full dataset but should have been in a 7pm differential that was skipped. We found that the missing update was not in any later differential and the only way to get the update would be to use the next full dataset which would be a huge amount of work for some large (1 million+ rows) datasets, this is a problem we have not solved yet as it is pretty challenging.

Aasim.Y.704 · Answer

Hey @Donna.Schipper2598,

Thanks for reaching out to us here at the Brightspace Community.

It looks like March 9 differential did not generate. This can happen if for instance a CD Update or Hotfix was being applied to the site and interfered with the data set generation. This is documented as a known issue.

You can also see that the data set successfully generated on March 10 - and that the March 10 file is larger than most other differentials - this is because it will contain the changes from both March 9 and March 10.

I hope this information helps!