Can we get clarification about differential data sets?

Hugh.Gleaves7274

It's not clear to me from reading the documentation if every differential data set contains changes relative to the preceding differential data set or not.

 

Consider:

 

1. Monday 4am - Full data set is generated
2. Monday 8am - Diff data set is generated
3. Tuesday 8am - Diff data set is generated
4. Wednesday 8am - Diff data set is generated
5. Thursday 8am - Diff data set is generated
6. Friday 8am - Diff data set is generated
7. Saturday 8am - Diff data set is generated
8. Sunday 8am - Diff data set is generated
9. Monday 4am - Full data set is generated
10. Monday 8am - Diff data set is generated.

 

What is not clear to is whether data set 10 only contains differences with respect to data set 9 or to the preceding differential data set 8.

 

In other words can we disregard full data sets (once we have the first full data set, i.e. 1) and consider ever differential data set to always contain changes relative to the preceding differential data set?

 

Thanks

 

 

 

 

[Deleted User]

Hi @Hugh Gleaves​ 

 

Based on https://community.brightspace.com/s/article/Brightspace-Data-Sets-Differential-Data-Sets-Client-Example

"These differential extracts are generated at a set interval, and contains data newly available since the previous differential extracts were generated."

and

https://documentation.brightspace.com/EN/insights/data_hub/admin/bds_intro.htm?tocpath=Administrators%7CBrightspace%20Analytics%7CData%20Hub%20tool%7CBrightspace%20Data%20Sets%7C_____0

"Differential (diff) data sets contain the differences (diffs) of the data that has been updated or inserted in the previous day, and refreshed daily."

it seems every new differential contains differences relative to the previous differential. So differential 10 will have new data relative to differential 9.

 

Today's full file = Yesterday's full file + Today's differential.

 

I hope this helps clarify.

 

Thanks

Rahul.

 

Kara.Schimmelfing6598

Hi @Hugh Gleaves​,

 

Rahul definitely called out some great resources, but I wanted to clarify one point:

Differential 10 will have new data relative to differential 8 (9 in the example is a full data set not a differential).

 

Once you have consumed a full data set you can proceed using only differentials for the inserts and updates going forward, you would only need to use the fulls again if you miss a file or to capture deleted entities in data sets without a soft deletion field(s). You will not lose any inserts and updates data if you do not consume full data set 9 but consume all the other data sets in the example.

Renee.J.194

@Rahul Desai​ , if @Hugh Gleaves​ does not incorporate Full File 9, will he miss new data that is not delivered in Differential File 8 (because it was generated after Differential 8) OR in Differential File 10 (because it was generated between after 8 and before Full 10 and therefore was inlcuded in 9 andis no longer new)? So to catch what happens in that 4-hr window, Hugh will always need a full dataset?

 

I ask because we have a similar situation, and have only been collecting differentials after our initial set-up.

Hugh.Gleaves7274

Thanks to each of you for taking the time to explain this, very much appreciated.

 

Renee.J.194

Ah, thank you, @Kara Schimmelfing​! I wrote my question before your answer appeared for me! That is the exact answer I was hoping for, because that is what I promised our data architect. I would have hated to be wrong about something so fundamental!

[Deleted User]

Thank you @Kara Schimmelfing​  for your clarification, it really did help out heaps.

 

Apologies for any confusion Hugh and Renee.

 

Thanks,

Rahul.