Originally Published February 19, 2019
This article highlights the differences between Advanced Data Sets (ADS) and Brightspace Data Sets (BDS) and focuses on the process needed to maximize your use of Brightspace Data Sets. BDS and ADS are both located in the Data Hub. Both consist of data exports in a standard CSV format with header and include string wrapping using double quote characters when key characters (i.e. commas) are present within the string.
Differences between Advanced and Brightspace Data Sets
Brightspace Data Sets (BDS) can be found in the Data Hub along with the Advanced Data Sets (ADS) but have some key differences:
An individual ADS is intended to be used independently, but individual BDS are intended to be used with other BDS.
Advanced Data Sets can be used on their own as table-based reports because they pull together attributes and values into legible rows in each report. Brightspace Data Sets on the other hand are not intended to be interpreted individually and use integer values as IDs to represent entities in the context of other values. In order to provide context and make the data legible for your reporting needs you will need to combine them together. For example, if you wanted to pull data on the average number of quiz attempts by students per quiz across different courses, you would need to pull at least 3 BDS (Org Units, Users, and Quiz Attempts) in order to correctly identify the data you need.
ADS can be filtered as you request the data set while BDS gives you all of the data and you must filter later.
When requesting the generation of an ADS you can provide filters to limit the data provided. Common filters are on Org Unit and a relevant date range, though filters vary for each data set. BDS, instead have all the data available (they are constrained to 150 million rows of the most recent data; see Data Limits documentation for specifics). One advantage is that you do not have to wait for the data set to generate before you can download it. A disadvantage is you must move far more data to your system which will eventually be filtered out.
ADS generate on demand, whereas the BDS Full Data Sets are delivered weekly with Daily Differentials as a default set-up.
If the date parameters are set accordingly, the ADS will include data up to the moment the request is initiated. BDS are generated on a schedule, with the Full data sets generated weekly and the Differentials generated every day (differentials can be hourly with optional add-on). If you use the Fulls in combination with the Differentials, (without the frequency upgrade) the data you will have available should be as fresh as yesterday - exact timing is determined by the time the job to generate the dataset runs. This will be fresh enough for most reporting needs but is important to be aware of when planning and documenting end reports.
Process to use Brightspace Data Sets successfully
In order to effectively use Brightspace data sets regularly you will need to set up the following pieces in your reporting process that are unique to working with BDS over other data sources in Brightspace:
A database to store the data in.
Because the context of BDS is made by joining sets together, a location is needed to house the data sets so they can be joined. Typically, this is done by creating tables in a relational database in the same structure as the data set files but with correct data types and key relationships. The final format may look different based on the intended usage of the data.
A process to extract, transform and load (ETL) the data from the zipped flat files available for downloading from Brightspace, into a usable format in your database.
Manually pulling datasets for a one-time analysis is reasonable, but in order to keep the data as fresh as possible you will likely need to be ETLing the data once a week or even more frequently if you are using the differentials. Ideally this process is completely automated (Using the API) and robust.
Queries or another method to join, filter, and aggregate the data in order to get the values you need from it.
BDS requires the data sets to be joined to understand the context. There are far more insights available using BDS then other data sources in Brightspace purely because you are flexible in your ability to filter and aggregate the data. However, in order to set the context and see the actionable insights, joining and other manipulations of the data need to be done before the data can be used.
The key resource that is essential to the usage of both ADS and BDS is our Data Hub documentation. This defines the fields used in all the Data Hub Data Sets as well as information on the relationships between the BDS used for joining them.
Need further help working with data? Stuck on any of the above steps? Lacking one of the skill sets needed? D2L provides flexible levels of Data Solutions Consulting services that can assist with any or all steps in the process detailed in this post. If you are interested, please contact your D2L Customer Success representative.