Written By Don Brinn, Senior Test Developer 2
This article introduces using the Data Hub APIs to retrieve Brightspace Data Sets (BDS) and gives a practical example using the process.
What information can I currently get from Data Hub?
Currently, customers can access more than 130 Brightspace Data Sets and this keeps growing. Brightspace Data Set extracts are available as .CSV files. Each Brightspace Data Set is available in both a full version and a differential version (each differential extract file includes updates and changes since the previous differential file). A history of previous differential files is retrievable via Data Hub and the API, with differential extract files from the past 14 days available via the API. Additionally, Advanced Data Sets are also located in the Data Hub but are outside the scope of this article.
How do I access Data Hub?
Data Hub can be accessed in Brightspace. For convenience, a Data Hub link can be added to your navbar. Alternatively, you can also navigate to /d2l/datahub/dataexport/list in your browser.
Data Hub can also be accessed via API. The following sections will walk you through accessing Data Hub through the API.
What is the expected flow when using Data Hub's API?
Note: authorization is not covered in this section. See the code walkthrough below or the Refresh Token blog post for more information.
The routes below are documented within our API reference documentation.
See All Brightspace Data Sets
First, decide which Brightspace Data Set you would like to retrieve. There is an API route to see the available Brightspace Data Sets: /d2l/api/lp/1.43/datasets/bds.
For each Brightspace Data Set, this will provide the schema ID and a link to view the extract files available to download. For the full and differential versions of each data set, it will include their name and description, the plugin ID, and a link to view the extract files available to download of that type (full or differential).
Example output:
{
"Objects": [
{
"SchemaId": "4320d948-b06a-4fa2-b4ad-723ed9b002aa",
"Full": {
"PluginId": "553ee539-6ecc-4096-b845-acb1e8efd9eb",
"Name": "Activity Feed Post Objects",
"Description": "The Activity Feed Post Objects data set returns details of the posts that have been created in the organization. This includes deleted posts.",
"ExtractsLink": "https://myacademy.desire2learn.com/d2l/api/lp/1.43/datasets/bds/4320d948-b06a-4fa2-b4ad-723ed9b002aa/plugins/553ee539-6ecc-4096-b845-acb1e8efd9eb/extracts"
},
"Differential": {
"PluginId": "15679ea1-ef36-4839-a57e-8e70da5a98ac",
"Name": "Activity Feed Post Objects Differential",
"Description": "The Activity Feed Post Objects differential data set returns details of the posts that have been created in the organization since the last differential was generated. This includes deleted posts.",
"ExtractsLink": "https://myacademy.desire2learn.com/d2l/api/lp/1.43/datasets/bds/4320d948-b06a-4fa2-b4ad-723ed9b002aa/plugins/15679ea1-ef36-4839-a57e-8e70da5a98ac/extracts"
},
"ExtractsLink": "https://myacademy.desire2learn.com/d2l/api/lp/1.43/datasets/bds/4320d948-b06a-4fa2-b4ad-723ed9b002aa/extracts"
},
{
"SchemaId": "b1f01838-2cee-44d5-a731-189a15086bd2",
"Full": {
"PluginId": "b639f054-10fc-47af-8fd7-1e7d57b2c79e",
"Name": "Survey Question Answer Options",
"Description": "The survey question answer options data set returns details about each answer option available for each survey question.",
"ExtractsLink": "https://myacademy.desire2learn.com/d2l/api/lp/1.43/datasets/bds/b1f01838-2cee-44d5-a731-189a15086bd2/plugins/b639f054-10fc-47af-8fd7-1e7d57b2c79e/extracts"
},
"Differential": {
"PluginId": "43eb179b-342a-40a1-b029-809141955254",
"Name": "Survey Question Answer Options Differential",
"Description": "The survey question answer options data set differential returns details about each answer option available for each survey question since the last differential was generated.",
"ExtractsLink": "https://myacademy.desire2learn.com/d2l/api/lp/1.43/datasets/bds/b1f01838-2cee-44d5-a731-189a15086bd2/plugins/43eb179b-342a-40a1-b029-809141955254/extracts"
},
"ExtractsLink": "https://myacademy.desire2learn.com/d2l/api/lp/1.43/datasets/bds/b1f01838-2cee-44d5-a731-189a15086bd2/extracts"
},
... (Note this response is significantly longer but has been truncated for article readability)
],
"Next": "https://myacademy.desire2learn.com/d2l/api/lp/1.43/datasets/bds?bookmark=MTY4NjgzNDAwMA"
}
Note that these results are paginated, so you should follow the link in the “Next” field to see more of the list of data sets, and continue that until “Next” is null.
See Extract Files Available for a Brightspace Data Set
Brightspace Data Sets, unlike the Advanced Data Sets, do not take filter parameters, so you only need the schema ID and plugin ID to see the data set extract files available and to retrieve one of them.
Additionally, the Brightspace Data Sets have already been generated so you do not need to check their status before retrieving to confirm they are complete.
To view the list of data set extract files available to retrieve for the full or the differential version of a data set, use the API route /d2l/api/lp/1.43/datasets/bds/(schemaId)/plugins/(pluginId)/extracts.
Replace "(schemaId)" and "(pluginId)" with the values for a data set from the API route that listed all Brightspace Data Sets, as shown in the previous section of this article. You can also get the full API route including all parameters by using the "ExtractsLink" links from the API route that listed all Brightspace Data Sets.
Each object listed by this API is an extract file available to retrieve.
Example Output:
{
"Objects": [
{
"SchemaId": "b1f01838-2cee-44d5-a731-189a15086bd2",
"PluginId": "43eb179b-342a-40a1-b029-809141955254",
"BdsType": "Differential",
"CreatedDate": "2023-06-15T16:26:09.433Z",
"DownloadLink": "https://myacademy.desire2learn.com/d2l/api/lp/1.43/datasets/bds/b1f01838-2cee-44d5-a731-189a15086bd2/plugins/43eb179b-342a-40a1-b029-809141955254/extracts/MTY4Njg0NjYwMA",
"DownloadSize": 225,
"QueuedForProcessingDate": "2023-06-15T16:25:57.000Z",
"Version": "8.7"
},
{
"SchemaId": "b1f01838-2cee-44d5-a731-189a15086bd2",
"PluginId": "43eb179b-342a-40a1-b029-809141955254",
"BdsType": "Differential",
"CreatedDate": "2023-06-15T15:26:09.347Z",
"DownloadLink": "https://myacademy.desire2learn.com/d2l/api/lp/1.43/datasets/bds/b1f01838-2cee-44d5-a731-189a15086bd2/plugins/43eb179b-342a-40a1-b029-809141955254/extracts/MTY4Njg0NjMAYw",
"DownloadSize": 225,
"QueuedForProcessingDate": "2023-06-15T15:25:57.000Z",
"Version": "8.7"
},
... (Note this response is significantly longer but has been truncated for article readability)
],
"Next": "https://myacademy.desire2learn.com/d2l/api/lp/1.44/datasets/bds/a7d6e843-bf8d-4965-9274-95028f3c4d86/plugins/f1623581-c5d7-4562-93fe-6ad16010c96b/extracts?bookmark=MTYjg0NjMAYw4N"
}
Note that these results are paginated, so you should follow the link in the “Next” field to see more of the list of data sets and repeat that process until “Next” is null.
Retrieve a Data Set Extract File
Once you have chosen a data set, a version of that data set (differential or full), and one of the available extract files for it, there is only one step remaining – retrieving the extract file.
The API route /d2l/api/lp/1.43/datasets/bds/(schemaId)/plugins/(pluginId)/extracts/(extractId) will retrieve your chosen extract file. You should get the full API route with all parameter values filled in (schemaId, pluginId, and extractId) by using the “DownloadLink” link for your chosen extract file in the response from the API shown in the previous section of this article.
Need further help working with data? Stuck on any of the above steps? Lacking one of the skill sets needed? D2L provides flexible levels of Learning Analytics services that can assist with any or all steps in the process detailed in this post. If you are interested, please contact your D2L representative.
This article was originally published February 21, 2019 and has been updated to reflect our current processes.