This article introduces using the Data Hub APIs to retrieve an Advanced Data Set and gives a practical example using the process.
Data Hub houses data exports from Brightspace. Data Hub contains a selection of data set that authorized users can run and export to CSV.
Once Data Hub has been enabled for your Brightspace instance, you can access it in Brightspace or via API.
What information can I currently get from Data Hub?
Currently, Core customers can access seven Advanced Data Sets – All Grades, Course Learning Outcome Evaluation (CLOE), Instructor Usage, Learner Usage, Impersonated Session History, Enrollments and Withdrawals and Final Grades. Additionally, Brightspace Data Sets are also located in the Data Hub but are outside the scope of this article. Refer to How to get started with Data Hub's APIs: Brightspace Data Sets for additional information.
How do I access Data Hub?
Data Hub can be accessed in Brightspace. A Data Hub link can be added to your navbar for additional convenience. Alternatively, you can also navigate to /d2l/datahub/dataexport/list in your browser.
Data Hub can also be accessed via API. The following sections will walk you through accessing Data Hub through the API.
What is the expected flow when using Data Hub's API?
Note: Authorization is not covered in this section. See the code walkthrough below or the refresh token blog post for more information.
The routes below are documented within our API reference documentation.
First, decide which data set you would like to run. You can use /d2l/api/lp/1.13/dataExport/list to see all available data sets. An example response may look like the following:
[
{
DataSetId: "ff842aae-84d0-4e39-9db4-3ae51d36bb0e",
Name: "Final Grades",
Description: "The Final Grades dataset returns final grades for all learners in course offerings.",
Category: "AdvancedDataSets",
Filters:
[
{
Name: "startDate",
Type: 1
},
{
Name: "endDate",
Type: 1
}
]
},
{
DataSetId: "c1bf7603-669f-4bef-8cf4-651b914c4678",
Name: "Enrollments and Withdrawals",
Description: "An Enrollments and Withdrawals dataset consisting of Org Unit, User and Role attributes, along with enrollment status for a given date range.",
Category: "AdvancedDataSets",
Filters:
[
{
Name: "startDate",
Type: 1
},
{
Name: "endDate",
Type: 1
}
]
},
{
DataSetId: "be5e86b8-9723-4ad7-af63-86783a77a1a1",
Name: "All Grades",
Description: "The All Grades dataset returns the grades for all gradable items for all learners in course offerings.",
Category: "AdvancedDataSets",
Filters:
[
{
Name: "parentOrgUnitId",
Type: 2
},
{
Name: "startDate",
Type: 1
},
{
Name: "endDate",
Type: 1
}
]
},
{
DataSetId: "c195aa85-b2be-4444-aa52-570e19bfee9e",
Name: "Learner Usage",
Description: "The Learner Usage dataset returns the activity for all learners in course offerings.",
Category: "AdvancedDataSets",
Filters:
[
{
Name: "parentOrgUnitId",
Type: 2
},
{
Name: "startDate",
Type: 1
},
{
Name: "endDate",
Type: 1
},
{
Name: "roles",
Type: 3
}
]
},
{
DataSetId: "76351c8d-e7dc-4b43-844a-14eb2c02193d",
Name: "Impersonated Session History",
Description: "The Impersonated Session History data set returns all impersonated sessions for a specific or all users over a date range.",
Category: "AdvancedDataSets",
Filters:
[
{
Name: "userId",
Type: 4
},
{
Name: "startDate",
Type: 1
},
{
Name: "endDate",
Type: 1
}
]
},
{
DataSetId: "af389020-439b-4d54-969e-edb9ff243a47",
Name: "CLOE",
Description: "The Course Learning Outcome Evaluation dataset returns the learner's outcomes for required competencies.",
Category: "AdvancedDataSets",
Filters:
[
{
Name: "parentOrgUnitId",
Type: 2
},
{
Name: "roles",
Type: 3
}
]
},
{
DataSetId: "df7d0ef3-bcab-4171-bc37-5929d5867cd8",
Name: "Instructor Usage",
Description: "The Instructor Usage report provides information about how instructors are using LMS functionality within their courses.",
Category: "AdvancedDataSets",
Filters:
[
{
Name: "parentOrgUnitId",
Type: 2
},
{
Name: "startDate",
Type: 1
},
{
Name: "endDate",
Type: 1
},
{
Name: "roles",
Type: 3
}
]
}
]
This response means that there are six data sets available and lists the filters used for each one. For example the CLOE dataset takes an input of ParentOrgUnitId and Roles for filters.
Say we want to run Enrollments and Withdrawals. We take the DataSetId from above ( c1bf7603-669f-4bef-8cf4-651b914c4678 ) and run the job by sending a POST to /d2l/api/lp/1.13/dataExport/create.
{
"DataSetId": "c1bf7603-669f-4bef-8cf4-651b914c4678",
"Filters": [
{
"name": "startDate",
"value": "2015-08-05T00:00:00.000Z"
},
{
"name": "endDate",
"value": "2016-08-05T23:59:59.999Z"
}
]
}
This will tell Data Hub to create an Enrollments and Withdrawals data set with data from August 5th 2015 to August 5th 2016.
Note: Only the UTC timezone is supported.
The create route will return an export job ID you can use to query the status of your export job.
{
"ExportJobId": "7699bad4-0ae1-4772-8960-390081d2feea",
"DataSetId": "c1bf7603-669f-4bef-8cf4-651b914c4678",
"Name": "Enrollments and Withdrawals",
"SubmitDate": "2016-08-05T18:28:47.130Z",
"Status": 0,
"Category": "Insights"
}
Using the identifier from creating the export job, we can view its status using /d2l/api/lp/1.13/dataExport/jobs/7699bad4-0ae1-4772-8960-390081d2feea.
{
"ExportJobId": "7699bad4-0ae1-4772-8960-390081d2feea",
"DataSetId": "c1bf7603-669f-4bef-8cf4-651b914c4678",
"Name": "Enrollments and Withdrawals",
"SubmitDate": "2016-08-05T18:28:47.130Z",
"Status": 2,
"Category": "Insights"
}
Note: The Status is now 2 which means the job completed successfully. In addition to querying the status of a specific job, we can also view all past jobs using /d2l/api/lp/1.13/dataExport/jobs.
Since the status of our export job is now Complete we can then download the data using the download route and the ExportJobId from above /d2l/api/lp/1.13/dataExport/download/7699bad4-0ae1-4772-8960-390081d2feea.
Code Example Walkthrough
We have created a sample application to help developers start using the Data Hub API. The following section will walk through important sections of the code.
To run the application, we first need a refresh token (obtained by following the instructions in the Refresh Token blog post). Once we have a refresh token, we need to trade it in for an access token.
First, read the refresh token from a file.
Path refreshTokenPath = Paths.get(refreshTokenFile);
String oldRefreshToken = Files.readAllLines(refreshTokenPath).get(0);
Then create our authorization string from our client ID and client secret. This lets the authorization service know which application we are.
String authHeaderValue = getClientAuthHeaderValue(clientId, clientSecret);
Then we create the request and POST to the authorization service.
List<NameValuePair> payload = Form.form()
.add("grant_type", "refresh_token")
.add("refresh_token", oldRefreshToken)
.build();
HttpResponse response = Post(tokenEndpoint)
.setHeader(HttpHeaders.AUTHORIZATION, authHeaderValue)
.bodyForm(payload)
.execute()
.returnResponse();
The authorization service returns the access token to us. We will send this token with our Data Hub API requests.
JsonObject responseJson = httpResponseToJsonObject(response);
String accessToken = responseJson
.get("access_token")
.getAsString();
Lastly, we save the new refresh token to file. The next time we run the application, we will need this refresh token to get another access token.
String newRefreshToken = responseJson
.get("refresh_token")
.getAsString();
Files.write(refreshTokenPath, newRefreshToken.getBytes());
This sample application assumes we have already called the route to see all data sets. For completeness, we want to make sure the data set (in this case, Enrollments and Withdrawals) exists. We do this by querying the Data Hub API using the data set ID.
private static void assertDataSetIdExists(
String accessToken,
String hostUrl,
String dataSetId) throws IOException {
makeGetRequest(accessToken,
format("%s/d2l/api/lp/1.13/dataExport/list/%s", hostUrl, dataSetId),
format("cannot find the specified data set [dataSetId = %s]", dataSetId));
}
Now that we know the data set exists, we can submit an export job to be processed. We do this by sending a POST to the Data Hub API containing information about which data set we want data for and what dates the data is filtered by. The Data Hub API will return information about the submitted export job, including the export job ID.
private static String submitExportJob(
String accessToken,
String hostUrl,
String dataSetId,
String startDate,
String endDate) throws IOException {
String bodyString = format(
"{\"DataSetId\": \"%s\", \"Filters\": [{\"Name\": \"startDate\", \"Value\": \"%s\"}, {\"Name\": \"endDate\", \"Value\": \"%s\"}]}",
dataSetId, startDate, endDate);
HttpResponse response = makePostRequest(
accessToken,
format("%s/d2l/api/lp/1.13/dataExport/create", hostUrl),
bodyString,
format("cannot create job [requestBody = %s]", bodyString)
);
String exportJobId = httpResponseToJsonObject(response)
.get("ExportJobId")
.getAsString();
return exportJobId;
}
With the export job ID from the previous step, we can periodically check the status of our export job using the Data Hub API. The function below is called every 10 seconds to ensure that the job is either still queued, processing, or complete.
private static int doPollForCompletedExportJob(
String accessToken,
String hostUrl,
String exportJobId) throws IOException {
HttpResponse response = makeGetRequest(
accessToken,
format("%s/d2l/api/lp/1.13/dataExport/jobs/%s", hostUrl, exportJobId),
format("cannot check the export-job status [exportJobId = %s]", exportJobId)
);
int status = httpResponseToJsonObject(response)
.get("Status")
.getAsInt();
assertValidStatus(exportJobId, status);
return status;
}
Finally, our export job has completed successfully. We can now download the ZIP file containing our CSV using the Data Hub API.
private static void downloadCompletedExportJob(
String accessToken,
String hostUrl,
String exportJobId,
String outputFolder) throws IOException {
File tempFile = createTempFile(
exportJobId,
".zip",
new File(outputFolder)
);
try (FileOutputStream fileOutputStream = new FileOutputStream(tempFile)) {
makeGetRequest(
accessToken,
format(
"%s/d2l/api/lp/1.13/dataExport/download/%s",
hostUrl,
exportJobId
),
format(
"cannot download the export job [exportJobId = %s]",
exportJobId
)
)
.getEntity()
.writeTo(fileOutputStream);
}
}
Need further help working with data? Stuck on any of the above steps? Lacking one of the skill sets needed? D2L provides flexible levels of Learning Analytics services that can assist with any or all steps in the process detailed in this post. If you are interested, please contact your D2L Customer Success representative.