Feature - Explore the CKAN open data portal API

[stuartlynn]

Currently the only open data portals that scout indexes are ones run by Socrata. We would love to extend this to other open data portals starting with CKAN.

For a given CKAN portal we need to be able to obtain a list of datasets available on that portal and then map the meta data for those datasets in to the same standard format we are using for the Socrata datasets.

The function that does this for the socrata datasets can be found in the code here:

scout/src/utils/socrata.js

Lines 179 to 220 in 86cab19

	export const datasetToDB = (dataset) => {
	const { resource, metadata, classification } = dataset;
	const domain_metadata = classification
	? classification.domain_metadata
	: null;
	const updatedAutomation = domain_metadata?.find(
	({ key, value }) => key === 'Update_Automation' && value === 'No',
	)?.value;
	const updateFrequency = domain_metadata?.find(
	({ key }) => key === 'Update_Update-Frequency',
	)?.value;
	const department = domain_metadata?.find(
	({ key }) => key === 'Dataset-Information_Agency',
	)?.value;
	return {
	id: resource.id,
	name: resource.name,
	portal: metadata.domain,
	columns: resource.columns_name.map((c) => c.trim()),
	columnFields: resource.columns_field_name.map((c) => c.trim()),
	columnTypes: resource.columns_datatype,
	metaDataUpdatedAt: resource.metadata_updated_at,
	updatedAt: resource.data_updated_at,
	createdAt: resource.createdAt,
	description: resource.description,
	views: resource.page_views.page_views_total,
	categories: classification.categories,
	domainCategory: classification.domainCategory,
	tags: classification.domain_tags,
	type: resource.type,
	updateFrequency,
	department,
	permaLink: dataset.permalink,
	parentDatasetID: resource.parent_fxf[0],
	updatedAutomation,
	owner: dataset.owner.display_name,
	};
	};

Some useful resources for this:

• API Docs : The CKAN API docs
• ckan javascript client : A javascript client for the API
• ckanapi : A python package for interacting with a CKAN portal

Note, it makes sense to add this to the scout-multi branch rather than the main branch

[jps327]

Switching to "frozen" because there are no imminent plans on scoping, but this is in the roadmap for EOY.