Ig 447

Created by: PaulMcCaffertyPA

Checkin of caching code.

The run will first get a count of the MDC datasets and a count of the datasets in HDR DB, if lower than 10% it will not continue unless there is an override. It will then pull the dataset details from the MDC and also pull the metadata quality info in. Next it will cycle through the list of datasets, call the MDC for the schema.org data and then decide if it needs to update the current dataset info in the HDR DB or add a new entry. While it is going through the list of MDC datasets it records the ID, after it has finished going through the MDC list it compares it to a list of IDs from HDR DB, if there is one that appears in the HDR list but not the MDC list then it then changes that state to archived so it will no longer appear on the search - when MDC have released their update we can look at this code again.