Automatic Update use case.
A fast moving challenge was presented by a Major Broadcaster and Media client to enhance the previous 2 services provided in order to produce update packages based on detection of metadata updates within the Gracenote On Api.
Initial SOW and Challenges.
The client SOW was to create a software service that could actively poll the Gracenote OnApi services in order to detect metadata updates and generate valid ADI ingest packages; once generated the packages should be fed into the previous Enrichment service as outlined in the following blog post: Gracenote Metadata Enrichment
The main challenges during the initial concept are outlined below:
- Api results limit to ensure correct amount of data is retrieved.
- Ensuring results retrieved from the API are filtered for relating items while reducing impact to existing services.
- Ensure the next poll event uses a the next valid update id for data retrieval.
- Ensure all 3 layers of the api are checked for updates and ensure no duplicates are processed within the same update across the 3 layers.
- Ensure the any id changes such as TmsId are reflected across the database tables correctly.
- Ensure new ingests and Content provider updates add the correct data to the new Data tables used to track the auto enrichment processes.
- Ensure that no overlap between the software services occur to again prevent duplication.
These were the core challenges faced during design and implementation of the core service and as development continued more and more smaller items were encountered and effectively resolved.
Solutions Approach and process.
The software was created using C# .net with MSSQL database to store data and configuration.
Utilising the previous database from the Gracenote Metadata Enrichment process, the implementation was to create new tables in order carry out the following processes:
- Map the correct fields needed from each of the 3 Gracenote layers into its corresponding tracking table.
- Allow for a tracking table independent of the layer tables in order to hold the last id used against each Gracenote API call to each layer.
- Allow for a flag in the tracking table to set when the Auto updates service is running in order to prevent the Package generation service from creating packages during an update operation.
Two services were created in order to achieve delivery of updated packages:
- Initial updates service.
- Polls Gracenote using a configurable limit and next available update id.
- Sets a boolean flag in the correct layer table in the database to indicate a package has new metadata available.
- Package Creation service.
- Polls the database when the Updates service running flag is false.
- Builds a list of all items requiring an update.
- Takes the previously stored ADI update data from the existing database.
- Use the retrieved Adi data to create a new update ADI template.
- Package the ADI file and deliver to the Gracenote Metadata Enrichment service.
Although a large amount of low level operations are occuring that are not mentioned here, the services are able to poll the Gracenote API and generate updates in record time and can pass in 100’s of updates to the Gracenote metadata Enrichment service in less than a minute.
The core workflow software is built in a modular fashion allowing libraries to be updated independently of the executable process which allows more flexibility and speed.
The project has now grown from a single requirement to provide metadata enrichment services, then allowing migration of QAM content into ADI format and enrichment to feed into this service; now we have 2 more additional services in order to detect, process and generate Proactive Metadata updates thus becoming a complete service platform.
The project allowed for 8 weeks of development, however the initial deployment was produced and delivered inside of 5 weeks, which was a great position to be in as it allows extra time for any PR’s and possible CR requests to be implemented and tested all within the original timeframe!
The net result is the customer now has an end 2 end solution that can take in CP (content provider) delivered files and have these Pre-Enriched by the SCH Tech Gracenote Metadata Enrichment Service, while allowing automatic updates to occur if images or metadata are updated by Gracenote.