This is a console application that has a hard coded list of service directory base URLs that are then read and put into a single database instance. However only the web service data that matches the
is imported to ensure it matches the database structure. The database is wiped on each export.
Lessons Learned
Child data ids need to be converted either to unique ids within the database or guides and then universally updated throughout the imported data, which significantly complicates data combining. I think the question here is should all ids be guid from the start as otherwise there will always need to be a conversion to a unique id.
are included in the results, this ensures it can be worked out the structure correctly.
Foreign keys are then turned into constraints to ensure that any values added into a foreign key column match the column that it should be referencing.
A customisable config file has also been created that allows columns to be hidden from the export, comments to be added to column headers. As well as additional columns that can be defined to provide features such as a lookup on column values, and concatenation of values to help with the understanding of the data.
The extent column is excluded from the spreadsheet due to the size of the column.
reads all data in the service directory to determine if it is following the standard; this is because some values may not be output because they are null.
The web service checks that it performs are:
The service is paginatable with the form: /services?page={nn}
The service has a detail page with the form: /services/{id}
It then applies the following rules to each column were the