Discussion and ideas to archive products done with a previous version of a pipeline and allow to create new versions.
Objective, definitions
Until now, most of our pipelines changes could be labeled as "revisions" : scripts were improved but results were expected to remain the same.
We aim now to deal with "versions" changes where a script modifications can include complete workflow overload and different (hopefully better) results.
In the event of a pipeline version change, it is expected that pre-existing products could be re-created using the newer pipelines, while keeping the previous result as an "archived" state.
It is not expected for archived products to be advertised in the interface, but they *must* be accessible for the purpose of reproducible science.
Design
Archived products accessibility
Archived products are to be explicitly accessed by productids. Archived versions of a same productid should be distinct by version, keep their creation time and also present an archival time.
As stated before, while archived product access is mandatory, a user should not be exposed to "deprecated" versions in their day-to-day workflow. Archive presence will not be exposed in the current interface elements such as map/search[1]/workspace view.
[1] Maybe with a specific keyword?
Dependencies
MarsSI products have a dependency fields since the major DB rewrite. Archived product must reference their dependencies, be them active or archived products.
When a product is archived, all product that depends on it must be archived. TODO: choose version to use for those derived products.
Implementation
Data
Files are to be moved to /data/archive/$TARGET/$INSTHOSTID/$INSTID/$VERSION (ex /data/archive/MARS/MRO/CTX/CTX_008716_1998_008782_1999/V1)
Create an archive job and trigger it from the database? Less efficient but cleaner and safer than managing DB on one hand, CPs on the other.
Database
Rework a bit the Product model:
- Create a BaseProduct abstract model: hold common fields
- Product keep the name, inherit BaseProduct and status/depends fields, most methods
- Create an ArchivedProduct modeI, BaseProduct + version fields
Web
Provide a set of search/display interface that allow for copy operations, like workspace?
