Handling of metadata headers for tape operations (archiving and staging) #8011
Description
Dear dCache developers,
Following the discussions from the WLCG OTF #8, I'd like to document the feature request of handling metadata in dCache, which is provided by experiments with FTS requests via headers.
This would allow for more flexibility at sites to organize archiving to (via archival_metadata, see contribution) and staging from (via staging_metadata, see contribution ) tapes.
During the OTF it was noted by ATLAS, that they would like to get rid of the current implementation for KIT of providing metadata for writing to tapes, since this causes problems with data catalog integrity during deletion campaigns. If I understood it correctly, then it would be best to do that at some point this year for them already.
In consequence, we at KIT would then loose the possibility of our special treatment of large datasets from ATLAS (in particular for sizes from 0.5 to 1.X PB) when such archival requests appear at our system. This is a bit alarming to us, so we would be very happy, if you could introduce metadata handling in the dCache software also at some point this year.
Some considerations from our side on this feature request:
- To avoid too much of performance issues, we wouldn't need the metadata attached to the requests stored in dCache database/namespace for all times. We only need it for the lifetime of an archival/staging request of a particular file/bulk.
- To be able to pick up on the metadata in our interfaces, it would be good to pass the metadata down to the HSM provider used by our systems
- Since at KIT, we are making use of the Script HSM Provider for writing files to our tape system (to be more precise, to the shared disk buffer of our tape system), it would be good in addition, if this very provider is also able to handle the metadata and passes it to the script to be executed via arguments.
Of course, we at KIT are willing and volunteer to test any of your developments in this direction. In addition, if any further help is needed, I'm happy to provide support also from my side in addition.
Thank you very much advance for considering this feature request!
Best regards,
Artur Gottmann for the KIT storage admin team