[ome-devel] Archive integrity check during OMERO Insight import

[Chris Allan]

Hi Alex,

On 28 Jun 2012, at 15:00, Alex Herbert wrote:

> Dear All,
> 
> When importing and archiving images into OMERO is there an integrity check on the archived data? For example a comparison between an MD5 sum on the original data on either side of the connection. I assume that the overhead of such a check would be far less than the current work of translating the pixel data into the OMERO format and the time taken to transmit it via the network.

The answer to your question is that we only do this (via a SHA1 sum) to the pixel data on import. There is provision to do it to the original files that are uploaded to the server via the "archive" process but we don't mostly for performance reasons. In fact there is even a column in the database for storing a SHA1 sum. Generating a reasonably secure cryptographic hash (such as MD5, SHA1 or SHA256) would reduce the upload performance several fold.

Some questions so that we could adequately scope adding this functionality, likely in a configurable manner, to future OMERO versions:

 * When you say that you'd like an integrity check are you expecting a hash digest on the client, then one on the server, then a comparison?

 * What sort of archiving performance burden would you be willing to accept? 2x slower? 4x? 10x?

 * How secure are you expecting the hash digest to be? Would a CRC be sufficient?

 * Are you expecting the hash digest to be stored in the database? If so are you worried about collisions?

> Currently we archive all images from our microscopes into OMERO. We would like to be sure that the archive has been successfully transferred before deleting the original from the local filesystem.
> 
> I have had people ask me about the archive integrity and would like to find out the answer.
> 
> Thanking you in advance.
> 
> Alex

-Chris