[ome-devel] [ome-users] In place imports and omero 5

[Josh Moore]

Hi all,

> On Wed, Sep 11, 2013 at 8:17 PM, Anatole Chessel <ac744 at cam.ac.uk> wrote:
>> 
>> Hi all,
>> 
>> 
>>    OMERO 5 looks really great and I been meaning to try it, but as
>> discussed in Paris, one of the crucial things for us would be to be able
>> to import files 'in place', ie without copying them (we now have around
>> 35Tb of screening data, for example...). It has been said it's a planned
>> feature, but would you have more details? In particular can I ask is
>> there any ETA on it?

Not at the moment, no. The next planned "feature" is to provide upgrades from the 4.4.x series (5.0.0-Beta2) 


>> Would there be a 'quick & dirty' way to do it rather easily with the
>> current version? There is a thing or two for us for which it could
>> really come in handy (even in a beta stage).

We might be able to come up with something with your help. Could you provide a recursive file listing (e.g. ls -lhR) of the data you want to in-place import, off-list?


>> Many thanks...
>> Anatole


On Sep 11, 2013, at 10:00 PM, Graeme Ball wrote:

> Hi,
> 
> I've been thinking about this again, and in case no-one else thought
> of the following approach already, here goes:-
> 
...
> 
> OMERO doesn't really need to own the data, since it basically treats
> data as read-only (right?) Upon import it just needs to put the
> metadata in its database, and make a note of where the data are the
> filesystem, i.e. assuming they are in a location it has read access
> to.

In general (caveat, caveat, caveat...), this will likely work. However more than just OMERO treating the fileset directories as read-only, they should be considered read-only *everyone*. Otherwise, the representation of the data may change as other files are added/modified.


> Obviously the user would have issues if they subsequently moved the
> data... The easiest way to tackle this problem without giving OMERO
> root access to the filesystem is to mark the data as read-only after
> import, which the user could either do manually (quick hack); or from
> inside the OMERO client, since it is already running with the user's
> credentials.

OMERO's not running with the user's credentials typically. If yes, then you're right, Graeme. If not, then I would say the rule-of-thumb is that the data should have already been made read-only before importing-in-place.


> Not foolproof, but then what is?

Nothing!

> Regards,
> Graeme

Cheers,
~Josh.