[ome-users] Bulk downloading data (was: Follow up from OME meeting)

[Simon Li]

Hi David

I’ve just checked and it’s easier than I expected, there’s a method im.getImportedImageFiles() which returns all the original files relating to an image. I thought I’d update my script https://gist.github.com/manics/5720713 so now it takes an additional -o flag.

Hopefully this is enough to get you started!

Simon

On 9 Jun 2014, at 12:38, Mason, David [dnmason] <D.N.Mason at liverpool.ac.uk<mailto:D.N.Mason at liverpool.ac.uk>> wrote:

Hi Simon,

Thanks for the quick reply, I don't need this functionality immediately but it's a (surprisingly) huge deal when reassuring people about committing their data fully to OMERO.

No problem about posting on the mailing list, I was going to put something on the forum once I had a workable solution.

I had a look at the python script and I can cope with having to pass a list of Project IDs, but could you recommend a quick change to download in the original format instead of OME.tiff? From my read, it's something around line 147:

        size, data = im.exportOmeTiff(1024 * 1024)

What would be the correct term here to use the original format/dimensions?

Thanks for all your help,

Dave

________________________________
From: Simon Li [s.p.li at dundee.ac.uk<mailto:s.p.li at dundee.ac.uk>]
Sent: 09 June 2014 12:12
To: Mason, David [dnmason]
Cc: Josh Moore; Jean-Marie Burel
Subject: Re: Follow up from OME meeting

Hi David

It was great to talk with you too. I've got a Python script that can recursively download a list of projects/datasets/images as ome-tiffs specified on the command line:
https://gist.github.com/manics/5720713

It was originally written for OMERO 4.4, hence the reason for exporting the images as OME-TIFFs, but it should be possible to change this to download the original image files in 5.0. It requires a list of project IDs to be passed on the command line rather than listing them automatically, but this would also be fairly easy to change.

It's not exactly user-friendly at the moment since it was for my own use, and there are several limitations, such as limited error handling and limits on the maximum size of image that can be retrieved. With that in mind if you still want to try running it you'll need to create a plain text file (e.g. 'login.config') containing the login details:

omero.user=username
omero.pass=password
omero.host=hostname

Then in your shell
export ICE_CONFIG=path/to/login.config
export PYTHONPATH=/path/to/OMERO.server/lib/python:$PYTHONPATH

Help: python batch_download_ometiff.py -h
Download project-ids 1 and 2: python batch_download_ometiff.py -p 1 -p 2
Dry-run, show what would be downloaded: python batch_download_ometiff.py -p 1 -p 2 -n

How soon do you require the ability to bulk export a user's data- is it something you're looking at in the longer-term or something you require now?

Finally, would you mind if I forward this to the ome-users mailing list for the benefit of anyone else who might be interested?

Best wishes

Simon


On 9 Jun 2014, at 11:01, "Mason, David [dnmason]" <D.N.Mason at liverpool.ac.uk<mailto:D.N.Mason at liverpool.ac.uk>> wrote:

Hi Simon,

Nice to talk to you at the OME meeting. I just wanted to follow up with what we were discussing regarding bulk export of files from OMERO, whilst retaining Project/Dataset structure. I would like to specify a user, then have every project, dataset, and file downloaded into the same [Project>Dataset>File] structure. I don't care about preserving annotations. Having attachments would be useful but not necessary.

To give you some context for a reply, I'm running my server on ubuntu (12.04LTS), and am well versed in bash, with (presently) little knowledge of python or Java (although I'm happy to run scripts in these languages).

Please let me know if you need any more details, thanks for your help so far,

Dave Mason


The University of Dundee is a registered Scottish Charity, No: SC015096


The University of Dundee is a registered Scottish Charity, No: SC015096
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-users/attachments/20140610/dc708fa3/attachment.html>