[ome-devel] batch image export

William Moore will at lifesci.dundee.ac.uk
Thu Sep 4 11:38:07 BST 2014


Hi Yanling,

Using Insight to download is probably better for large files than web.
There's no size limit that I'm aware of.
Is it 'not runnable on Mac' because of Java 8? If so, the recent 5.0.4 release should fix it.


I created a little script for deleting all 'Batch Image Export' attachments
https://gist.github.com/will-moore/a23dc5ffa6a5394bae89


# we can also get Files by name - but this won't have the linked file annotations
files = conn.getObjects("OriginalFile", attributes={'name': 'Batch_Image_Export.zip'})
for fa in files:
    print fa.id, fa.getName()


# For most flexibility, use the Query Service
params = omero.sys.ParametersI()
params.addLong('size', 4000)    # 4 kb
params.addString('name', 'Batch_Image_Export.zip')

query = "select an from Annotation as an join fetch an.file as f" \
    " where f.name=:name and f.size>:size"

# NB - since we are doing a direct query here, we get omero.model Objects returned...
fileAnns = conn.getQueryService().findAllByQuery(query, params, conn.SERVICE_OPTS)
for fa in fileAnns:
    # ... and need to use .val to unwrap attributes 
    # can use the fileAnnotation Id to delete it as in the gist example above
    print fa.getId().val, fa.getFile().getName().val


# Final example of filtering by time

params = omero.sys.ParametersI()
d1 = datetime.strptime("2014-09-04 00:00:00", "%Y-%m-%d %H:%M:%S")
created = time.mktime(d1.timetuple())*1000
params.map["start"] = omero.rtypes.rtime(created);

query = "select an from Annotation as an join fetch an.file as f" \
    " join fetch an.details.creationEvent as ce" \
    " where ce.time>:start"

# do query as above

 Hope that helps,

   Will.



On 3 Sep 2014, at 21:44, Yanling Liu <vrnova at gmail.com> wrote:

> Hi Will,
> 
> Thanks for your reply.
> 
> Is there a way to query OME for a list of attachment for name, date, and size? Could I remove unneeded attachment as the root user from command line? I see this may need to be run periodically to remove old attachment files as a system maintenance procedure.
> 
> Would it be better to download image using the insight tool? Any size limitation in the insight tool? I can not test it as the OME client is currently not runnable on Mac system.
> 
> Thanks,
> Yanling
> 
> 
> 
> On Wed, Sep 3, 2014 at 4:25 PM, William Moore <will at lifesci.dundee.ac.uk> wrote:
> Hi Yanling,
> 
>  Yes, the Batch Image Export zip is permanently attached to the dataset until someone deletes it.
> This functionality was developed as a script (same script can be run in both clients) but 
> unfortunately we don't have a nice way for the export script to pass the zip back to the client
> once it's finished. So attaching it is the only option.
> 
> The main limitation we have with downloading a whole Project or Dataset at a time is that
> this could easily exceed a practical size of data to download.
> Particularly in the web, where all the data for a single download needs to be collated into
> a single directory, zipped up and downloaded in one shot.
> The size of this zip, and the time it takes to generate the zip need to be compatible with http requests.
> E.g. the web server (apache etc) will usually time-out after 1 or 2 minutes, and will struggle
> to download multi-gigabyte files.
> 
> I guess we could add the option to download a Dataset at a time, and provide good indication
> to the user of the size of the download before-hand, so they could reconsider if it looks too big?
> 
>   Regards,
> 
>    Will.
> 
> 
> 
> On 3 Sep 2014, at 21:17, Yanling Liu <vrnova at gmail.com> wrote:
> 
>> My mistake, what I mean "the mouse right click" is not right click, it should be selecting images and then click the down arrow at the right side to download images.
>> 
>> It would be great to be able to select one project or data set and click the down arrow at the right side to download.
>> 
>> Thanks,
>> Yanling
>> 
>> 
>> 
>> On Wed, Sep 3, 2014 at 4:08 PM, Yanling Liu <vrnova at gmail.com> wrote:
>> Dear OME developers,
>> 
>> After running batch exporting script, the zipped images is linked as attachment to the dataset being exported. My question is how long will this attachment live in the system? If user forget to delete the zip file after downloading, will it be kept in the system forever? If so then potentially people can generate lots of zip packages and waiting storage space.
>> 
>> The mouse right click and download feature is a much better solution. However, for now only images can take advantages of this solution, will it be extended to dataset and project so user can mouse right click on one dataset to download as original files?
>> 
>> Thanks,
>> Yanling
>> 
>> _______________________________________________
>> ome-devel mailing list
>> ome-devel at lists.openmicroscopy.org.uk
>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-devel/attachments/20140904/b67d4eb0/attachment-0001.html>


More information about the ome-devel mailing list