[ome-devel] batch image export

[Yanling Liu]

Hi Will,

Thanks for creating the python script. It works as expected.

I'll create a cron job from your script to delete batch exported attachment
files in a weekly basis. It would be very nice if OMERO has this built in
and provides one option to set clean up frequency.

Thanks,
Yanling


On Thu, Sep 4, 2014 at 6:38 AM, William Moore <will at lifesci.dundee.ac.uk>
wrote:

> Hi Yanling,
>
> Using Insight to download is probably better for large files than web.
> There's no size limit that I'm aware of.
> Is it 'not runnable on Mac' because of Java 8? If so, the recent 5.0.4
> release should fix it.
>
>
> I created a little script for deleting all 'Batch Image Export' attachments
> https://gist.github.com/will-moore/a23dc5ffa6a5394bae89
>
>
> # we can also get Files by name - but this won't have the linked file
> annotations
> files = conn.getObjects("OriginalFile", attributes={'name':
> 'Batch_Image_Export.zip'})
> for fa in files:
>     print fa.id, fa.getName()
>
>
> # For most flexibility, use the Query Service
> params = omero.sys.ParametersI()
> params.addLong('size', 4000)    # 4 kb
> params.addString('name', 'Batch_Image_Export.zip')
>
> query = "select an from Annotation as an join fetch an.file as f" \
>     " where f.name=:name and f.size>:size"
>
> # NB - since we are doing a direct query here, we get omero.model Objects
> returned...
> fileAnns = conn.getQueryService().findAllByQuery(query, params,
> conn.SERVICE_OPTS)
> for fa in fileAnns:
>     # ... and need to use .val to unwrap attributes
>     # can use the fileAnnotation Id to delete it as in the gist example
> above
>     print fa.getId().val, fa.getFile().getName().val
>
>
> # Final example of filtering by time
>
> params = omero.sys.ParametersI()
> d1 = datetime.strptime("2014-09-04 00:00:00", "%Y-%m-%d %H:%M:%S")
> created = time.mktime(d1.timetuple())*1000
> params.map["start"] = omero.rtypes.rtime(created);
>
> query = "select an from Annotation as an join fetch an.file as f" \
>     " join fetch an.details.creationEvent as ce" \
>     " where ce.time>:start"
>
> # do query as above
>
>  Hope that helps,
>
>    Will.
>
>
>
> On 3 Sep 2014, at 21:44, Yanling Liu <vrnova at gmail.com> wrote:
>
> Hi Will,
>
> Thanks for your reply.
>
> Is there a way to query OME for a list of attachment for name, date, and
> size? Could I remove unneeded attachment as the root user from command
> line? I see this may need to be run periodically to remove old attachment
> files as a system maintenance procedure.
>
> Would it be better to download image using the insight tool? Any size
> limitation in the insight tool? I can not test it as the OME client is
> currently not runnable on Mac system.
>
> Thanks,
> Yanling
>
>
>
> On Wed, Sep 3, 2014 at 4:25 PM, William Moore <will at lifesci.dundee.ac.uk>
> wrote:
>
>> Hi Yanling,
>>
>>  Yes, the Batch Image Export zip is permanently attached to the dataset
>> until someone deletes it.
>> This functionality was developed as a script (same script can be run in
>> both clients) but
>> unfortunately we don't have a nice way for the export script to pass the
>> zip back to the client
>> once it's finished. So attaching it is the only option.
>>
>> The main limitation we have with downloading a whole Project or Dataset
>> at a time is that
>> this could easily exceed a practical size of data to download.
>> Particularly in the web, where all the data for a single download needs
>> to be collated into
>> a single directory, zipped up and downloaded in one shot.
>> The size of this zip, and the time it takes to generate the zip need to
>> be compatible with http requests.
>> E.g. the web server (apache etc) will usually time-out after 1 or 2
>> minutes, and will struggle
>> to download multi-gigabyte files.
>>
>> I guess we could add the option to download a Dataset at a time, and
>> provide good indication
>> to the user of the size of the download before-hand, so they could
>> reconsider if it looks too big?
>>
>>   Regards,
>>
>>    Will.
>>
>>
>>
>> On 3 Sep 2014, at 21:17, Yanling Liu <vrnova at gmail.com> wrote:
>>
>> My mistake, what I mean "the mouse right click" is not right click, it
>> should be selecting images and then click the down arrow at the right side
>> to download images.
>>
>> It would be great to be able to select one project or data set and click
>> the down arrow at the right side to download.
>>
>> Thanks,
>> Yanling
>>
>>
>>
>> On Wed, Sep 3, 2014 at 4:08 PM, Yanling Liu <vrnova at gmail.com> wrote:
>>
>>> Dear OME developers,
>>>
>>> After running batch exporting script, the zipped images is linked as
>>> attachment to the dataset being exported. My question is how long will this
>>> attachment live in the system? If user forget to delete the zip file after
>>> downloading, will it be kept in the system forever? If so then potentially
>>> people can generate lots of zip packages and waiting storage space.
>>>
>>> The mouse right click and download feature is a much better solution.
>>> However, for now only images can take advantages of this solution, will it
>>> be extended to dataset and project so user can mouse right click on one
>>> dataset to download as original files?
>>>
>>> Thanks,
>>> Yanling
>>>
>>
>> _______________________________________________
>> ome-devel mailing list
>> ome-devel at lists.openmicroscopy.org.uk
>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-devel/attachments/20140905/e65880c1/attachment.html>