[ome-users] Importing large image series and Cell^R data

[Colin Rickman]

Hi

Sorry for the delay. I've been testing a few things.

On 15 May 2013, at 11:13, Roger Leigh <r.leigh at DUNDEE.AC.UK> wrote:

> On 13/05/2013 20:48, Colin Rickman wrote:
>> Hi
>> 
>> I'm just testing our new OMERO installation and have a few
>> questions/problems with importing.
> 
> Which version of the OMERO client and server are you using?

The server is OMERO 4.4.8 with Ice 3.4 along with the current client launched using Java Webstart.


>> 1. I'm uploading a 740,000 frame tiff series with a total size just over
>> 90GB. Using the importer it says uploading 0 of 1 so I assume it has
>> decided to convert it to a single multidimensional ome.tif. After 15
>> hours it failed with the statement file invalid. My questions are, is
>> there a maximum upload size, is there a maximum ome.tif size, how can I
>> choose to upload all files of a series as a series rather than convert
>> them in to one file?
> 
> If the ome.tiff is a BigTIFF, then the maximum size will be the maximum
> size of BigTIFF (2^64).  If it's not BigTIFF, then it will be the normal
> 2^32 (4 GiB).
> 
> Is your 740,000 frame tiff series a single file, or multiple files?  Is
> it in any particular format which bioformats knows about ("showinf" will
> tell you)?   If it's multiple files, are these structured in any
> particular way (e.g. naming scheme) to give information about the
> dimensionality?

It is a folder containing a tiff series (ie 740,000 individual 16bit tiff files). It opens perfectly fine using FIJI and bioformats. The files are a time series with the name Frame_XXXXXX going from 000001 to 740000. 

> Can you reproduce the failure with a smaller series?

I've tested this and I can with what seems to be any number of individual files. In all cases the importer states "imported 0 of 1" and sits scanning indefinitely. However, I did try entering the directory in the import window and selecting all of the individual tiff files (a 10,000 file subset) and after a long wait to start the importer stated "imported 0 of 10,000" and slowly started working through the files - it is currently just over half way through. 

> We can certainly look at why the series fails to import, but we'll need
> a copy of the original data for that, and 90GB is quite large.
> 
>> While this was uploading I attempted to convert the series to an ome.tif
>> in FIJI and this failed at just over 4GB. FIJI was able to convert the
>> series to a single .tif without any problems.
> 
> This is a known bug; we currently don't automatically enable BigTIFF
> writing when the data is too big for normal TIFF, and we don't provide a
> means to enable it by hand.
> 
> http://trac.openmicroscopy.org.uk/ome/ticket/6589
> 
> You can certainly convert using "bfconvert" and its "-bigtiff" option as
> a workaround for the time being.
> 
>> 2. My other problem is uploading Olympus Cell^R/Cell Excellence data.
>> Bioformats in FIJI seems to understand the database structure ok but in
>> OMERO it uploads some of the data and names them after the database file
>> names (e.g. xxxx.apl now contains image data) rather than the data names.
> 
> I'm afraid there's not much information to go on here.  We'll need some
> more detail about exactly what you observed and what you expected.  If
> you wish to provide us with an example, you can zip the data up and
> upload it using http://qa.openmicroscopy.org.uk/qa/upload/ and we can
> look at it in more detail.

I've uploaded a test data set to the link above. I also tested uploading the same dataset to our OMERO server. The dataset consists of *.apl, *.mtb and *.tnb files which contain all of the metadata etc. In a separate subdirectory are the multidimensional tiff files. Importing the whole dataset sort of worked. In the example I sent you the resulting upload contained 16 image stacks (the correct number). However,  one of the .tif files was now given the *.apl file name and the metadata was not imported fully. All of the remaining .tif files had their cryptic name used by the database rather than their human understandable name given in the metadata. There were a couple of errors reported trying to access this dataset on the OMERO server and these have been reported under my email address.

Hope that helps. Let me know what else you need to know.

Colin

Dr Colin Rickman
Institute of Biological Chemistry, Biophysics and Bioengineering
School of Engineering and Physical Sciences
Heriot-Watt University
Edinburgh
EH14 4AS

Tel: +44 131 4514193 (Office)

http://www.ib3.eps.hw.ac.uk
http://www.esric.org





----- 
Sunday Times Scottish University of the Year 2011-2013
Top in the UK for student experience
Fourth university in the UK and top in Scotland (National Student Survey 2012)

We invite research leaders and ambitious early career researchers to 
join us in leading and driving research in key inter-disciplinary themes. 
Please see www.hw.ac.uk/researchleaders for further information and how
to apply.

Heriot-Watt University is a Scottish charity
registered under charity number SC000278.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-users/attachments/20130521/13d5f4dc/attachment.html>