[ome-users] 1. Slow upload of large sets of files 2. VTK files?

[Chris Allan]

Hi,

On 28 Jan 2013, at 21:26, "Sluka, James Patrick" <jsluka at indiana.edu> wrote:

> New user here exploring the use of OMERO to log the output from multicell modeling projects. Multicell computational models generate data sets that are basically images (2D, 3D and 4D). It would be very useful to have an archiving system, such as OMERO, that can capture the large number of files a simulation generates and that can also capture metadata associated with the simulations, for example the complete script that generated the simulation.
> 
> We have installed OMERO server on a university server. I am testing loading simulation output files into the repository from Windows and Ubuntu PCs. Two issues have come up:
> 
> 1. Uploading a large set of PNG files (~30K files total, 5KB/file) takes a long time, roughly two minutes per file on an Ubuntu system. Uploading a much smaller list of the same types of files is much faster, a couple seconds per file. It appears the very large number of files is making the time per file very large. Any thoughts on what might cause this?

That's really unfortunate. I expect this will be down to large amounts of directory scanning.

Would you be able to put up the files somewhere for us to work with? How are you actually performing the imports? What type of filesystem are the PNGs located on? Are they in the same directory? If not what does the file layout look like?

> 2. Any thoughts on how VTK files might be loaded into OMERO? In particular, VTK STRUCTURED_POINTS format files, but any VTK file format would be helpful. We use VTK files to store not only the assignment data that puts a given pixel (or voxel) into a particular cell of particular cell type but to also log pixel values for diffusible fields such as oxygen.

It should be possible to upload a VTK structured point file as an attachment to an OMERO Image and then write an OMERO.script which parses that file and produces OMERO regions of interest. That would give you visibility of the contents of the VTK structured point file in OMERO.insight and OMERO.web.

Bio-Formats has no current native support for any VTK file formats. If you were to provide us with one or more of your VTK structured point files and corresponding data file we should be able to guide you in the right direction.

> 
> Does anyone have any experience using OMERO for this type of application (logging of simulation images)?
> 
> Thanks for any guidance folks can provide.

You can upload a ZIP of your PNG or other format source data and VTK files to:

http://qa.openmicroscopy.org.uk/qa/upload/

-Chris