[ome-users] Import speed

[Paulo Almeida]

On Tuesday 12 February 2008 00:03, Jason Swedlow wrote:
> Hi Paulo-

Hi Jason,

> Import speed with OMERO  varies with the file format, and it can be a
> problem.   For OMERO to provide images for visualization, it requires
> knowledge of the underlying metadata (min max for each channel,
> etc.).  In some file formats, this is available in the file metadata
> (and is correct!), so we use that.  In other file formats, the data
> is not available, so we have to calculate it.  This really slows
> import down, because there is real calculation to do besides simply
> copying the file.

>
> Note also that there are number of fixes in Beta-2.3 that will help
> server performance.  Just in case, what kinds of file formats are you
> having problems with?

I only tested STKs, because I was just thinking of file sizes. I suspected the 
importer was acquiring metadata, but I wasn't sure. I will test Beta-2.3 and 
see how it improves.

> We are working on a general solution for image import, that we hope
> will mean that physically importing the data into the server is not
> necessary.  This will be especially helpful for installations where
> there is already a substantial amount of data and importing into an
> OMERO server is prohibitive.  This won't be available til later this
> year.

That is an interesting solution because the users can then quickly copy their 
images to a file server and later import them to OMERO. The downside is that 
they have to do two operations (at least with the way things work here), but 
you can't have everything. I would say researchers have to weigh the time 
they will lose with the new system and the time and safety they will win by 
having their images organized and stored, and see if it fits their needs. 
Maybe it is not so much a problem of time as it is of changing old habits. 


> We announced the Beta2.3 release several weeks ago on this list.  Let
> us know if you are not seeing the emails and we'll look into it.

I was on vacations until late January, so I probably just missed the 
announcement. I think I have been getting mail from the list normally

Thanks,
Paulo

>
> Cheers,
>
> Jason
>
> On 11 Feb 2008, at 20:05, Paulo Almeida wrote:
> > Hi,
> >
> > Some users have complained that it takes a long time to import
> > images with
> > OMERO, as opposed to copying images to the server. I suppose this is
> > inevitable, because it is not a network problem. For instance, I
> > could copy
> > files at 30 Mb/s to the OMERO server using SAMBA (and 40 Mb/s to
> > the server
> > where people store their images currently), but an OMERO import
> > took much
> > longer, at about 4 Mb/s.  I tried importing files from the the
> > server's hard
> > disk but that was not much faster. Do you have any thoughts on
> > this? Is it
> > possible that it is a problem with my setup, or is this to be
> > expected? This
> > is a real problem for us because researchers may not adopt OMERO
> > because of
> > this concern.
> >
> > (I am using OMERO 2.2 server and client. I just noticed 2.3
> > versions were
> > released)
> >
> > Thanks,
> > Paulo
> > _______________________________________________
> > ome-users mailing list
> > ome-users at lists.openmicroscopy.org.uk
> > http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>
> **************************
> Wellcome Trust Centre for Gene Regulation & Expression
> College of Life Sciences
> MSI/WTB/JBC Complex
> University of Dundee
> Dow Street
> Dundee  DD1 5EH
> United Kingdom
>
> phone (01382) 385819
> Intl phone:  44 1382 385819
> FAX   (01382) 388072
> email: jason at lifesci.dundee.ac.uk
>
> Lab Page: http://www.dundee.ac.uk/lifesciences/swedlow/
> Open Microscopy Environment: http://openmicroscopy.org
> **************************