[ome-devel] Omero server performance issue

[Luca Lianas]

Hi Chris

thanks for your advice. As suggested by you, I modified the script and
created only 4 LogicalChannelI objects linked to the image's channels (I
looked into the xml model file and found the many-to-one relationship
between channel and logicalchannel), now the execution time takes only 13
minutes.
I attach the script in order to provide an example if anybody needs one.

Luca


2009/8/5 Chris Allan <callan at blackcat.ca>

> Hi Luca,
>
> One big thing is to be very careful with enumerations:
>
> 26       do = om.model.DimensionOrderI()
> 27       do.setValue(omt.rstring('XYZCT'))
> 28       pt = om.model.PixelsTypeI()
> 29       pt.setValue(omt.rstring('uint16'))
>
> This is creating brand new enumerations for every image inserted, which
> will slow any enumeration queries to a crawl and is causing 3000 extra
> INSERTs, significant graph inspection overhead, etc. You want to
> retrieve the existing enumerations through IQuery and then use an
> unloaded version of the object to help you out, for example
> (pseudo-code):
>
> ...
> dimension_orders = iquery.findAll("DimensionOrder")
> xyzct = filter(lambda a: a.value.val == 'XYZCT')[0]
> syzct.unload()
> ...
> for image in range(0, 100):
> ...
>    p.setDimensionOrder(xyzct)
>
> The above applies to FormatI, DimensionOrderI and PixelsTypeI. In fact,
> you've sort of corrupted your database in a way for the particular user
> you've logged in as by adding 1000's of bogus enumerations.
>
> Give that a try first after deleting your bogus enums and see where you
> get to.
>
> -Chris
>
> On Wed, 2009-08-05 at 16:06 +0200, Luca Lianas wrote:
> > Sorry, i forgot the attachment...
> >
> >
> > 2009/8/5 Luca Lianas <luca.lianas at crs4.it>
> >         I belong to the biomedical reserch group at CRS4, a research
> >         centre in Italy; we are currently using omero in several
> >         projects and we are running some performance tests during
> >         these days.
> >         We noticed that the server has low performances when loading a
> >         large amount of data (I tried to load the meta-informations
> >         for 50.000 4-channel images).
> >         I did a smaller test loading 1000 images using a python script
> >         and it took 1 minute and 42 seconds to load the data (as said
> >         before, I only wrote the meta-data of the images into the
> >         database, the real pixels are stored into a HDFS file system).
> >         I used the compiled version of Omero downloaded from the
> >         website and with default configuration. Omero runs on a Linux
> >         server (Fedora core 11) with a dual opteron processor (248
> >         model) and 4GB of RAM.
> >         I'm wondering what is the problem and if there are some hints
> >         to improve the performances on the server. Any help is
> >         appreciated.
> >
> >         Please see the script I'm using, as per attachment (maybe is
> >         the script itself my problem).
> >
> >         Thanks for you attention
> >
> >         Luca
> >
> > _______________________________________________
> > ome-devel mailing list
> > ome-devel at lists.openmicroscopy.org.uk
> > http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.openmicroscopy.org.uk/pipermail/ome-devel/attachments/20090806/fc78b8f5/attachment-0001.htm 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: save_images.py
Type: text/x-python
Size: 3337 bytes
Desc: not available
Url : http://lists.openmicroscopy.org.uk/pipermail/ome-devel/attachments/20090806/fc78b8f5/attachment-0001.py