[ome-devel] Omero server performance issue

[Luca Lianas]

Hi Chris

I modified the script as you suggested putting a comment on the
"saveAndReturn" and "unload" lines for the logical channel then I launched
the script with 10 blocks of 100 images.
I made a select count(*) from logicalchannel to verify if everything was
fine and i found that the table contained 40 rows, it appears that every
time I call the saveArray method in the script four new logicalchannel items
are saved into the database.
If I remove the comments from the lines that save the logicalchannel first,
unload the returned object and I use that object when inserting data I have
only four rows into the database.
Am I doing something wrong or is it an issue of the server?

Luca

2009/8/6 Chris Allan <callan at blackcat.ca>

> On Thu, 2009-08-06 at 12:50 +0200, Luca Lianas wrote:
> > Hi Chris
>
> Hi Luca.
>
> >
> > thanks for your advice. As suggested by you, I modified the script and
> > created only 4 LogicalChannelI objects linked to the image's channels
> > (I looked into the xml model file and found the many-to-one
> > relationship between channel and logicalchannel), now the execution
> > time takes only 13 minutes.
>
> 1 minute and 42 seconds --> 13 minutes? (did you mean seconds? :))
>
> As OMERO's objects are graph oriented you also don't need to save the
> LogicalChannels you are re-using. So just, the following at the top of
> your try block is fine:
>
>  adenine = om.model.LogicalChannelI()
>  adenine.setName(omt.rstring('ADENINE'))
>  citosine = om.model.LogicalChannelI()
>  citosine.setName(omt.rstring('CITOSINE'))
>  guanine = om.model.LogicalChannelI()
>  guanine.setName(omt.rstring('GUANINE'))
>  thymine = om.model.LogicalChannelI()
>  thymine.setName(omt.rstring('THYMINE'))
>
> OMERO knows how to inspect the graph for duplicates, how to ensure
> references are properly represented, calculate the INSERT plan, etc.
>
> > I attach the script in order to provide an example if anybody needs
> > one.
> >
> > Luca
>
> -Chris
>
> > 2009/8/5 Chris Allan <callan at blackcat.ca>
> >         Hi Luca,
> >
> >         One big thing is to be very careful with enumerations:
> >
> >         26       do = om.model.DimensionOrderI()
> >         27       do.setValue(omt.rstring('XYZCT'))
> >         28       pt = om.model.PixelsTypeI()
> >         29       pt.setValue(omt.rstring('uint16'))
> >
> >         This is creating brand new enumerations for every image
> >         inserted, which
> >         will slow any enumeration queries to a crawl and is causing
> >         3000 extra
> >         INSERTs, significant graph inspection overhead, etc. You want
> >         to
> >         retrieve the existing enumerations through IQuery and then use
> >         an
> >         unloaded version of the object to help you out, for example
> >         (pseudo-code):
> >
> >         ...
> >         dimension_orders = iquery.findAll("DimensionOrder")
> >         xyzct = filter(lambda a: a.value.val == 'XYZCT')[0]
> >         syzct.unload()
> >         ...
> >         for image in range(0, 100):
> >         ...
> >            p.setDimensionOrder(xyzct)
> >
> >         The above applies to FormatI, DimensionOrderI and PixelsTypeI.
> >         In fact,
> >         you've sort of corrupted your database in a way for the
> >         particular user
> >         you've logged in as by adding 1000's of bogus enumerations.
> >
> >         Give that a try first after deleting your bogus enums and see
> >         where you
> >         get to.
> >
> >         -Chris
> >
> >
> >         On Wed, 2009-08-05 at 16:06 +0200, Luca Lianas wrote:
> >         > Sorry, i forgot the attachment...
> >         >
> >         >
> >         > 2009/8/5 Luca Lianas <luca.lianas at crs4.it>
> >         >         I belong to the biomedical reserch group at CRS4, a
> >         research
> >         >         centre in Italy; we are currently using omero in
> >         several
> >         >         projects and we are running some performance tests
> >         during
> >         >         these days.
> >         >         We noticed that the server has low performances when
> >         loading a
> >         >         large amount of data (I tried to load the
> >         meta-informations
> >         >         for 50.000 4-channel images).
> >         >         I did a smaller test loading 1000 images using a
> >         python script
> >         >         and it took 1 minute and 42 seconds to load the data
> >         (as said
> >         >         before, I only wrote the meta-data of the images
> >         into the
> >         >         database, the real pixels are stored into a HDFS
> >         file system).
> >         >         I used the compiled version of Omero downloaded from
> >         the
> >         >         website and with default configuration. Omero runs
> >         on a Linux
> >         >         server (Fedora core 11) with a dual opteron
> >         processor (248
> >         >         model) and 4GB of RAM.
> >         >         I'm wondering what is the problem and if there are
> >         some hints
> >         >         to improve the performances on the server. Any help
> >         is
> >         >         appreciated.
> >         >
> >         >         Please see the script I'm using, as per attachment
> >         (maybe is
> >         >         the script itself my problem).
> >         >
> >         >         Thanks for you attention
> >         >
> >         >         Luca
> >         >
> >
> >         > _______________________________________________
> >         > ome-devel mailing list
> >         > ome-devel at lists.openmicroscopy.org.uk
> >         >
> >         http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
> >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.openmicroscopy.org.uk/pipermail/ome-devel/attachments/20090807/6ae1e105/attachment.htm