[ome-devel] Omero server performance issue

Chris Allan callan at blackcat.ca
Thu Aug 6 13:01:52 BST 2009


On Thu, 2009-08-06 at 13:31 +0200, Luca Lianas wrote:
> Hi Chris

Hi Luca.

> 
> 13 minutes is the result for 50.000 inserts, sorry I reported only my
> last test. The test with 1000 images was completed in about 36
> seconds.

That sounds about right for 4.0.x. With larger "INSERT and reference
only" (no UPDATEs) type graphs we have also implemented significant
improvements in Subversion Trunk with the new saveAndReturnIds()
infrastructure; sometimes exceeding 10 fold.

> 
> Luca

-Chris

> 
> 2009/8/6 Chris Allan <callan at blackcat.ca>
>         On Thu, 2009-08-06 at 12:50 +0200, Luca Lianas wrote:
>         > Hi Chris
>         
>         Hi Luca.
>         
>         >
>         > thanks for your advice. As suggested by you, I modified the
>         script and
>         > created only 4 LogicalChannelI objects linked to the image's
>         channels
>         > (I looked into the xml model file and found the many-to-one
>         > relationship between channel and logicalchannel), now the
>         execution
>         > time takes only 13 minutes.
>         
>         
>         1 minute and 42 seconds --> 13 minutes? (did you mean
>         seconds? :))
>         
>         As OMERO's objects are graph oriented you also don't need to
>         save the
>         LogicalChannels you are re-using. So just, the following at
>         the top of
>         your try block is fine:
>         
>          adenine = om.model.LogicalChannelI()
>          adenine.setName(omt.rstring('ADENINE'))
>          citosine = om.model.LogicalChannelI()
>          citosine.setName(omt.rstring('CITOSINE'))
>          guanine = om.model.LogicalChannelI()
>          guanine.setName(omt.rstring('GUANINE'))
>          thymine = om.model.LogicalChannelI()
>          thymine.setName(omt.rstring('THYMINE'))
>         
>         OMERO knows how to inspect the graph for duplicates, how to
>         ensure
>         references are properly represented, calculate the INSERT
>         plan, etc.
>         
>         > I attach the script in order to provide an example if
>         anybody needs
>         > one.
>         >
>         > Luca
>         
>         
>         -Chris
>         
>         
>         > 2009/8/5 Chris Allan <callan at blackcat.ca>
>         >         Hi Luca,
>         >
>         >         One big thing is to be very careful with
>         enumerations:
>         >
>         >         26       do = om.model.DimensionOrderI()
>         >         27       do.setValue(omt.rstring('XYZCT'))
>         >         28       pt = om.model.PixelsTypeI()
>         >         29       pt.setValue(omt.rstring('uint16'))
>         >
>         >         This is creating brand new enumerations for every
>         image
>         >         inserted, which
>         >         will slow any enumeration queries to a crawl and is
>         causing
>         >         3000 extra
>         >         INSERTs, significant graph inspection overhead, etc.
>         You want
>         >         to
>         >         retrieve the existing enumerations through IQuery
>         and then use
>         >         an
>         >         unloaded version of the object to help you out, for
>         example
>         >         (pseudo-code):
>         >
>         >         ...
>         >         dimension_orders = iquery.findAll("DimensionOrder")
>         >         xyzct = filter(lambda a: a.value.val == 'XYZCT')[0]
>         >         syzct.unload()
>         >         ...
>         >         for image in range(0, 100):
>         >         ...
>         >            p.setDimensionOrder(xyzct)
>         >
>         >         The above applies to FormatI, DimensionOrderI and
>         PixelsTypeI.
>         >         In fact,
>         >         you've sort of corrupted your database in a way for
>         the
>         >         particular user
>         >         you've logged in as by adding 1000's of bogus
>         enumerations.
>         >
>         >         Give that a try first after deleting your bogus
>         enums and see
>         >         where you
>         >         get to.
>         >
>         >         -Chris
>         >
>         >
>         >         On Wed, 2009-08-05 at 16:06 +0200, Luca Lianas
>         wrote:
>         >         > Sorry, i forgot the attachment...
>         >         >
>         >         >
>         >         > 2009/8/5 Luca Lianas <luca.lianas at crs4.it>
>         >         >         I belong to the biomedical reserch group
>         at CRS4, a
>         >         research
>         >         >         centre in Italy; we are currently using
>         omero in
>         >         several
>         >         >         projects and we are running some
>         performance tests
>         >         during
>         >         >         these days.
>         >         >         We noticed that the server has low
>         performances when
>         >         loading a
>         >         >         large amount of data (I tried to load the
>         >         meta-informations
>         >         >         for 50.000 4-channel images).
>         >         >         I did a smaller test loading 1000 images
>         using a
>         >         python script
>         >         >         and it took 1 minute and 42 seconds to
>         load the data
>         >         (as said
>         >         >         before, I only wrote the meta-data of the
>         images
>         >         into the
>         >         >         database, the real pixels are stored into
>         a HDFS
>         >         file system).
>         >         >         I used the compiled version of Omero
>         downloaded from
>         >         the
>         >         >         website and with default configuration.
>         Omero runs
>         >         on a Linux
>         >         >         server (Fedora core 11) with a dual
>         opteron
>         >         processor (248
>         >         >         model) and 4GB of RAM.
>         >         >         I'm wondering what is the problem and if
>         there are
>         >         some hints
>         >         >         to improve the performances on the server.
>         Any help
>         >         is
>         >         >         appreciated.
>         >         >
>         >         >         Please see the script I'm using, as per
>         attachment
>         >         (maybe is
>         >         >         the script itself my problem).
>         >         >
>         >         >         Thanks for you attention
>         >         >
>         >         >         Luca
>         >         >
>         >
>         >         > _______________________________________________
>         >         > ome-devel mailing list
>         >         > ome-devel at lists.openmicroscopy.org.uk
>         >         >
>         >
>         http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
>         >
>         >
>         
>         
> 



More information about the ome-devel mailing list