[ome-devel] Omero server performance issue

[Chris Allan]

On Thu, 2009-08-06 at 12:50 +0200, Luca Lianas wrote:
> Hi Chris

Hi Luca.

> 
> thanks for your advice. As suggested by you, I modified the script and
> created only 4 LogicalChannelI objects linked to the image's channels
> (I looked into the xml model file and found the many-to-one
> relationship between channel and logicalchannel), now the execution
> time takes only 13 minutes.

1 minute and 42 seconds --> 13 minutes? (did you mean seconds? :))

As OMERO's objects are graph oriented you also don't need to save the
LogicalChannels you are re-using. So just, the following at the top of
your try block is fine:

  adenine = om.model.LogicalChannelI()
  adenine.setName(omt.rstring('ADENINE'))
  citosine = om.model.LogicalChannelI()
  citosine.setName(omt.rstring('CITOSINE'))
  guanine = om.model.LogicalChannelI()
  guanine.setName(omt.rstring('GUANINE'))
  thymine = om.model.LogicalChannelI()
  thymine.setName(omt.rstring('THYMINE'))

OMERO knows how to inspect the graph for duplicates, how to ensure
references are properly represented, calculate the INSERT plan, etc.

> I attach the script in order to provide an example if anybody needs
> one.
> 
> Luca

-Chris

> 2009/8/5 Chris Allan <callan at blackcat.ca>
>         Hi Luca,
>         
>         One big thing is to be very careful with enumerations:
>         
>         26       do = om.model.DimensionOrderI()
>         27       do.setValue(omt.rstring('XYZCT'))
>         28       pt = om.model.PixelsTypeI()
>         29       pt.setValue(omt.rstring('uint16'))
>         
>         This is creating brand new enumerations for every image
>         inserted, which
>         will slow any enumeration queries to a crawl and is causing
>         3000 extra
>         INSERTs, significant graph inspection overhead, etc. You want
>         to
>         retrieve the existing enumerations through IQuery and then use
>         an
>         unloaded version of the object to help you out, for example
>         (pseudo-code):
>         
>         ...
>         dimension_orders = iquery.findAll("DimensionOrder")
>         xyzct = filter(lambda a: a.value.val == 'XYZCT')[0]
>         syzct.unload()
>         ...
>         for image in range(0, 100):
>         ...
>            p.setDimensionOrder(xyzct)
>         
>         The above applies to FormatI, DimensionOrderI and PixelsTypeI.
>         In fact,
>         you've sort of corrupted your database in a way for the
>         particular user
>         you've logged in as by adding 1000's of bogus enumerations.
>         
>         Give that a try first after deleting your bogus enums and see
>         where you
>         get to.
>         
>         -Chris
>         
>         
>         On Wed, 2009-08-05 at 16:06 +0200, Luca Lianas wrote:
>         > Sorry, i forgot the attachment...
>         >
>         >
>         > 2009/8/5 Luca Lianas <luca.lianas at crs4.it>
>         >         I belong to the biomedical reserch group at CRS4, a
>         research
>         >         centre in Italy; we are currently using omero in
>         several
>         >         projects and we are running some performance tests
>         during
>         >         these days.
>         >         We noticed that the server has low performances when
>         loading a
>         >         large amount of data (I tried to load the
>         meta-informations
>         >         for 50.000 4-channel images).
>         >         I did a smaller test loading 1000 images using a
>         python script
>         >         and it took 1 minute and 42 seconds to load the data
>         (as said
>         >         before, I only wrote the meta-data of the images
>         into the
>         >         database, the real pixels are stored into a HDFS
>         file system).
>         >         I used the compiled version of Omero downloaded from
>         the
>         >         website and with default configuration. Omero runs
>         on a Linux
>         >         server (Fedora core 11) with a dual opteron
>         processor (248
>         >         model) and 4GB of RAM.
>         >         I'm wondering what is the problem and if there are
>         some hints
>         >         to improve the performances on the server. Any help
>         is
>         >         appreciated.
>         >
>         >         Please see the script I'm using, as per attachment
>         (maybe is
>         >         the script itself my problem).
>         >
>         >         Thanks for you attention
>         >
>         >         Luca
>         >
>         
>         > _______________________________________________
>         > ome-devel mailing list
>         > ome-devel at lists.openmicroscopy.org.uk
>         >
>         http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
>         
>