[ome-devel] Omero server performance issue
Chris Allan
callan at blackcat.ca
Thu Aug 6 13:01:52 BST 2009
On Thu, 2009-08-06 at 13:31 +0200, Luca Lianas wrote:
> Hi Chris
Hi Luca.
>
> 13 minutes is the result for 50.000 inserts, sorry I reported only my
> last test. The test with 1000 images was completed in about 36
> seconds.
That sounds about right for 4.0.x. With larger "INSERT and reference
only" (no UPDATEs) type graphs we have also implemented significant
improvements in Subversion Trunk with the new saveAndReturnIds()
infrastructure; sometimes exceeding 10 fold.
>
> Luca
-Chris
>
> 2009/8/6 Chris Allan <callan at blackcat.ca>
> On Thu, 2009-08-06 at 12:50 +0200, Luca Lianas wrote:
> > Hi Chris
>
> Hi Luca.
>
> >
> > thanks for your advice. As suggested by you, I modified the
> script and
> > created only 4 LogicalChannelI objects linked to the image's
> channels
> > (I looked into the xml model file and found the many-to-one
> > relationship between channel and logicalchannel), now the
> execution
> > time takes only 13 minutes.
>
>
> 1 minute and 42 seconds --> 13 minutes? (did you mean
> seconds? :))
>
> As OMERO's objects are graph oriented you also don't need to
> save the
> LogicalChannels you are re-using. So just, the following at
> the top of
> your try block is fine:
>
> adenine = om.model.LogicalChannelI()
> adenine.setName(omt.rstring('ADENINE'))
> citosine = om.model.LogicalChannelI()
> citosine.setName(omt.rstring('CITOSINE'))
> guanine = om.model.LogicalChannelI()
> guanine.setName(omt.rstring('GUANINE'))
> thymine = om.model.LogicalChannelI()
> thymine.setName(omt.rstring('THYMINE'))
>
> OMERO knows how to inspect the graph for duplicates, how to
> ensure
> references are properly represented, calculate the INSERT
> plan, etc.
>
> > I attach the script in order to provide an example if
> anybody needs
> > one.
> >
> > Luca
>
>
> -Chris
>
>
> > 2009/8/5 Chris Allan <callan at blackcat.ca>
> > Hi Luca,
> >
> > One big thing is to be very careful with
> enumerations:
> >
> > 26 do = om.model.DimensionOrderI()
> > 27 do.setValue(omt.rstring('XYZCT'))
> > 28 pt = om.model.PixelsTypeI()
> > 29 pt.setValue(omt.rstring('uint16'))
> >
> > This is creating brand new enumerations for every
> image
> > inserted, which
> > will slow any enumeration queries to a crawl and is
> causing
> > 3000 extra
> > INSERTs, significant graph inspection overhead, etc.
> You want
> > to
> > retrieve the existing enumerations through IQuery
> and then use
> > an
> > unloaded version of the object to help you out, for
> example
> > (pseudo-code):
> >
> > ...
> > dimension_orders = iquery.findAll("DimensionOrder")
> > xyzct = filter(lambda a: a.value.val == 'XYZCT')[0]
> > syzct.unload()
> > ...
> > for image in range(0, 100):
> > ...
> > p.setDimensionOrder(xyzct)
> >
> > The above applies to FormatI, DimensionOrderI and
> PixelsTypeI.
> > In fact,
> > you've sort of corrupted your database in a way for
> the
> > particular user
> > you've logged in as by adding 1000's of bogus
> enumerations.
> >
> > Give that a try first after deleting your bogus
> enums and see
> > where you
> > get to.
> >
> > -Chris
> >
> >
> > On Wed, 2009-08-05 at 16:06 +0200, Luca Lianas
> wrote:
> > > Sorry, i forgot the attachment...
> > >
> > >
> > > 2009/8/5 Luca Lianas <luca.lianas at crs4.it>
> > > I belong to the biomedical reserch group
> at CRS4, a
> > research
> > > centre in Italy; we are currently using
> omero in
> > several
> > > projects and we are running some
> performance tests
> > during
> > > these days.
> > > We noticed that the server has low
> performances when
> > loading a
> > > large amount of data (I tried to load the
> > meta-informations
> > > for 50.000 4-channel images).
> > > I did a smaller test loading 1000 images
> using a
> > python script
> > > and it took 1 minute and 42 seconds to
> load the data
> > (as said
> > > before, I only wrote the meta-data of the
> images
> > into the
> > > database, the real pixels are stored into
> a HDFS
> > file system).
> > > I used the compiled version of Omero
> downloaded from
> > the
> > > website and with default configuration.
> Omero runs
> > on a Linux
> > > server (Fedora core 11) with a dual
> opteron
> > processor (248
> > > model) and 4GB of RAM.
> > > I'm wondering what is the problem and if
> there are
> > some hints
> > > to improve the performances on the server.
> Any help
> > is
> > > appreciated.
> > >
> > > Please see the script I'm using, as per
> attachment
> > (maybe is
> > > the script itself my problem).
> > >
> > > Thanks for you attention
> > >
> > > Luca
> > >
> >
> > > _______________________________________________
> > > ome-devel mailing list
> > > ome-devel at lists.openmicroscopy.org.uk
> > >
> >
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
> >
> >
>
>
>
More information about the ome-devel
mailing list