[ome-users] Import error

Douglas Russell douglas_russell at hms.harvard.edu
Tue Jan 9 13:20:15 GMT 2018


Hi,

Sorry for delay following this up.

These OMERO instances are in Docker, yes, but otherwise I don't think there
is anything remarkable about the configuration. I have allocated postgres
5GBs of RAM and am not seeing any messages about that running out of
memory. The OMERO server has 20GBs of RAM.

The only errors in the Blitz log are:

/opt/omero/server/OMERO.server/var/log/Blitz-0.log:2018-01-09 00:15:32,910
ERROR [        ome.services.util.ServiceHandler] (l.Server-7) Method
interface ome.api.ThumbnailStore.createThumbnailsByLongestSideSet
invocation took 26125
/opt/omero/server/OMERO.server/var/log/Blitz-0.log:2018-01-09 00:15:33,090
ERROR [o.s.t.interceptor.TransactionInterceptor] (2-thread-4) Application
exception overridden by rollback exception
/opt/omero/server/OMERO.server/var/log/Blitz-0.log:2018-01-09 00:15:33,090
ERROR [        ome.services.util.ServiceHandler] (2-thread-4) Method
interface ome.services.util.Executor$Work.doWork invocation took 17514887

The only thing I haven't yet tried is moving postgres into the same
container as OMERO. I can try that if it would help, but I highly doubt it
will make any difference as in this setup, there is only one t2.2xlarge
instance running everything. It was using a load balancer (easiest way to
connect things up should they actually be on different hosts), but I tried
it without that where I just give the IP of the postgres docker container
to the OMERO instance configuration and I got the same result, so it's not
the timeout of the load balancer at fault.

Thanks,

Douglas

On Wed, 3 Jan 2018 at 06:56 Mark Carroll <m.t.b.carroll at dundee.ac.uk> wrote:

>
> On 12/23/2017 12:32 PM, Douglas Russell wrote:
> > I'd checked master logs files and there was nothing of interest in
> > there. dmesg is more promising though, good idea. It looks like a memory
> > issue. I've increased the amount of memory available to 20GBs from 4GBs
> > and now it does not fail in the same way. Not sure why so much RAM is
> > needed when each image in the screen is only 2.6MBs. Now there is a nice
> > new error.
>
> You have me wondering if the server does the whole plate import in only
> one transaction. Also, if memory issues could be due to PostgreSQL or
> instead Java (e.g., Hibernate) and, assuming Java-side, if the issue is
> pixel data size (do the TIFF files use compression?) or metadata (e.g.,
> tons of ROIs?). Scalability has been an ongoing focus for us: we have
> done much but there is much more yet to be done.
>
> > Going by the error that I see when the database tries to rollback, I
> > think it is timeout related.
>
> I'm not seeing an obvious timeout issue here but I may well be missing
> something and maybe over the holiday period you have noticed more clues
> yourself too?
>
> > The import log: https://s3.amazonaws.com/dpwr/pat/import_log.txt
> > The server logs (I tried the import twice):
> > https://s3.amazonaws.com/dpwr/pat/omero_logs.zip
> >
> > There are a couple of these in the database logs as you'd expect for the
> > two import attempts, but nothing else of interest.
> >
> > LOG: unexpected EOF on client connection with an open transaction
>
> Mmmm, late in the import process the EOFException from
> PGStream.ReceiveChar looks key. I'm trying to think what in PostgreSQL's
> pg_* tables might give some hint as to relevant activity or locks at the
> time (if it's a timeout, maybe a deadlock?). I guess there's nothing
> particularly exciting about how your OMERO server connects to
> PostgreSQL? It's simply across a LAN, perhaps via Docker or somesuch?
>
> How large is the plate? Given the 5.4 database changes I am wondering if
> this could possibly be a regression since 5.3.5 and how easy the error
> might be to reproduce in a test environment.
>
> Now the holiday season is behind us, at OME we're starting to return to
> the office. Happy New Year! With luck we'll get this issue figured out
> promptly. My apologies if I missed some existing context from the thread
> that I didn't realize already bears on some of my questions.
>
> -- Mark
>
> The University of Dundee is a registered Scottish Charity, No: SC015096
> _______________________________________________
> ome-users mailing list
> ome-users at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-users/attachments/20180109/23634ac6/attachment.html>


More information about the ome-users mailing list