[ome-users] Core dump attempting to perfrom bulk upload to Omero
John Webber (NBI)
John.Webber at nbi.ac.uk
Wed May 28 12:10:40 BST 2014
Hi Josh,
As per your email last week, please see below a breakdown of the prerequisites that we are currently running:
Are there any notable pre-requisites that I've missed?
Thanks
John
Installed through YUM:
------------------------------
Name : postgresql
Version : 8.4.20
Release : 1.el6_5
Name : postgresql-server
Version : 8.4.20
Release : 1.el6_5
Name : java-1.7.0-openjdk
Version : 1.7.0.55
Release : 2.4.7.1.el6_5
Name : python
Version : 2.6.6
Release : 52.el6
Name : python-devel
Version : 2.6.6
Release : 52.el6
Name : python-imaging
Arch : x86_64
Version : 1.1.6
Release : 19.el6
Name : gcc
Version : 4.4.7
Release : 4.el6
Name : gcc-c++
Version : 4.4.7
Release : 4.el6
Name : freetype-devel
Version : 2.3.11
Release : 14.el6_3.1
Name : libpng-devel
Version : 1.2.49
Release : 1.el6_2
Name : python-ctypes
Version : 1.0.2
Release : 2.el5
Name : gcc-gfortran
Version : 4.4.7
Release : 4.el6
Name : atlas
Version : 3.8.4
Release : 2.el6
Name : atlas-devel
Version : 3.8.4
Release : 2.el6
Name : httpd
Version : 2.2.15
Release : 30.el6.centos
Name : python-setuptools
Version : 0.6.10
Release : 3.el6
Installed from TAR balls:
--------------------------------
numpy-1.8.1
Ice-3.5.1
matplotlib-1.2.1
numexpr-2.1
Cython-0.20.1
hdf5-1.8.12
tables-3.1.1
scipy-0.14.0b1
Pillow-2.4.0
-----Original Message-----
From: Josh Moore [mailto:josh at glencoesoftware.com]
Sent: 23 May 2014 15:22
To: John Webber (NBI)
Cc: OME Users
Subject: Re: [ome-users] Core dump attempting to perfrom bulk upload to Omero
On May 23, 2014, at 4:13 PM, John Webber (NBI) wrote:
> Hi Josh,
Hi John,
> Yes, I have tried the same command you used below (gdb <java executable> <core file>), but I am seeing an error from this as well:
>
> warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff671fe000
> Core was generated by `java -Xmx512M -XX:MaxPermSize=128m -Djava.awt.headless=true -Dlogback.configura'.
> Program terminated with signal 6, Aborted.
> #0 0x0000003d7a432925 in raise () from /lib64/libc.so.6
> Missing separate debuginfos, use: debuginfo-install java-1.7.0-openjdk-1.7.0.55-2.4.7.1.el6_5.x86_64
> (gdb)
Getting into uncharted waters, but could you try enabling the debug repo and installing the debug rpm for your JDK?
See http://fedoraproject.org/wiki/Java/Troubleshooting
> No, I actually performed a complete rebuild of the server - I have
> upgraded the versions of CentOS, Java, Ice, postgres, Python, etc.
> Once the new version was running, I was able to restore a the Omero
> database, using the process documented in
> http://www.openmicroscopy.org/site/support/omero5/sysadmins/server-upg
> rade.html#upgrade-your-database
>
> Would you like me to send though a complete list of the versions of the prerequisites we am currently running?
Guess that's the next best step. Having never seen this before, we have to start somewhere. :)
~J
> Thanks
> John
>
>
> -----Original Message-----
> From: Josh Moore [mailto:josh at glencoesoftware.com]
> Sent: 23 May 2014 14:37
> To: John Webber (NBI)
> Cc: OME Users
> Subject: Re: [ome-users] Core dump attempting to perfrom bulk upload
> to Omero
>
>
> On May 23, 2014, at 2:35 PM, John Webber (NBI) wrote:
>
>> Hi Josh,
>>
>> Thanks for the response.
>>
>> I do indeed still have the core dump file located in: /var/spool/abrt/ccpp-2014-05-22-17:20:04-1561.
>>
>> I have also installed gdb, but I have not used this before, and am not getting a lot of joy from trying to use it. When I have attempted to run the following command:
>>
>> gdb '/usr/lib/jvm/jre-1.7.0-openjdk.x86_64/bin/java -Xmx512M -XX:MaxPermSize=128m -Djava.awt.headless=true
>> -Dlogback.configura'
>> /var/spool/abrt/ccpp-2014-05-22-17:20:04-1561/coredump
>>
>> I see the error:
>> warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff671fe000
>> Core was generated by `java -Xmx512M -XX:MaxPermSize=128m -Djava.awt.headless=true -Dlogback.configura'.
>> Program terminated with signal 6, Aborted.
>>
>> Am I running GDB incorrectly?
>
> Usually I'd just do:
>
> gdb '/usr/lib/jvm/jre-1.7.0-openjdk.x86_64/bin/java
> /var/spool/abrt/ccpp-2014-05-22-17:20:04-1561/coredump
>
>
>> You asked if this server had had any system maintenance recently: This server is actually one that I have just upgraded to Omero 5. It is a server that I am using to test the upgrade process. It was therefore installed with Omero 4.4.8 (the current version of Omero that we run in production) and had several items of test data uploaded into Omero. It was then upgraded to Omero 5 to ensure that the upgrade process is working correctly. The upgrade itself all seems to be successful, but these bulk upload scripts are not functioning as expected.
>
> Definitely not. But just to be clear: other than the upgrade from 4.4.8 (which was working fine) to 5.0, no other upgrades took place?
>
>
>> One this to note, however, is that the bulk upload scripts have not been run on this specific server before, but are run regularly on the Production server with no issues.
>
> Is anything else running on this server? Any other Java processes?
>
> Cheers,
> ~Josh.
>
>
>
>> Thanks
>> John
>>
>>
>>
>> -----Original Message-----
>> From: Josh Moore [mailto:josh at glencoesoftware.com]
>> Sent: 23 May 2014 11:36
>> To: John Webber (NBI)
>> Cc: OME Users
>> Subject: Re: [ome-users] Core dump attempting to perfrom bulk upload
>> to Omero
>>
>>
>> On May 23, 2014, at 10:33 AM, John Webber (NBI) wrote:
>>
>>> Hi Josh,
>>
>> Hi John,
>>
>>> Thanks for the quick response - as per normal! :-)
>>
>> Gladly. Thanks for all the log data!
>>
>>> Please see attached a zip file containing the files in my Omero log folder (/opt/Omero/var/log). As well as the log files that normally reside in this location, I have also added the following files:
>>>
>>> bulk-upload-script.log The logfile for my bulk upload script - so you can see what that is doing.
>>> var-log-messages-file The main server log (/var/log/messages) file.
>>> emails The emails which were sent for these core dumps.
>>>
>>> Any pointers you can give as to where things are going wrong would be gratefully received! Do you also need to know the software versions that I have running, (for example Java and Python)?
>>
>> Can you check that you still have the file:
>>
>> May 22 17:20:11 v0246 abrt[2613]: Saved core dump of pid 1561
>> (/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.55.x86_64/jre/bin/java) to
>> /var/spool/abrt/ccpp-2014-05-22-17:20:04-1561 (741801984 bytes)
>>
>> If so, do you have gdb installed? It would be good to try to get a stack trace of what core dumped.
>>
>> Here are the restarts in your server:
>>
>> 2014-05-22 10:17:23,158 INFO [ ome.services.util.ServerVersionCheck] ( main) OMERO Version: 5.0.0-ice35-b19 Ready.
>> 2014-05-22 10:58:26,515 INFO [ ome.services.util.ServerVersionCheck] ( main) OMERO Version: 5.0.0-ice35-b19 Ready.
>> 2014-05-22 11:58:43,658 INFO [ ome.services.util.ServerVersionCheck] ( main) OMERO Version: 5.0.0-ice35-b19 Ready.
>> 2014-05-22 14:52:47,780 INFO [ ome.services.util.ServerVersionCheck] ( main) OMERO Version: 5.0.0-ice35-b19 Ready.
>> 2014-05-22 14:55:34,882 INFO [ ome.services.util.ServerVersionCheck] ( main) OMERO Version: 5.0.0-ice35-b19 Ready.
>> 2014-05-22 15:15:26,549 INFO [ ome.services.util.ServerVersionCheck] ( main) OMERO Version: 5.0.0-ice35-b19 Ready.
>> 2014-05-22 16:38:54,761 INFO [ ome.services.util.ServerVersionCheck] ( main) OMERO Version: 5.0.0-ice35-b19 Ready.
>> 2014-05-22 17:11:40,885 INFO [ ome.services.util.ServerVersionCheck] ( main) OMERO Version: 5.0.0-ice35-b19 Ready.
>> 2014-05-22 17:20:38,436 INFO [ ome.services.util.ServerVersionCheck] ( main) OMERO Version: 5.0.0-ice35-b19 Ready.
>> 2014-05-22 17:23:23,171 INFO [ ome.services.util.ServerVersionCheck] ( main) OMERO Version: 5.0.0-ice35-b19 Ready.
>>
>>
>> If they match any pattern you can detect, please let me know.
>>
>> Also, has there been any system maintenance recently? New Java versions, etc?
>> Cheers,
>> ~Josh.
>>
>>
>>> Thanks
>>> John
>>>
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: Josh Moore [mailto:josh at glencoesoftware.com]
>>> Sent: 22 May 2014 21:23
>>> To: John Webber (NBI)
>>> Subject: Re: [ome-users] Core dump attempting to perfrom bulk upload
>>> to Omero
>>>
>>>
>>> On May 22, 2014, at 6:17 PM, John Webber (NBI) wrote:
>>>
>>>> Hi,
>>>
>>> Hi John,
>>>
>>>> I am currently testing the same process on a new server which is built with Omero 5.0 and Ice 3.5. I am seeing the following error, but not after a consistent time:
>>>>
>>>> Using session feccfbae-b08e-4298-9f3c-5f31139af486
>>>> (webberj at localhost:4064). Idle timeout: 10.0 min. Current group:
>>>> system Previous session expired for webberj on localhost:4064
>>>> InternalException: Failed to connect: exception
>>>> ::Glacier2::CannotCreateSessionException
>>>> {
>>>> reason = Server not fully initialized }
>>>
>>> This message is coming from the Java process and means that it's just restarted. Likely your server is bouncing, up/down up/down. Could you send us the OMERO log files under var/log zipped up, incl master.out and master.err?
>>>
>>>
>>>> A core dump is then generated. The following are the log entries (/var/log/messages) when this occurs:
>>>> ...
>>>> The email that is sent as a result of this crash is very long, but I have attached the first part of an example to this email.
>>>>
>>>> Does anyone have any ideas why Omero / Ice is crashing in this way? Does this look like an Omero Issue, an Ice issue or a Java issue?
>>>
>>> It's the Java process which is aborting. More of the sos email could be useful if you could zip that up as well. For anyone who's wondering about OS & versions for a comparison to their own system:
>>>
>>> executable: /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.55.x86_64/jre/bin/java
>>> kernel: 3.10.34-11.el6.centos.alt.x86_64
>>>
>>>
>>>> Thanks
>>>> John
>>>
>>> Cheers,
>>> ~Josh
>>> <logfile.tar.gz>
More information about the ome-users
mailing list