[ome-users] Core dump attempting to perfrom bulk upload to Omero

Simon Li s.p.li at dundee.ac.uk
Mon Jun 9 11:13:36 BST 2014 

Hi John

At the gdb prompt enter 'bt' to get a stack-trace. Don't be surprised if the output is unexpectedly long.

Simon

On 6 Jun 2014, at 17:32, John Webber (NBI) <John.Webber at nbi.ac.uk> wrote:

> Josh / Simon,
>
> Thank you for your help with this issue.
>
> As per the instructions below, I have manually installed the debuginfo package for java-1.7.0-openjdk.  This is now installed on this server.
>
> I am seeing the following error message when I attempt to use the Omero CLI to run a (bulk) upload to Omero:
>
>          -! 06/06/14 10:31:10.835 warning: Proxy keep alive failed.
>
> The upload process appears to still be running, but it has hung.
>
> There is no other information on the command line, but I see the following in the /var/log/messages file:
>
> Jun  6 10:30:20 v0246 abrt[12673]: Saved core dump of pid 3671 (/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.55.x86_64/jre/bin/java) to /var/spool/abrt/ccpp-2014-06-06-10:30:13-3671 (635846656 bytes)
> Jun  6 10:30:20 v0246 abrtd: Directory 'ccpp-2014-06-06-10:30:13-3671' creation detected
> Jun  6 10:30:20 v0246 abrt[12673]: /var/spool/abrt is 1378680445 bytes (more than 1279MiB), deleting 'ccpp-2014-06-05-18:21:44-2485'
> Jun  6 10:30:20 v0246 abrtd: Size of '/var/spool/abrt' >= 1000 MB, deleting 'ccpp-2014-06-05-18:21:44-2485'
> Jun  6 10:30:20 v0246 abrtd: Lock file '/var/spool/abrt/ccpp-2014-06-05-18:21:44-2485/.lock' is locked by process 12673
> Jun  6 10:30:20 v0246 abrtd: '/var/spool/abrt/ccpp-2014-06-05-18:21:44-2485' does not exist
> Jun  6 10:30:27 v0246 kernel: end_request: I/O error, dev fd0, sector 0
> Jun  6 10:30:27 v0246 kernel: end_request: I/O error, dev fd0, sector 0
> Jun  6 10:31:45 v0246 abrtd: Sending an email...
> Jun  6 10:31:45 v0246 abrtd: Email was sent to: root at localhost
> Jun  6 10:31:47 v0246 abrtd: New problem directory /var/spool/abrt/ccpp-2014-06-06-10:30:13-3671, processing
>
> I have run the following gdb command, but am not sure how to progress:
>
> gdb '/usr/lib/jvm/jre-1.7.0-openjdk.x86_64/bin/java -Xmx512M -XX:MaxPermSize=128m -Djava.awt.headless=true -Dlogback.configuration' /var/spool/abrt/ccpp-2014-06-06-10:30:13-3671/coredump
>
> The following is the end of the dgb:
>
> Core was generated by `java -Xmx512M -XX:MaxPermSize=128m -Djava.awt.headless=true -Dlogback.configura'.
> Program terminated with signal 6, Aborted.
> #0  0x0000003d7a432925 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
> 64        return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
> (gdb)
>
> Please can you advise how I can progress with this?
>
> Thanks
> John
>
>
>
> -----Original Message-----
> From: Simon Li [mailto:s.p.li at dundee.ac.uk]
> Sent: 02 June 2014 11:31
> To: John Webber (NBI)
> Cc: OME Users
> Subject: Re: [ome-users] Core dump attempting to perfrom bulk upload to Omero
>
>
> On 2 Jun 2014, at 10:57, Josh Moore <josh at glencoesoftware.com>
> wrote:
>
>>
>> On May 29, 2014, at 6:41 PM, John Webber (NBI) wrote:
>>
>>> Hi Josh,
>>>
>>>> Did you have any success installing the debugging symbols?
>>>
>>> Unfortunately, no!
>>>
>>> I've had a look for "java-1.7.0-openjdk-debuginfo", but have not been able to locate the RPM for this for our version of Linux (which is CentOS). We don't use the standard repos and I haven't been able to locate one for CentOS 6.
>>>
>>> Any ideas?
>>
>> Which repositories are you using? Would https://oss.oracle.com/el6/debuginfo/ work? It's not exactly the same build, but close:
>>
>>
>> https://oss.oracle.com/el6/debuginfo/java-1.7.0-openjdk-debuginfo-1.7.
>> 0.55-2.4.7.1.0.1.el6_5.x86_64.rpm
>>
>> Perhaps a CentOS guru on the list could suggest whether mixing is a smart idea.
>
> Hi John
>
> Could you paste the output of these two commands:
> rpm -qi java-1.7.0-openjdk
> yum info java-1.7.0-openjdk
>
> If the latter command shows "From repo:", "updates" or "base" it's probably from one of the standard CentOS repos, in which case you can enable the debug repo:
>
> yum --enablerepo=debug search java-1.7.0-openjdk yum --enablerepo=debug install java-1.7.0-openjdk-debuginfo
>
> If it helps this is /etc/yum.repos.d/CentOS-Debuginfo.repo:
> [debug]
> name=CentOS-6 - Debuginfo
> baseurl=http://debuginfo.centos.org/6/$basearch/
> gpgcheck=1
> gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-Debug-6
> enabled=0
>
> So you could also try to manually install the debuginfo package from http://debuginfo.centos.org/6/x86_64
>
> Simon
>
>> Cheers,
>> ~Josh
>>
>>
>>> Thanks
>>> John
>>>
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: Josh Moore [mailto:josh at glencoesoftware.com]
>>> Sent: 28 May 2014 20:37
>>> To: John Webber (NBI)
>>> Cc: OME Users
>>> Subject: Re: [ome-users] Core dump attempting to perfrom bulk upload
>>> to Omero
>>>
>>>
>>> On May 28, 2014, at 1:10 PM, John Webber (NBI) wrote:
>>>
>>>> Hi Josh,
>>>
>>> Hi John,
>>>
>>>> As per your email last week, please see below a breakdown of the prerequisites that we are currently running:
>>>>
>>>> Are there any notable pre-requisites that I've missed?
>>>
>>> For the base install, nothing is glaringly missing. It did occur to us that you have both python-imaging and Pillow-2.4.0, the latter being a fork of the former, but that shouldn't cause any problems, especially not with Java.
>>>
>>> Did you have any success installing the debugging symbols?
>>> Cheers,
>>> ~Josh
>>>
>>>
>>>> -----Original Message-----
>>>> From: Josh Moore [mailto:josh at glencoesoftware.com]
>>>> Sent: 23 May 2014 15:22
>>>> To: John Webber (NBI)
>>>> Cc: OME Users
>>>> Subject: Re: [ome-users] Core dump attempting to perfrom bulk upload
>>>> to Omero
>>>>
>>>>
>>>> On May 23, 2014, at 4:13 PM, John Webber (NBI) wrote:
>>>>
>>>>> Hi Josh,
>>>>
>>>> Hi John,
>>>>
>>>>> Yes, I have tried the same command you used below (gdb <java executable> <core file>), but I am seeing an error from this as well:
>>>>>
>>>>> warning: no loadable sections found in added symbol-file
>>>>> system-supplied DSO at 0x7fff671fe000  Core was generated by `java -Xmx512M -XX:MaxPermSize=128m -Djava.awt.headless=true -Dlogback.configura'.
>>>>> Program terminated with signal 6, Aborted.
>>>>> #0  0x0000003d7a432925 in raise () from /lib64/libc.so.6  Missing
>>>>> separate debuginfos, use: debuginfo-install
>>>>> java-1.7.0-openjdk-1.7.0.55-2.4.7.1.el6_5.x86_64
>>>>> (gdb)
>>>>
>>>> Getting into uncharted waters, but could you try enabling the debug repo and installing the debug rpm for your JDK?
>>>>
>>>> See http://fedoraproject.org/wiki/Java/Troubleshooting
>>>>
>>>>
>>>>> No, I actually performed a complete rebuild of the server - I have
>>>>> upgraded the versions of CentOS, Java, Ice, postgres, Python, etc.
>>>>> Once the new version was running, I was able to restore a the Omero
>>>>> database, using the process documented in
>>>>> http://www.openmicroscopy.org/site/support/omero5/sysadmins/server-
>>>>> up
>>>>> g
>>>>> rade.html#upgrade-your-database
>>>>>
>>>>> Would you like me to send though a complete list of the versions of the prerequisites we am currently running?
>>>>
>>>> Guess that's the next best step. Having never seen this before, we
>>>> have to start somewhere. :)
>>>>
>>>> ~J
>>>>
>>>>
>>>>> Thanks
>>>>> John
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: Josh Moore [mailto:josh at glencoesoftware.com]
>>>>> Sent: 23 May 2014 14:37
>>>>> To: John Webber (NBI)
>>>>> Cc: OME Users
>>>>> Subject: Re: [ome-users] Core dump attempting to perfrom bulk
>>>>> upload to Omero
>>>>>
>>>>>
>>>>> On May 23, 2014, at 2:35 PM, John Webber (NBI) wrote:
>>>>>
>>>>>> Hi Josh,
>>>>>>
>>>>>> Thanks for the response.
>>>>>>
>>>>>> I do indeed still have the core dump file located in:  /var/spool/abrt/ccpp-2014-05-22-17:20:04-1561.
>>>>>>
>>>>>> I have also installed gdb, but I have not used this before, and am not getting a lot of joy from trying to use it.  When I have attempted to run the following command:
>>>>>>
>>>>>> gdb '/usr/lib/jvm/jre-1.7.0-openjdk.x86_64/bin/java -Xmx512M
>>>>>> -XX:MaxPermSize=128m -Djava.awt.headless=true -Dlogback.configura'
>>>>>> /var/spool/abrt/ccpp-2014-05-22-17:20:04-1561/coredump
>>>>>>
>>>>>> I see the error:
>>>>>> warning: no loadable sections found in added symbol-file
>>>>>> system-supplied DSO at 0x7fff671fe000 Core was generated by `java -Xmx512M -XX:MaxPermSize=128m -Djava.awt.headless=true -Dlogback.configura'.
>>>>>> Program terminated with signal 6, Aborted.
>>>>>>
>>>>>> Am I running GDB incorrectly?
>>>>>
>>>>> Usually I'd just do:
>>>>>
>>>>> gdb '/usr/lib/jvm/jre-1.7.0-openjdk.x86_64/bin/java
>>>>> /var/spool/abrt/ccpp-2014-05-22-17:20:04-1561/coredump
>>>>>
>>>>>
>>>>>> You asked if this server had had any system maintenance recently:  This server is actually one that I have just upgraded to Omero 5.  It is a server that I am using to test the upgrade process.  It was therefore installed with Omero 4.4.8 (the current version of Omero that we run in production) and had several items of test data uploaded into Omero.  It was then upgraded to Omero 5 to ensure that the upgrade process is working correctly.  The upgrade itself all seems to be successful, but these bulk upload scripts are not functioning as expected.
>>>>>
>>>>> Definitely not. But just to be clear: other than the upgrade from 4.4.8 (which was working fine) to 5.0, no other upgrades took place?
>>>>>
>>>>>
>>>>>> One this to note, however, is that the bulk upload scripts have not been run on this specific server before, but are run regularly on the Production server with no issues.
>>>>>
>>>>> Is anything else running on this server? Any other Java processes?
>>>>>
>>>>> Cheers,
>>>>> ~Josh.
>>>>>
>>>>>
>>>>>
>>>>>> Thanks
>>>>>> John
>>>>>>
>>>>>>
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Josh Moore [mailto:josh at glencoesoftware.com]
>>>>>> Sent: 23 May 2014 11:36
>>>>>> To: John Webber (NBI)
>>>>>> Cc: OME Users
>>>>>> Subject: Re: [ome-users] Core dump attempting to perfrom bulk
>>>>>> upload to Omero
>>>>>>
>>>>>>
>>>>>> On May 23, 2014, at 10:33 AM, John Webber (NBI) wrote:
>>>>>>
>>>>>>> Hi Josh,
>>>>>>
>>>>>> Hi John,
>>>>>>
>>>>>>> Thanks for the quick response - as per normal! :-)
>>>>>>
>>>>>> Gladly. Thanks for all the log data!
>>>>>>
>>>>>>> Please see attached a zip file containing the files in my Omero log folder (/opt/Omero/var/log).  As well as the log files that normally reside in this location, I have also added the following files:
>>>>>>>
>>>>>>> bulk-upload-script.logThe logfile for my bulk upload script - so you can see what that is doing.
>>>>>>> var-log-messages-fileThe main server log (/var/log/messages) file.
>>>>>>> emailsThe emails which were sent for these core dumps.
>>>>>>>
>>>>>>> Any pointers you can give as to where things are going wrong would be gratefully received!  Do you also need to know the software versions that I have running, (for example Java and Python)?
>>>>>>
>>>>>> Can you check that you still have the file:
>>>>>>
>>>>>> May 22 17:20:11 v0246 abrt[2613]: Saved core dump of pid 1561
>>>>>> (/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.55.x86_64/jre/bin/java) to
>>>>>> /var/spool/abrt/ccpp-2014-05-22-17:20:04-1561 (741801984 bytes)
>>>>>>
>>>>>> If so, do you have gdb installed? It would be good to try to get a stack trace of what core dumped.
>>>>>>
>>>>>> Here are the restarts in your server:
>>>>>>
>>>>>> 2014-05-22 10:17:23,158 INFO  [    ome.services.util.ServerVersionCheck] (      main) OMERO Version: 5.0.0-ice35-b19 Ready.
>>>>>> 2014-05-22 10:58:26,515 INFO  [    ome.services.util.ServerVersionCheck] (      main) OMERO Version: 5.0.0-ice35-b19 Ready.
>>>>>> 2014-05-22 11:58:43,658 INFO  [    ome.services.util.ServerVersionCheck] (      main) OMERO Version: 5.0.0-ice35-b19 Ready.
>>>>>> 2014-05-22 14:52:47,780 INFO  [    ome.services.util.ServerVersionCheck] (      main) OMERO Version: 5.0.0-ice35-b19 Ready.
>>>>>> 2014-05-22 14:55:34,882 INFO  [    ome.services.util.ServerVersionCheck] (      main) OMERO Version: 5.0.0-ice35-b19 Ready.
>>>>>> 2014-05-22 15:15:26,549 INFO  [    ome.services.util.ServerVersionCheck] (      main) OMERO Version: 5.0.0-ice35-b19 Ready.
>>>>>> 2014-05-22 16:38:54,761 INFO  [    ome.services.util.ServerVersionCheck] (      main) OMERO Version: 5.0.0-ice35-b19 Ready.
>>>>>> 2014-05-22 17:11:40,885 INFO  [    ome.services.util.ServerVersionCheck] (      main) OMERO Version: 5.0.0-ice35-b19 Ready.
>>>>>> 2014-05-22 17:20:38,436 INFO  [    ome.services.util.ServerVersionCheck] (      main) OMERO Version: 5.0.0-ice35-b19 Ready.
>>>>>> 2014-05-22 17:23:23,171 INFO  [    ome.services.util.ServerVersionCheck] (      main) OMERO Version: 5.0.0-ice35-b19 Ready.
>>>>>>
>>>>>>
>>>>>> If they match any pattern you can detect, please let me know.
>>>>>>
>>>>>> Also, has there been any system maintenance recently? New Java versions, etc?
>>>>>> Cheers,
>>>>>> ~Josh.
>>>>>>
>>>>>>
>>>>>>> Thanks
>>>>>>> John
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: Josh Moore [mailto:josh at glencoesoftware.com]
>>>>>>> Sent: 22 May 2014 21:23
>>>>>>> To: John Webber (NBI)
>>>>>>> Subject: Re: [ome-users] Core dump attempting to perfrom bulk
>>>>>>> upload to Omero
>>>>>>>
>>>>>>>
>>>>>>> On May 22, 2014, at 6:17 PM, John Webber (NBI) wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>
>>>>>>> Hi John,
>>>>>>>
>>>>>>>> I am currently testing the same process on a new server which is built with Omero 5.0 and Ice 3.5.  I am seeing the following error, but not after a consistent time:
>>>>>>>>
>>>>>>>> Using session feccfbae-b08e-4298-9f3c-5f31139af486
>>>>>>>> (webberj at localhost:4064). Idle timeout: 10.0 min. Current group:
>>>>>>>> system Previous session expired for webberj on localhost:4064
>>>>>>>> InternalException: Failed to connect: exception
>>>>>>>> ::Glacier2::CannotCreateSessionException
>>>>>>>> {
>>>>>>>> reason = Server not fully initialized }
>>>>>>>
>>>>>>> This message is coming from the Java process and means that it's just restarted. Likely your server is bouncing, up/down up/down. Could you send us the OMERO log files under var/log zipped up, incl master.out and master.err?
>>>>>>>
>>>>>>>
>>>>>>>> A core dump is then generated.  The following are the log entries (/var/log/messages) when this occurs:
>>>>>>>> ...
>>>>>>>> The email that is sent as a result of this crash  is very long, but I have attached the first part of an example to this email.
>>>>>>>>
>>>>>>>> Does anyone have any ideas why Omero / Ice is crashing in this way?  Does this look like an Omero Issue, an Ice issue or a Java issue?
>>>>>>>
>>>>>>> It's the Java process which is aborting. More of the sos email could be useful if you could zip that up as well. For anyone who's wondering about OS & versions for a comparison to their own system:
>>>>>>>
>>>>>>> executable:     /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.55.x86_64/jre/bin/java
>>>>>>> kernel:         3.10.34-11.el6.centos.alt.x86_64
>>>>>>>
>>>>>>>
>>>>>>>> Thanks
>>>>>>>> John
>>>>>>>
>>>>>>> Cheers,
>>>>>>> ~Josh
>>>>>>> <logfile.tar.gz>
>> _______________________________________________
>> ome-users mailing list
>> ome-users at lists.openmicroscopy.org.uk
>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>
>
> The University of Dundee is a registered Scottish Charity, No: SC015096


The University of Dundee is a registered Scottish Charity, No: SC015096

More information about the ome-users mailing list