[ome-users] Problems with OMERO Imports

Thu Feb 11 16:40:52 GMT 2016

A quick update:

On Thu, Jan 7, 2016 at 2:10 PM, Josh Moore <josh at glencoesoftware.com> wrote:
> On Thu, Jan 7, 2016 at 10:45 AM, Christian Carsten Sachs
>>
>> Any ideas what the problem might be? The strange thing is, that the
>> '--auto_close --minutes_wait=0' seems to work fine with smaller files, and
>> I'd like to have the import call within the script to return as fast as
>> possible...
>
> Unless you know of something in your setup which is actively closing
> sessions / logging out, this sounds very much like a race condition
> bug in auto_close. ObjectNotExistException means that the service
> which is needed to perform further work has been closed and removed by
> another process. If possible, I'd suggest you progress without
> auto_close while we look into the issue.

For anyone who is runs into similar issues, please see
https://github.com/openmicroscopy/openmicroscopy/pull/4473

Cheers,
~Josh.

> Sorry for the troubles & thanks for the digging.
> ~Josh.
>
>
>> Thank you,
>> best regards
>> Christian Sachs
>>
>> On 01/05/2016 08:54 AM, Josh Moore wrote:
>>>
>>> Happy New Year, Christian and others.
>>>
>>> On Fri, Dec 18, 2015 at 5:29 PM, Christian Carsten Sachs
>>> <c.sachs at fz-juelich.de> wrote:
>>>>
>>>> Hello Josh,
>>>>
>>>> On 12/18/2015 09:58 AM, Josh Moore wrote:
>>>>>
>>>>>
>>>>> Good morning,
>>>>>
>>>>> On Thu, Dec 17, 2015 at 6:26 PM, Christian Carsten Sachs
>>>>> <c.sachs at fz-juelich.de> wrote:
>>>>>
>>>>>> On 12/17/2015 05:25 PM, Josh Moore wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Dec 17, 2015 at 5:10 PM, Christian Carsten Sachs
>>>>>
>>>>>
>>>>>
>>>>>>>> Nikon ND2 image stacks (.nd2 files) are imported using the command
>>>>>>>> line
>>>>>>>> client, in ln_s mode (called by a custom OMERO.dropbox-like script,
>>>>>>>> enforcing some additional naming rules etc.).
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Is the script available somewhere?
>>>>>>
>>>>>>
>>>>>>
>>>>>> No, and we cannot easily upload is somewhere as it contains
>>>>>> confidential
>>>>>> info ...
>>>>>
>>>>>
>>>>>
>>>>> Understood, but debugging without seeing what's going on can be
>>>>> difficult. Do you have log files from your script? Can you elide out
>>>>> the sensitive information?
>>>>>
>>>>
>>>> Here is the interaction with the OMERO cli tool:
>>>>
>>>> https://fz-juelich.sciebo.de/index.php/s/kUQMagLbBlPRQuk/download
>>>>
>>>> The Add_Annotation.py used is available here (third parties interested
>>>> in this, please be careful and don't blindly install it ...)
>>>> https://fz-juelich.sciebo.de/index.php/s/gzn9YU9QkfHvqW6/download
>>>>
>>>> The other script does not interact any further with OMERO besides the
>>>> commands given in the transcript above.
>>>
>>>
>>>
>>> Thanks for these. I noticed the use of --auto_close. That itself
>>> shouldn't cause problems as long as the number of imports is in
>>> reason, but combined with a very low memory setting this could mean
>>> you were not getting an exception that would otherwise shown up
>>> earlier. If you are still having troubles after increasing the memory,
>>> you might want to try temporarily removing --auto_close and see if you
>>> get any more feedback.
>>>
>>> ...
>>>
>>>>>>>> Import proceeds like normal, and all positions are added to OMERO,
>>>>>>>> but
>>>>>>>> the actual image data remains inaccessible. (Only the clock icon in
>>>>>>>> thumbnail view in the webclient).
>>>>>>>>
>>>>>>>> In Blitz-0 (or similarily the individual file log file), the
>>>>>>>> following
>>>>>>>> errors are visible:
>>>>>>>>
>>>>>>>> (repeated for each plane)
>>>>>>>> 2015-12-17 16:07:10,458 ERROR [ ome.formats.OMEROMetadataStoreClient]
>>>>>>>> (2-thread-4) Server error setting extended properties for Pixels:8145
>>>>>>>> Target file:sachs_102/2015-12/17/15-07-53.074/nd009.nd2
>>>>>
>>>>>
>>>>>
>>>>> Unfortunately, that line:
>>>>>
>>>>>
>>>>>
>>>>> https://github.com/openmicroscopy/openmicroscopy/blob/v5.2.0/components/blitz/src/ome/formats/OMEROMetadataStoreClient.java#L1882
>>>>>
>>>>> swallows the exception. We could try to come up with a build that
>>>>> would print out what is going on, but it might be that another log
>>>>> file has related information. Could you possibly send us the
>>>>> var/log/master* log files? The postgresql log files may also have some
>>>>> information.
>>>>>
>>>>>
>>>>
>>>> master.err and master.out are both empty, and have not been touched
>>>> since some time (they're supposed to be in the same folder as Blitz-0,
>>>> right?)
>>>>
>>>> I can't access the Postgres log as of now.
>>>
>>>
>>> No problem.
>>>
>>>
>>>>>>> Is there any stack trace above this line? Perhaps send the whole log
>>>>>>> (dir)?
>>>>>>
>>>>>>
>>>>>>
>>>>>> I did not see any stack trace, this is why I'm rather puzzled ...
>>>>>> The whole logfile is some hundred megabytes large, I've split off the
>>>>>> last import process (actually, since the last restart, as I had asked
>>>>>> our admin to restart the system right before attempting the last
>>>>>> import).
>>>>>>
>>>>>> As it's still too large to comfortably add it as attachment, please see
>>>>>> following (temporary) link:
>>>>>>
>>>>>> https://fz-juelich.sciebo.de/index.php/s/zeltfZ4hfrJUMLU
>>>>>
>>>>>
>>>>>
>>>>> Downloaded, thanks, but you're right, there's not enough info there.
>>>>>
>>>>>
>>>>>> My wording might've been misunderstandable: It has worked in both the
>>>>>> testing and production system; the production system was tested with
>>>>>> the
>>>>>> import script and some datasets in late November and everything worked
>>>>>> fine.
>>>>>> Yesterday we wanted to 'go live' and let users start to import data ...
>>>>>> Thus we're a bit puzzled that problems occur now ...
>>>>>
>>>>>
>>>>>
>>>>> One guess: is this file larger than others that were attempted
>>>>> previously? The memory setting of your server is not particularly high
>>>>> for production:
>>>>>
>>>>> 2015-12-17 15:03:48,260 INFO  [
>>>>> ome.services.util.JvmSettingsCheck] (      main) Max Memory (MB):   =
>>>>>    2185
>>>>>
>>>>
>>>> Good hint!
>>>> A little background here: A colleague has initially set up the server,
>>>> and now left the institute, I 'inherited' the project and scripted some
>>>> necessities for our workflows, and the institute's admin is in physical
>>>> control of the machine and administrative tasks there, so I'm not aware
>>>> of every minute detail of the setup yet.
>>>>
>>>> The machine has 24 GB of physical RAM, I'll change the settings to give
>>>> OMERO access to more. Our image files (multi position multi channel time
>>>> lapse microscopy) range from 25-250 GB, so the current settings might
>>>> really be too small.
>>>>
>>>> Maybe that even solves the problem.
>>>>
>>>> Unfortunately, I can't try it out now, and I'll be away for the
>>>> holidays. I'll get back to test it next year.
>>>
>>>
>>> Sounds good. Let us know how it goes.
>>> ~Josh.
>>>
>>>
>>>> Thanks a lot,
>>>> best Regards,
>>>> Christian Sachs
>>>>
>>>>> Cheers,
>>>>> ~Josh
>>>>>
>>>>>
>>>>>> PS: I forgot to mention that the file itself is properly openable in
>>>>>> Fiji with current Bio-Formats.
>>>>>>
>>>>>> Thank you very much,
>>>>>> Best regards
>>>>>>
>>>>>> Christian Sachs
>>>>>>
>>>>>>>> We're using OMERO 5.2.0 on a Windows server (which has - to my
>>>>>>>> knowledge
>>>>>>>> - been upgraded multiple times from earlier versions).
>>>>>>>> As of the last three files imported, none was imported properly, we
>>>>>>>> have
>>>>>>>> told our users to stop imports till the problem is resolved.
>>>>>>>>
>>>>>>>> Thank you,
>>>>>>>> best regards
>>>>>>>>
>>>>>>>> Christian Sachs
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> ome-users mailing list
>>>>> ome-users at lists.openmicroscopy.org.uk
>>>>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>>>>>
>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------------------------------------------
>>>>
>>>> ------------------------------------------------------------------------------------------------
>>>> Forschungszentrum Juelich GmbH
>>>> 52425 Juelich
>>>> Sitz der Gesellschaft: Juelich
>>>> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
>>>> Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
>>>> Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
>>>> Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
>>>> Prof. Dr. Sebastian M. Schmidt
>>>>
>>>> ------------------------------------------------------------------------------------------------
>>>>
>>>> ------------------------------------------------------------------------------------------------
>>>>
>>>> _______________________________________________
>>>> ome-users mailing list
>>>> ome-users at lists.openmicroscopy.org.uk
>>>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>>>
>>> _______________________________________________
>>> ome-users mailing list
>>> ome-users at lists.openmicroscopy.org.uk
>>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>>>
>> _______________________________________________
>> ome-users mailing list
>> ome-users at lists.openmicroscopy.org.uk
>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users