[ome-users] Problems with OMERO Imports

Josh Moore josh at glencoesoftware.com
Mon Jan 11 08:16:13 GMT 2016 

Hi Christian,

On Fri, Jan 8, 2016 at 6:21 PM, Christian Carsten Sachs
<c.sachs at fz-juelich.de> wrote:
> Hi Josh,
>
> as a nice way to start the weekend, I found a surprising way to work-around
> the bug (so far?):

Glad to hear it.


> Thanks for the hint with the closed services!
> Setting omero.sessions.timeout to 86400000 (1 day instead of 10 minutes) has
> solved the problem (so far?)!
>
> Is it somehow possible, that while the running importer client keeps the
> session and the attached (downstream) services alive, an import process
> which instantly detaches from it leaves the steps further downstream to be
> cleaned up by some kind of session management?
> (I so far didn't really dig into the distributed nature of OMERO's
> structure, so I might be totally off with that interpretation ...)

I'd think your guess is correct or at least very close. You could test
it by trying to keep one of the sessions created for import open:

  bin/omero -k $UUID -s $SERVER sessions keepalive


> Are there any ill effects to be expected by having such a high session
> timeout?

It depends on your usage, really. You might monitor your server's RAM
consumption for a while to make sure nothing is growing without
bounds. A number of users who load a large number of large services
and then don't log out, for example, could cause issues.

Cheers,
~Josh.


> Best regards,
> Christian Sachs
>
>
> On 01/07/2016 02:10 PM, Josh Moore wrote:
>>
>> Hi Christian,
>>
>>
>> On Thu, Jan 7, 2016 at 10:45 AM, Christian Carsten Sachs
>> <c.sachs at fz-juelich.de> wrote:
>> ...
>>>
>>> errors without any further info, so I've changed
>>>
>>> https://github.com/openmicroscopy/openmicroscopy/blob/v5.2.0/components/blitz/src/ome/formats/OMEROMetadataStoreClient.java#L1594
>>> line 1594/1594 to log.error(..., e) to log the exception as well, and
>>> used
>>> the resulting custom built jar ...
>>>
>>> Here's an example:
>>>
>>> 2016-01-06 18:32:25,202 ERROR [    ome.formats.OMEROMetadataStoreClient]
>>> (2-thread-2) Server error setting extended properties for Pixels:2451
>>> Target
>>> file:sachs_202/2016-01/06/17-08-57.945/nd034.nd2
>>> Ice.ObjectNotExistException: null
>>
>> ...
>>>
>>>
>>> Any ideas what the problem might be? The strange thing is, that the
>>> '--auto_close --minutes_wait=0' seems to work fine with smaller files,
>>> and
>>> I'd like to have the import call within the script to return as fast as
>>> possible...
>>
>>
>> Unless you know of something in your setup which is actively closing
>> sessions / logging out, this sounds very much like a race condition
>> bug in auto_close. ObjectNotExistException means that the service
>> which is needed to perform further work has been closed and removed by
>> another process. If possible, I'd suggest you progress without
>> auto_close while we look into the issue.
>>
>> Sorry for the troubles & thanks for the digging.
>> ~Josh.
>>
>>
>>> Thank you,
>>> best regards
>>> Christian Sachs
>>>
>>> On 01/05/2016 08:54 AM, Josh Moore wrote:
>>>>
>>>>
>>>> Happy New Year, Christian and others.
>>>>
>>>> On Fri, Dec 18, 2015 at 5:29 PM, Christian Carsten Sachs
>>>> <c.sachs at fz-juelich.de> wrote:
>>>>>
>>>>>
>>>>> Hello Josh,
>>>>>
>>>>> On 12/18/2015 09:58 AM, Josh Moore wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> Good morning,
>>>>>>
>>>>>> On Thu, Dec 17, 2015 at 6:26 PM, Christian Carsten Sachs
>>>>>> <c.sachs at fz-juelich.de> wrote:
>>>>>>
>>>>>>> On 12/17/2015 05:25 PM, Josh Moore wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Dec 17, 2015 at 5:10 PM, Christian Carsten Sachs
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>>>> Nikon ND2 image stacks (.nd2 files) are imported using the command
>>>>>>>>> line
>>>>>>>>> client, in ln_s mode (called by a custom OMERO.dropbox-like script,
>>>>>>>>> enforcing some additional naming rules etc.).
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Is the script available somewhere?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> No, and we cannot easily upload is somewhere as it contains
>>>>>>> confidential
>>>>>>> info ...
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Understood, but debugging without seeing what's going on can be
>>>>>> difficult. Do you have log files from your script? Can you elide out
>>>>>> the sensitive information?
>>>>>>
>>>>>
>>>>> Here is the interaction with the OMERO cli tool:
>>>>>
>>>>> https://fz-juelich.sciebo.de/index.php/s/kUQMagLbBlPRQuk/download
>>>>>
>>>>> The Add_Annotation.py used is available here (third parties interested
>>>>> in this, please be careful and don't blindly install it ...)
>>>>> https://fz-juelich.sciebo.de/index.php/s/gzn9YU9QkfHvqW6/download
>>>>>
>>>>> The other script does not interact any further with OMERO besides the
>>>>> commands given in the transcript above.
>>>>
>>>>
>>>>
>>>>
>>>> Thanks for these. I noticed the use of --auto_close. That itself
>>>> shouldn't cause problems as long as the number of imports is in
>>>> reason, but combined with a very low memory setting this could mean
>>>> you were not getting an exception that would otherwise shown up
>>>> earlier. If you are still having troubles after increasing the memory,
>>>> you might want to try temporarily removing --auto_close and see if you
>>>> get any more feedback.
>>>>
>>>> ...
>>>>
>>>>>>>>> Import proceeds like normal, and all positions are added to OMERO,
>>>>>>>>> but
>>>>>>>>> the actual image data remains inaccessible. (Only the clock icon in
>>>>>>>>> thumbnail view in the webclient).
>>>>>>>>>
>>>>>>>>> In Blitz-0 (or similarily the individual file log file), the
>>>>>>>>> following
>>>>>>>>> errors are visible:
>>>>>>>>>
>>>>>>>>> (repeated for each plane)
>>>>>>>>> 2015-12-17 16:07:10,458 ERROR [
>>>>>>>>> ome.formats.OMEROMetadataStoreClient]
>>>>>>>>> (2-thread-4) Server error setting extended properties for
>>>>>>>>> Pixels:8145
>>>>>>>>> Target file:sachs_102/2015-12/17/15-07-53.074/nd009.nd2
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Unfortunately, that line:
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> https://github.com/openmicroscopy/openmicroscopy/blob/v5.2.0/components/blitz/src/ome/formats/OMEROMetadataStoreClient.java#L1882
>>>>>>
>>>>>> swallows the exception. We could try to come up with a build that
>>>>>> would print out what is going on, but it might be that another log
>>>>>> file has related information. Could you possibly send us the
>>>>>> var/log/master* log files? The postgresql log files may also have some
>>>>>> information.
>>>>>>
>>>>>>
>>>>>
>>>>> master.err and master.out are both empty, and have not been touched
>>>>> since some time (they're supposed to be in the same folder as Blitz-0,
>>>>> right?)
>>>>>
>>>>> I can't access the Postgres log as of now.
>>>>
>>>>
>>>>
>>>> No problem.
>>>>
>>>>
>>>>>>>> Is there any stack trace above this line? Perhaps send the whole log
>>>>>>>> (dir)?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I did not see any stack trace, this is why I'm rather puzzled ...
>>>>>>> The whole logfile is some hundred megabytes large, I've split off the
>>>>>>> last import process (actually, since the last restart, as I had asked
>>>>>>> our admin to restart the system right before attempting the last
>>>>>>> import).
>>>>>>>
>>>>>>> As it's still too large to comfortably add it as attachment, please
>>>>>>> see
>>>>>>> following (temporary) link:
>>>>>>>
>>>>>>> https://fz-juelich.sciebo.de/index.php/s/zeltfZ4hfrJUMLU
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Downloaded, thanks, but you're right, there's not enough info there.
>>>>>>
>>>>>>
>>>>>>> My wording might've been misunderstandable: It has worked in both the
>>>>>>> testing and production system; the production system was tested with
>>>>>>> the
>>>>>>> import script and some datasets in late November and everything
>>>>>>> worked
>>>>>>> fine.
>>>>>>> Yesterday we wanted to 'go live' and let users start to import data
>>>>>>> ...
>>>>>>> Thus we're a bit puzzled that problems occur now ...
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> One guess: is this file larger than others that were attempted
>>>>>> previously? The memory setting of your server is not particularly high
>>>>>> for production:
>>>>>>
>>>>>> 2015-12-17 15:03:48,260 INFO  [
>>>>>> ome.services.util.JvmSettingsCheck] (      main) Max Memory (MB):   =
>>>>>>     2185
>>>>>>
>>>>>
>>>>> Good hint!
>>>>> A little background here: A colleague has initially set up the server,
>>>>> and now left the institute, I 'inherited' the project and scripted some
>>>>> necessities for our workflows, and the institute's admin is in physical
>>>>> control of the machine and administrative tasks there, so I'm not aware
>>>>> of every minute detail of the setup yet.
>>>>>
>>>>> The machine has 24 GB of physical RAM, I'll change the settings to give
>>>>> OMERO access to more. Our image files (multi position multi channel
>>>>> time
>>>>> lapse microscopy) range from 25-250 GB, so the current settings might
>>>>> really be too small.
>>>>>
>>>>> Maybe that even solves the problem.
>>>>>
>>>>> Unfortunately, I can't try it out now, and I'll be away for the
>>>>> holidays. I'll get back to test it next year.
>>>>
>>>>
>>>>
>>>> Sounds good. Let us know how it goes.
>>>> ~Josh.
>>>>
>>>>
>>>>> Thanks a lot,
>>>>> best Regards,
>>>>> Christian Sachs
>>>>>
>>>>>> Cheers,
>>>>>> ~Josh
>>>>>>
>>>>>>
>>>>>>> PS: I forgot to mention that the file itself is properly openable in
>>>>>>> Fiji with current Bio-Formats.
>>>>>>>
>>>>>>> Thank you very much,
>>>>>>> Best regards
>>>>>>>
>>>>>>> Christian Sachs
>>>>>>>
>>>>>>>>> We're using OMERO 5.2.0 on a Windows server (which has - to my
>>>>>>>>> knowledge
>>>>>>>>> - been upgraded multiple times from earlier versions).
>>>>>>>>> As of the last three files imported, none was imported properly, we
>>>>>>>>> have
>>>>>>>>> told our users to stop imports till the problem is resolved.
>>>>>>>>>
>>>>>>>>> Thank you,
>>>>>>>>> best regards
>>>>>>>>>
>>>>>>>>> Christian Sachs
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> ome-users mailing list
>>>>>> ome-users at lists.openmicroscopy.org.uk
>>>>>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------------------------------
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------------------------------
>>>>> Forschungszentrum Juelich GmbH
>>>>> 52425 Juelich
>>>>> Sitz der Gesellschaft: Juelich
>>>>> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
>>>>> Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
>>>>> Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
>>>>> Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
>>>>> Prof. Dr. Sebastian M. Schmidt
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------------------------------
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------------------------------
>>>>>
>>>>> _______________________________________________
>>>>> ome-users mailing list
>>>>> ome-users at lists.openmicroscopy.org.uk
>>>>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>>>>
>>>>
>>>> _______________________________________________
>>>> ome-users mailing list
>>>> ome-users at lists.openmicroscopy.org.uk
>>>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>>>>
>>> _______________________________________________
>>> ome-users mailing list
>>> ome-users at lists.openmicroscopy.org.uk
>>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>>
>> _______________________________________________
>> ome-users mailing list
>> ome-users at lists.openmicroscopy.org.uk
>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>>
> _______________________________________________
> ome-users mailing list
> ome-users at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users

More information about the ome-users mailing list