[ome-users] Problems with OMERO Imports

Thu Jan 7 13:10:56 GMT 2016

Hi Christian,

On Thu, Jan 7, 2016 at 10:45 AM, Christian Carsten Sachs
<c.sachs at fz-juelich.de> wrote:
...
> errors without any further info, so I've changed
> https://github.com/openmicroscopy/openmicroscopy/blob/v5.2.0/components/blitz/src/ome/formats/OMEROMetadataStoreClient.java#L1594
> line 1594/1594 to log.error(..., e) to log the exception as well, and used
> the resulting custom built jar ...
>
> Here's an example:
>
> 2016-01-06 18:32:25,202 ERROR [    ome.formats.OMEROMetadataStoreClient]
> (2-thread-2) Server error setting extended properties for Pixels:2451 Target
> file:sachs_202/2016-01/06/17-08-57.945/nd034.nd2
> Ice.ObjectNotExistException: null
...
>
> Any ideas what the problem might be? The strange thing is, that the
> '--auto_close --minutes_wait=0' seems to work fine with smaller files, and
> I'd like to have the import call within the script to return as fast as
> possible...

Unless you know of something in your setup which is actively closing
sessions / logging out, this sounds very much like a race condition
bug in auto_close. ObjectNotExistException means that the service
which is needed to perform further work has been closed and removed by
another process. If possible, I'd suggest you progress without
auto_close while we look into the issue.

Sorry for the troubles & thanks for the digging.
~Josh.

> Thank you,
> best regards
> Christian Sachs
>
> On 01/05/2016 08:54 AM, Josh Moore wrote:
>>
>> Happy New Year, Christian and others.
>>
>> On Fri, Dec 18, 2015 at 5:29 PM, Christian Carsten Sachs
>> <c.sachs at fz-juelich.de> wrote:
>>>
>>> Hello Josh,
>>>
>>> On 12/18/2015 09:58 AM, Josh Moore wrote:
>>>>
>>>>
>>>> Good morning,
>>>>
>>>> On Thu, Dec 17, 2015 at 6:26 PM, Christian Carsten Sachs
>>>> <c.sachs at fz-juelich.de> wrote:
>>>>
>>>>> On 12/17/2015 05:25 PM, Josh Moore wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Dec 17, 2015 at 5:10 PM, Christian Carsten Sachs
>>>>
>>>>
>>>>
>>>>>>> Nikon ND2 image stacks (.nd2 files) are imported using the command
>>>>>>> line
>>>>>>> client, in ln_s mode (called by a custom OMERO.dropbox-like script,
>>>>>>> enforcing some additional naming rules etc.).
>>>>>>
>>>>>>
>>>>>>
>>>>>> Is the script available somewhere?
>>>>>
>>>>>
>>>>>
>>>>> No, and we cannot easily upload is somewhere as it contains
>>>>> confidential
>>>>> info ...
>>>>
>>>>
>>>>
>>>> Understood, but debugging without seeing what's going on can be
>>>> difficult. Do you have log files from your script? Can you elide out
>>>> the sensitive information?
>>>>
>>>
>>> Here is the interaction with the OMERO cli tool:
>>>
>>> https://fz-juelich.sciebo.de/index.php/s/kUQMagLbBlPRQuk/download
>>>
>>> The Add_Annotation.py used is available here (third parties interested
>>> in this, please be careful and don't blindly install it ...)
>>> https://fz-juelich.sciebo.de/index.php/s/gzn9YU9QkfHvqW6/download
>>>
>>> The other script does not interact any further with OMERO besides the
>>> commands given in the transcript above.
>>
>>
>>
>> Thanks for these. I noticed the use of --auto_close. That itself
>> shouldn't cause problems as long as the number of imports is in
>> reason, but combined with a very low memory setting this could mean
>> you were not getting an exception that would otherwise shown up
>> earlier. If you are still having troubles after increasing the memory,
>> you might want to try temporarily removing --auto_close and see if you
>> get any more feedback.
>>
>> ...
>>
>>>>>>> Import proceeds like normal, and all positions are added to OMERO,
>>>>>>> but
>>>>>>> the actual image data remains inaccessible. (Only the clock icon in
>>>>>>> thumbnail view in the webclient).
>>>>>>>
>>>>>>> In Blitz-0 (or similarily the individual file log file), the
>>>>>>> following
>>>>>>> errors are visible:
>>>>>>>
>>>>>>> (repeated for each plane)
>>>>>>> 2015-12-17 16:07:10,458 ERROR [ ome.formats.OMEROMetadataStoreClient]
>>>>>>> (2-thread-4) Server error setting extended properties for Pixels:8145
>>>>>>> Target file:sachs_102/2015-12/17/15-07-53.074/nd009.nd2
>>>>
>>>>
>>>>
>>>> Unfortunately, that line:
>>>>
>>>>
>>>>
>>>> https://github.com/openmicroscopy/openmicroscopy/blob/v5.2.0/components/blitz/src/ome/formats/OMEROMetadataStoreClient.java#L1882
>>>>
>>>> swallows the exception. We could try to come up with a build that
>>>> would print out what is going on, but it might be that another log
>>>> file has related information. Could you possibly send us the
>>>> var/log/master* log files? The postgresql log files may also have some
>>>> information.
>>>>
>>>>
>>>
>>> master.err and master.out are both empty, and have not been touched
>>> since some time (they're supposed to be in the same folder as Blitz-0,
>>> right?)
>>>
>>> I can't access the Postgres log as of now.
>>
>>
>> No problem.
>>
>>
>>>>>> Is there any stack trace above this line? Perhaps send the whole log
>>>>>> (dir)?
>>>>>
>>>>>
>>>>>
>>>>> I did not see any stack trace, this is why I'm rather puzzled ...
>>>>> The whole logfile is some hundred megabytes large, I've split off the
>>>>> last import process (actually, since the last restart, as I had asked
>>>>> our admin to restart the system right before attempting the last
>>>>> import).
>>>>>
>>>>> As it's still too large to comfortably add it as attachment, please see
>>>>> following (temporary) link:
>>>>>
>>>>> https://fz-juelich.sciebo.de/index.php/s/zeltfZ4hfrJUMLU
>>>>
>>>>
>>>>
>>>> Downloaded, thanks, but you're right, there's not enough info there.
>>>>
>>>>
>>>>> My wording might've been misunderstandable: It has worked in both the
>>>>> testing and production system; the production system was tested with
>>>>> the
>>>>> import script and some datasets in late November and everything worked
>>>>> fine.
>>>>> Yesterday we wanted to 'go live' and let users start to import data ...
>>>>> Thus we're a bit puzzled that problems occur now ...
>>>>
>>>>
>>>>
>>>> One guess: is this file larger than others that were attempted
>>>> previously? The memory setting of your server is not particularly high
>>>> for production:
>>>>
>>>> 2015-12-17 15:03:48,260 INFO  [
>>>> ome.services.util.JvmSettingsCheck] (      main) Max Memory (MB):   =
>>>>    2185
>>>>
>>>
>>> Good hint!
>>> A little background here: A colleague has initially set up the server,
>>> and now left the institute, I 'inherited' the project and scripted some
>>> necessities for our workflows, and the institute's admin is in physical
>>> control of the machine and administrative tasks there, so I'm not aware
>>> of every minute detail of the setup yet.
>>>
>>> The machine has 24 GB of physical RAM, I'll change the settings to give
>>> OMERO access to more. Our image files (multi position multi channel time
>>> lapse microscopy) range from 25-250 GB, so the current settings might
>>> really be too small.
>>>
>>> Maybe that even solves the problem.
>>>
>>> Unfortunately, I can't try it out now, and I'll be away for the
>>> holidays. I'll get back to test it next year.
>>
>>
>> Sounds good. Let us know how it goes.
>> ~Josh.
>>
>>
>>> Thanks a lot,
>>> best Regards,
>>> Christian Sachs
>>>
>>>> Cheers,
>>>> ~Josh
>>>>
>>>>
>>>>> PS: I forgot to mention that the file itself is properly openable in
>>>>> Fiji with current Bio-Formats.
>>>>>
>>>>> Thank you very much,
>>>>> Best regards
>>>>>
>>>>> Christian Sachs
>>>>>
>>>>>>> We're using OMERO 5.2.0 on a Windows server (which has - to my
>>>>>>> knowledge
>>>>>>> - been upgraded multiple times from earlier versions).
>>>>>>> As of the last three files imported, none was imported properly, we
>>>>>>> have
>>>>>>> told our users to stop imports till the problem is resolved.
>>>>>>>
>>>>>>> Thank you,
>>>>>>> best regards
>>>>>>>
>>>>>>> Christian Sachs
>>>>
>>>>
>>>> _______________________________________________
>>>> ome-users mailing list
>>>> ome-users at lists.openmicroscopy.org.uk
>>>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>>>>
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------------------------
>>>
>>> ------------------------------------------------------------------------------------------------
>>> Forschungszentrum Juelich GmbH
>>> 52425 Juelich
>>> Sitz der Gesellschaft: Juelich
>>> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
>>> Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
>>> Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
>>> Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
>>> Prof. Dr. Sebastian M. Schmidt
>>>
>>> ------------------------------------------------------------------------------------------------
>>>
>>> ------------------------------------------------------------------------------------------------
>>>
>>> _______________________________________________
>>> ome-users mailing list
>>> ome-users at lists.openmicroscopy.org.uk
>>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>>
>> _______________________________________________
>> ome-users mailing list
>> ome-users at lists.openmicroscopy.org.uk
>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>>
> _______________________________________________
> ome-users mailing list
> ome-users at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users