[ome-users] Problems with OMERO Imports

Tue Jan 5 07:54:22 GMT 2016

Happy New Year, Christian and others.

On Fri, Dec 18, 2015 at 5:29 PM, Christian Carsten Sachs
<c.sachs at fz-juelich.de> wrote:
> Hello Josh,
>
> On 12/18/2015 09:58 AM, Josh Moore wrote:
>>
>> Good morning,
>>
>> On Thu, Dec 17, 2015 at 6:26 PM, Christian Carsten Sachs
>> <c.sachs at fz-juelich.de> wrote:
>>
>>> On 12/17/2015 05:25 PM, Josh Moore wrote:
>>>>
>>>>
>>>> On Thu, Dec 17, 2015 at 5:10 PM, Christian Carsten Sachs
>>
>>
>>>>> Nikon ND2 image stacks (.nd2 files) are imported using the command line
>>>>> client, in ln_s mode (called by a custom OMERO.dropbox-like script,
>>>>> enforcing some additional naming rules etc.).
>>>>
>>>>
>>>> Is the script available somewhere?
>>>
>>>
>>> No, and we cannot easily upload is somewhere as it contains confidential
>>> info ...
>>
>>
>> Understood, but debugging without seeing what's going on can be
>> difficult. Do you have log files from your script? Can you elide out
>> the sensitive information?
>>
>
> Here is the interaction with the OMERO cli tool:
>
> https://fz-juelich.sciebo.de/index.php/s/kUQMagLbBlPRQuk/download
>
> The Add_Annotation.py used is available here (third parties interested
> in this, please be careful and don't blindly install it ...)
> https://fz-juelich.sciebo.de/index.php/s/gzn9YU9QkfHvqW6/download
>
> The other script does not interact any further with OMERO besides the
> commands given in the transcript above.

Thanks for these. I noticed the use of --auto_close. That itself
shouldn't cause problems as long as the number of imports is in
reason, but combined with a very low memory setting this could mean
you were not getting an exception that would otherwise shown up
earlier. If you are still having troubles after increasing the memory,
you might want to try temporarily removing --auto_close and see if you
get any more feedback.

...

>>>>> Import proceeds like normal, and all positions are added to OMERO, but
>>>>> the actual image data remains inaccessible. (Only the clock icon in
>>>>> thumbnail view in the webclient).
>>>>>
>>>>> In Blitz-0 (or similarily the individual file log file), the following
>>>>> errors are visible:
>>>>>
>>>>> (repeated for each plane)
>>>>> 2015-12-17 16:07:10,458 ERROR [ ome.formats.OMEROMetadataStoreClient]
>>>>> (2-thread-4) Server error setting extended properties for Pixels:8145
>>>>> Target file:sachs_102/2015-12/17/15-07-53.074/nd009.nd2
>>
>>
>> Unfortunately, that line:
>>
>>
>> https://github.com/openmicroscopy/openmicroscopy/blob/v5.2.0/components/blitz/src/ome/formats/OMEROMetadataStoreClient.java#L1882
>>
>> swallows the exception. We could try to come up with a build that
>> would print out what is going on, but it might be that another log
>> file has related information. Could you possibly send us the
>> var/log/master* log files? The postgresql log files may also have some
>> information.
>>
>>
>
> master.err and master.out are both empty, and have not been touched
> since some time (they're supposed to be in the same folder as Blitz-0,
> right?)
>
> I can't access the Postgres log as of now.

No problem.

>>>> Is there any stack trace above this line? Perhaps send the whole log
>>>> (dir)?
>>>
>>>
>>> I did not see any stack trace, this is why I'm rather puzzled ...
>>> The whole logfile is some hundred megabytes large, I've split off the
>>> last import process (actually, since the last restart, as I had asked
>>> our admin to restart the system right before attempting the last import).
>>>
>>> As it's still too large to comfortably add it as attachment, please see
>>> following (temporary) link:
>>>
>>> https://fz-juelich.sciebo.de/index.php/s/zeltfZ4hfrJUMLU
>>
>>
>> Downloaded, thanks, but you're right, there's not enough info there.
>>
>>
>>> My wording might've been misunderstandable: It has worked in both the
>>> testing and production system; the production system was tested with the
>>> import script and some datasets in late November and everything worked
>>> fine.
>>> Yesterday we wanted to 'go live' and let users start to import data ...
>>> Thus we're a bit puzzled that problems occur now ...
>>
>>
>> One guess: is this file larger than others that were attempted
>> previously? The memory setting of your server is not particularly high
>> for production:
>>
>> 2015-12-17 15:03:48,260 INFO  [
>> ome.services.util.JvmSettingsCheck] (      main) Max Memory (MB):   =
>>   2185
>>
>
> Good hint!
> A little background here: A colleague has initially set up the server,
> and now left the institute, I 'inherited' the project and scripted some
> necessities for our workflows, and the institute's admin is in physical
> control of the machine and administrative tasks there, so I'm not aware
> of every minute detail of the setup yet.
>
> The machine has 24 GB of physical RAM, I'll change the settings to give
> OMERO access to more. Our image files (multi position multi channel time
> lapse microscopy) range from 25-250 GB, so the current settings might
> really be too small.
>
> Maybe that even solves the problem.
>
> Unfortunately, I can't try it out now, and I'll be away for the
> holidays. I'll get back to test it next year.

Sounds good. Let us know how it goes.
~Josh.

> Thanks a lot,
> best Regards,
> Christian Sachs
>
>> Cheers,
>> ~Josh
>>
>>
>>> PS: I forgot to mention that the file itself is properly openable in
>>> Fiji with current Bio-Formats.
>>>
>>> Thank you very much,
>>> Best regards
>>>
>>> Christian Sachs
>>>
>>>>> We're using OMERO 5.2.0 on a Windows server (which has - to my
>>>>> knowledge
>>>>> - been upgraded multiple times from earlier versions).
>>>>> As of the last three files imported, none was imported properly, we
>>>>> have
>>>>> told our users to stop imports till the problem is resolved.
>>>>>
>>>>> Thank you,
>>>>> best regards
>>>>>
>>>>> Christian Sachs
>>
>> _______________________________________________
>> ome-users mailing list
>> ome-users at lists.openmicroscopy.org.uk
>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>>
>
>
> ------------------------------------------------------------------------------------------------
> ------------------------------------------------------------------------------------------------
> Forschungszentrum Juelich GmbH
> 52425 Juelich
> Sitz der Gesellschaft: Juelich
> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
> Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
> Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
> Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
> Prof. Dr. Sebastian M. Schmidt
> ------------------------------------------------------------------------------------------------
> ------------------------------------------------------------------------------------------------
>
> _______________________________________________
> ome-users mailing list
> ome-users at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users