[ome-users] Problems with OMERO Imports

Fri Dec 18 16:29:36 GMT 2015

Hello Josh,

On 12/18/2015 09:58 AM, Josh Moore wrote:
> Good morning,
>
> On Thu, Dec 17, 2015 at 6:26 PM, Christian Carsten Sachs
> <c.sachs at fz-juelich.de> wrote:
>
>> On 12/17/2015 05:25 PM, Josh Moore wrote:
>>>
>>> On Thu, Dec 17, 2015 at 5:10 PM, Christian Carsten Sachs
>
>>>> Nikon ND2 image stacks (.nd2 files) are imported using the command line
>>>> client, in ln_s mode (called by a custom OMERO.dropbox-like script,
>>>> enforcing some additional naming rules etc.).
>>>
>>> Is the script available somewhere?
>>
>> No, and we cannot easily upload is somewhere as it contains confidential
>> info ...
>
> Understood, but debugging without seeing what's going on can be
> difficult. Do you have log files from your script? Can you elide out
> the sensitive information?
>
>

Here is the interaction with the OMERO cli tool:

https://fz-juelich.sciebo.de/index.php/s/kUQMagLbBlPRQuk/download

The Add_Annotation.py used is available here (third parties interested
in this, please be careful and don't blindly install it ...)
https://fz-juelich.sciebo.de/index.php/s/gzn9YU9QkfHvqW6/download

The other script does not interact any further with OMERO besides the
commands given in the transcript above.

>> ... (Btw, having more complex import possibilities with OMERO.dropbox would
>> be great, should we investigate creating a patch to OMERO.dropbox to
>> enhance it with said functionality?)
>
> In general, sure! You might be interested in
> https://github.com/openmicroscopy/openmicroscopy/pull/4367 which will
> hopefully be a part of 5.2.2 which will already simplify your dataset
> creation.
>
>

Thank you, this looks interesting; I'll investigate this next year.

>>>> Import proceeds like normal, and all positions are added to OMERO, but
>>>> the actual image data remains inaccessible. (Only the clock icon in
>>>> thumbnail view in the webclient).
>>>>
>>>> In Blitz-0 (or similarily the individual file log file), the following
>>>> errors are visible:
>>>>
>>>> (repeated for each plane)
>>>> 2015-12-17 16:07:10,458 ERROR [ ome.formats.OMEROMetadataStoreClient]
>>>> (2-thread-4) Server error setting extended properties for Pixels:8145
>>>> Target file:sachs_102/2015-12/17/15-07-53.074/nd009.nd2
>
> Unfortunately, that line:
>
> https://github.com/openmicroscopy/openmicroscopy/blob/v5.2.0/components/blitz/src/ome/formats/OMEROMetadataStoreClient.java#L1882
>
> swallows the exception. We could try to come up with a build that
> would print out what is going on, but it might be that another log
> file has related information. Could you possibly send us the
> var/log/master* log files? The postgresql log files may also have some
> information.
>
>

master.err and master.out are both empty, and have not been touched
since some time (they're supposed to be in the same folder as Blitz-0,
right?)

I can't access the Postgres log as of now.

>>> Is there any stack trace above this line? Perhaps send the whole log
>>> (dir)?
>>
>> I did not see any stack trace, this is why I'm rather puzzled ...
>> The whole logfile is some hundred megabytes large, I've split off the
>> last import process (actually, since the last restart, as I had asked
>> our admin to restart the system right before attempting the last import).
>>
>> As it's still too large to comfortably add it as attachment, please see
>> following (temporary) link:
>>
>> https://fz-juelich.sciebo.de/index.php/s/zeltfZ4hfrJUMLU
>
> Downloaded, thanks, but you're right, there's not enough info there.
>
>
>> My wording might've been misunderstandable: It has worked in both the
>> testing and production system; the production system was tested with the
>> import script and some datasets in late November and everything worked fine.
>> Yesterday we wanted to 'go live' and let users start to import data ...
>> Thus we're a bit puzzled that problems occur now ...
>
> One guess: is this file larger than others that were attempted
> previously? The memory setting of your server is not particularly high
> for production:
>
> 2015-12-17 15:03:48,260 INFO  [
> ome.services.util.JvmSettingsCheck] (      main) Max Memory (MB):   =
>   2185
>

Good hint!
A little background here: A colleague has initially set up the server,
and now left the institute, I 'inherited' the project and scripted some
necessities for our workflows, and the institute's admin is in physical
control of the machine and administrative tasks there, so I'm not aware
of every minute detail of the setup yet.

The machine has 24 GB of physical RAM, I'll change the settings to give
OMERO access to more. Our image files (multi position multi channel time
lapse microscopy) range from 25-250 GB, so the current settings might
really be too small.

Maybe that even solves the problem.

Unfortunately, I can't try it out now, and I'll be away for the
holidays. I'll get back to test it next year.

Thanks a lot,
best Regards,
Christian Sachs

> Cheers,
> ~Josh
>
>
>> PS: I forgot to mention that the file itself is properly openable in
>> Fiji with current Bio-Formats.
>>
>> Thank you very much,
>> Best regards
>>
>> Christian Sachs
>>
>>>> We're using OMERO 5.2.0 on a Windows server (which has - to my knowledge
>>>> - been upgraded multiple times from earlier versions).
>>>> As of the last three files imported, none was imported properly, we have
>>>> told our users to stop imports till the problem is resolved.
>>>>
>>>> Thank you,
>>>> best regards
>>>>
>>>> Christian Sachs
> _______________________________________________
> ome-users mailing list
> ome-users at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>

------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------