[ome-users] Problems with OMERO Imports

Thu Dec 17 17:26:32 GMT 2015

Hi Josh,

On 12/17/2015 05:25 PM, Josh Moore wrote:
> Hi Christian,
>
> On Thu, Dec 17, 2015 at 5:10 PM, Christian Carsten Sachs
> <c.sachs at fz-juelich.de> wrote:
>> Hello,
>>
>> yesterday, we have just started with OMERO production use at our
>> institute; and while it had worked well in our preliminary tests, we
>> have now seen a strange issue with importing images.
>>
>> Nikon ND2 image stacks (.nd2 files) are imported using the command line
>> client, in ln_s mode (called by a custom OMERO.dropbox-like script,
>> enforcing some additional naming rules etc.).
>
> Is the script available somewhere?
>
>

No, and we cannot easily upload is somewhere as it contains confidential
info ... but I'll shortly summarize what it does.

We have decided to enforce certain folder structures / naming
conventions, have windows ACLs changed automatically, etc. Once a user
moves an experiment folder to the (script's) dropbox location ...

with our call to OMERO client being
omero_call = "omero --server localhost --port 4064 --sudo importer
--password <password> --user <targetuser>"

first a new dataset is created

omero_call obj new Dataset name="<dataset name>"

resulting in datasetId

Then, for any other (non image) files in the folder, they are being
added as file attachments / or text comments depending von file type.
Therefore, we have another, custom script using OMERO's scripting API:

omero_call script run path\Add_Annotation.py IDs=$datasetId
Annotation_Type=FileAnnotation "File_Source=path/to/file" ...

Actual file import happens by the following call (if there is more than
one image file in the folder, they all will be added to the same dataset).

omero_call import "path/to/image/file.nd2"  -- --auto_close
--minutes_wait=0 --checksum-algorithm=File-Size-64 --transfer=ln_s -d
$datasetId

(Btw, having more complex import possibilities with OMERO.dropbox would
be great, should we investigate creating a patch to OMERO.dropbox to
enhance it with said functionality?)

>> Import proceeds like normal, and all positions are added to OMERO, but
>> the actual image data remains inaccessible. (Only the clock icon in
>> thumbnail view in the webclient).
>>
>> In Blitz-0 (or similarily the individual file log file), the following
>> errors are visible:
>>
>> (repeated for each plane)
>> 2015-12-17 16:07:10,458 ERROR [ ome.formats.OMEROMetadataStoreClient]
>> (2-thread-4) Server error setting extended properties for Pixels:8145
>> Target file:sachs_102/2015-12/17/15-07-53.074/nd009.nd2
>
> Is there any stack trace above this line? Perhaps send the whole log (dir)?
>
>

I did not see any stack trace, this is why I'm rather puzzled ...
The whole logfile is some hundred megabytes large, I've split off the
last import process (actually, since the last restart, as I had asked
our admin to restart the system right before attempting the last import).

As it's still too large to comfortably add it as attachment, please see
following (temporary) link:

https://fz-juelich.sciebo.de/index.php/s/zeltfZ4hfrJUMLU

>> 2015-12-17 16:07:10,536 INFO  [ o.s.b.r.ManagedImportRequestI. at 49ccbb67]
>> (2-thread-4) Cancelled
>
>   I'd guess the above exception is leading to the import being cancelled.
>
>
>> 2015-12-17 16:07:10,552 DEBUG [ o.s.b.r.ManagedImportRequestI. at 49ccbb67]
>> ome.services.blitz.repo.ManagedImportRequestI.cleanup(ManagedImportRequestI.java:362)
>
> This is all secondary after the import has died (or been killed).
>
>
>> I will start trying to reproduce this with another test installation;
>> but maybe a dev could quickly point me into the right direction what the
>> problem might be?
>
> The fact that this is happening for each plane makes me think that
> somehow the fileset detection on the production system differs from
> the test system. Could you use `bin\omero import -f YOURDATAHERE` on
> both the test and the production system and send us the logs?
>
> Thanks,
> ~Josh.
>
>
My wording might've been misunderstandable: It has worked in both the
testing and production system; the production system was tested with the
import script and some datasets in late November and everything worked fine.
Yesterday we wanted to 'go live' and let users start to import data ...
Thus we're a bit puzzled that problems occur now ...

PS: I forgot to mention that the file itself is properly openable in
Fiji with current Bio-Formats.

Thank you very much,
Best regards

Christian Sachs

>> We're using OMERO 5.2.0 on a Windows server (which has - to my knowledge
>> - been upgraded multiple times from earlier versions).
>> As of the last three files imported, none was imported properly, we have
>> told our users to stop imports till the problem is resolved.
>>
>> Thank you,
>> best regards
>>
>> Christian Sachs
> _______________________________________________
> ome-users mailing list
> ome-users at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>

------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------