[ome-devel] Problem when importing >1000 images

Tue May 10 20:50:42 BST 2016

Hi Andrii,

thanks for the references.

Indeed, the two first EM tickets should have been addressed in
https://github.com/openmicroscopy/bioformats/pull/2100 and will be released
as part of Bio-Formats 5.2.0.

As per ticket 13057, adding support for stacks in the Gatan format is outside our
capacity for 5.2.0. If anyone in the community has the expertise to add this support,
contributions are warmly welcome.

Best,
Sebastien

On 10 May 2016, at 20:09, Andrii Iudin <andrii at ebi.ac.uk<mailto:andrii at ebi.ac.uk>> wrote:

Dear Josh,

The bug fixes that we are expecting are following:
[OME] #13058: BUG: .mrc reade
[OME] #13059 Spider files (Reader issues)

There also was a request for
[OME] #13057: Bug: Gatan Reader (Unsupported byte depth)

However, if I understand correctly, this has been pushed for later.

Best regards,
Andrii

On 10/05/2016 07:40, Josh Moore wrote:
On Mon, May 9, 2016 at 11:34 PM, Andrii Iudin <andrii at ebi.ac.uk<mailto:andrii at ebi.ac.uk>> wrote:
Dear Josh,
Hi Andrii,

Yes, we plan to do the update to the latest version, we just have been
waiting for the release of OMERO that will have Bio-Formats 5.2.0 since it
will introduce fixes that are required for us to use OMERO for the EMPIAR
images
Sorry if I've forgotten: which fixes are those?

and since we expect this to be a major update for our systems. Please
correct me if I am wrong, but Bio-Formats version that is in the latest
OMERO is below 5.2.0?
Correct. OMERO 5.2.3 ships with Bio-Formats 5.1.9. A version of OMERO
with Bio-Formats 5.2.0 (which is still unreleased) won't be available
until much later this year at the earliest.

Best regards,
Andrii
Cheers,
~Josh

On 09/05/2016 20:15, Josh Moore wrote:
On Fri, May 6, 2016 at 5:14 PM, Andrii Iudin <andrii at ebi.ac.uk<mailto:andrii at ebi.ac.uk>> wrote:
Dear Josh,
Hi Andrii,

The version we are running is 5.1.4-ice35-b55.
Have you considered upgrading recently? 5.2.3 just came out and with
it, the 5.2 series will soon be going into maintenance mode, while
support for 5.1 will be dropped.

With the latest version, I've just attempted importing 700-800 hundred
directories each with 6 images using:

   $ for x in $(seq 1 1000); do /opt/ome0/dist/bin/omero import
$(printf "%04d" "$x") ; done

So far I've had no exception with 5.2.3. If you'd like me to try with
one of your images (assuming they are all similar), feel free to
upload it to http://qa.openmicroscopy.org.uk/

Cheers,
~Josh

Best regards,
Andrii

On 06/05/2016 16:11, Josh Moore wrote:
On Fri, May 6, 2016 at 12:49 PM, Andrii Iudin <andrii at ebi.ac.uk<mailto:andrii at ebi.ac.uk>> wrote:
Dear Josh,
Hi Andrii,

I have added a logout after to the script after each import call. This
time
more than 2000 entries have been imported, however an error happened.
Please
could you check the attached log? Is this the same issue with NFS or
something different? Is it possible that using sessions might help?
It does look like you're still running in the session/service
exhaustion as you were seeing earlier.  If using a single session
doesn't solve the problem, the only other thing I can think to try at
this point is a forcible closing of services. What version of OMERO
are you using?

Cheers,
~Josh.

Thank you and best regards,
Andrii

On 02/05/2016 06:44, Josh Moore wrote:
On Fri, Apr 29, 2016 at 12:41 PM, Andrii Iudin <andrii at ebi.ac.uk<mailto:andrii at ebi.ac.uk>>
wrote:
Dear Josh,
Hi Andrii,

Thank you for providing the possible solution to our problem. We will
test
the session usage and get back with the results. Please could you
clarify
a
few things about your propositions?

Is it possible to add a wait time somewhere in the code to compensate
for
the slower NFS locking?
Cconceivably, but considering the state the serve could possibly be in
at that point (shutdown, etc) it's difficult to know. One option is to
put your /OMERO directory on a non-NFS filesystem and then symlink in
individual directories from NFS. Ultimately, though, this points to an
issue with the remote fileshare that needs to be looked into.

As far as I can see we do not call
bin/omero login
`bin/omero import` calls `login` if no login is present.

explicitly at this moment. Is it an integral part of the import?
There
is
also BlitzGateway.connect() call before the script goes into the loop
over
all images.
Agreed. There are a couple of different logins in play here which
makes it all a bit complicated. One option would be to get everything
into the same process with no subprocess calls to `bin/omero import`.

Does this mean then that we should call logout after each import?
That's probably the easiest thing to test. Longer-term, it'd be better
to use a session key.

Thank you and best regards,
Andrii
Cheers,
~Josh.

On 28/04/2016 10:17, Josh Moore wrote:
On Wed, Apr 27, 2016 at 11:40 AM, Andrii Iudin <andrii at ebi.ac.uk<mailto:andrii at ebi.ac.uk>>
wrote:
Dear Josh,
Hi Andrii,

Thank you for pointing to the documentation on the remote shares.
Those
.lock files usually appear if we stop the server after one of the
"crashes".
When stopping and starting the server during its normal functioning
they
seem to be not created.
It sounds like a race condition. When the server is under pressure,
etc., then there's no time for the slower NFS locking implementation
to do what it should. This is what makes the remote share not behave
as a posix filesystem should. There has been some success with other
versions of NFS and lockd tuning.

The run_command definition is following:
        def run_command(self, command, logFile=None):
Thanks for the definition. I don't see anything off-hand in your
code.
If there's a keep alive bug in the import code itself, you might
trying running a separate process with:

        bin/omero sessions keepalive

You can either do that in a console for testing, or via your Python
driver itself. If that fixes the problem, then we can help you
integrate that code into your main script without the need for a
subprocess. Additionally, the session UUID that is created by that
method could be used in all of your import subprocesses which would
1)
protect the use of the password and 2) lower the overhead on the
server.

(In fact, now that I think of it, if you don't have a call to
`bin/omero logout` anywhere in your code, this may be exactly the
problem that you are running into. Each call to `bin/omero login`
creates a new session which is kept alive for the default session
timeout.)

Cheers,
~Josh.

Best regards,
Andrii

On 26/04/2016 21:00, Josh Moore wrote:
Hi Andrii,

On Tue, Apr 26, 2016 at 10:56 AM, Andrii Iudin <andrii at ebi.ac.uk<mailto:andrii at ebi.ac.uk>>
wrote:
Dear Josh,

Please find attached the import script. For each EMDB entry it
performs
an
import of six images - three sides and their thumbnails.
Thanks for this. And where's the definition of `run_command`?

To stop OMERO we use "omero web stop" and then "omero admin stop"
commands.
After this it is necessary to remove
var/OMERO.data/.omero/repository/*/.lock files before starting
OMERO
again.
The system is NFS.
I'd assume then that disconnections & the .lock files are
unrelated.
Please see

https://www.openmicroscopy.org/site/support/omero5.2/sysadmins/unix/server-binary-repository.html#locking-and-remote-shares
regarding using remote shares.

Cheers,
~Josh.

Best regards,
Andrii

On 25/04/2016 16:21, Josh Moore wrote:
On Fri, Apr 22, 2016 at 12:41 PM, Andrii Iudin
<andrii at ebi.ac.uk>
wrote:
Dear OMERO developers,

We are experiencing an issue when importing a large number of
images
in
a
single consequent go. This usually happens after importing more
than
a
thousand images. Please see below excerpts from the logs.
Increasing
a
time
period between each import seemed to helped a bit, however this
issue
ultimately happened anyway.
Is this script available publicly? It would be useful to see how
it's
working.

To get OMERO server working after this happens,
it is necessary to stop it, remove .lock files and start the
server
again.
It would be much appreciated if you could point out to a
possible
way
to
solve this issue.
How did you stop OMERO? Is your file system on NFS or another
remote
share?

Thank you and with best regards,
Andrii
Cheers,
~Josh.
_______________________________________________
ome-devel mailing list
ome-devel at lists.openmicroscopy.org.uk<mailto:ome-devel at lists.openmicroscopy.org.uk>
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel

_______________________________________________
ome-devel mailing list
ome-devel at lists.openmicroscopy.org.uk<mailto:ome-devel at lists.openmicroscopy.org.uk>
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel

The University of Dundee is a registered Scottish Charity, No: SC015096
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-devel/attachments/20160510/7659ca19/attachment.html>