<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
Hi Andrii,
<div class=""><br class="">
</div>
<div class="">thanks for the references.</div>
<div class=""><br class="">
</div>
<div class="">Indeed, the two first EM tickets should have been addressed in</div>
<div class=""><a href="https://github.com/openmicroscopy/bioformats/pull/2100" class="">https://github.com/openmicroscopy/bioformats/pull/2100</a> and will be released</div>
<div class="">as part of Bio-Formats 5.2.0.</div>
<div class=""><br class="">
</div>
<div class="">As per ticket 13057, adding support for stacks in the Gatan format is outside our</div>
<div class="">capacity for 5.2.0. If anyone in the community has the expertise to add this support,</div>
<div class="">contributions are warmly welcome.</div>
<div class=""><br class="">
</div>
<div class="">Best,</div>
<div class="">Sebastien</div>
<div class=""><br class="">
<div>
<blockquote type="cite" class="">
<div class="">On 10 May 2016, at 20:09, Andrii Iudin <<a href="mailto:andrii@ebi.ac.uk" class="">andrii@ebi.ac.uk</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">Dear Josh,<br class="">
<br class="">
The bug fixes that we are expecting are following:<br class="">
[OME] #13058: BUG: .mrc reade<br class="">
[OME] #13059 Spider files (Reader issues)<br class="">
<br class="">
There also was a request for<br class="">
[OME] #13057: Bug: Gatan Reader (Unsupported byte depth)<br class="">
<br class="">
However, if I understand correctly, this has been pushed for later.<br class="">
<br class="">
Best regards,<br class="">
Andrii<br class="">
<br class="">
<br class="">
On 10/05/2016 07:40, Josh Moore wrote:<br class="">
<blockquote type="cite" class="">On Mon, May 9, 2016 at 11:34 PM, Andrii Iudin <<a href="mailto:andrii@ebi.ac.uk" class="">andrii@ebi.ac.uk</a>> wrote:<br class="">
<blockquote type="cite" class="">Dear Josh,<br class="">
</blockquote>
Hi Andrii,<br class="">
<br class="">
<br class="">
<blockquote type="cite" class="">Yes, we plan to do the update to the latest version, we just have been<br class="">
waiting for the release of OMERO that will have Bio-Formats 5.2.0 since it<br class="">
will introduce fixes that are required for us to use OMERO for the EMPIAR<br class="">
images<br class="">
</blockquote>
Sorry if I've forgotten: which fixes are those?<br class="">
<br class="">
<br class="">
<blockquote type="cite" class="">and since we expect this to be a major update for our systems. Please<br class="">
correct me if I am wrong, but Bio-Formats version that is in the latest<br class="">
OMERO is below 5.2.0?<br class="">
</blockquote>
Correct. OMERO 5.2.3 ships with Bio-Formats 5.1.9. A version of OMERO<br class="">
with Bio-Formats 5.2.0 (which is still unreleased) won't be available<br class="">
until much later this year at the earliest.<br class="">
<br class="">
<br class="">
<blockquote type="cite" class="">Best regards,<br class="">
Andrii<br class="">
</blockquote>
Cheers,<br class="">
~Josh<br class="">
<br class="">
<br class="">
<blockquote type="cite" class="">On 09/05/2016 20:15, Josh Moore wrote:<br class="">
<blockquote type="cite" class="">On Fri, May 6, 2016 at 5:14 PM, Andrii Iudin <<a href="mailto:andrii@ebi.ac.uk" class="">andrii@ebi.ac.uk</a>> wrote:<br class="">
<blockquote type="cite" class="">Dear Josh,<br class="">
</blockquote>
Hi Andrii,<br class="">
<br class="">
<blockquote type="cite" class="">The version we are running is 5.1.4-ice35-b55.<br class="">
</blockquote>
Have you considered upgrading recently? 5.2.3 just came out and with<br class="">
it, the 5.2 series will soon be going into maintenance mode, while<br class="">
support for 5.1 will be dropped.<br class="">
<br class="">
With the latest version, I've just attempted importing 700-800 hundred<br class="">
directories each with 6 images using:<br class="">
<br class="">
$ for x in $(seq 1 1000); do /opt/ome0/dist/bin/omero import<br class="">
$(printf "%04d" "$x") ; done<br class="">
<br class="">
So far I've had no exception with 5.2.3. If you'd like me to try with<br class="">
one of your images (assuming they are all similar), feel free to<br class="">
upload it to <a href="http://qa.openmicroscopy.org.uk/" class="">http://qa.openmicroscopy.org.uk/</a><br class="">
<br class="">
Cheers,<br class="">
~Josh<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
<blockquote type="cite" class="">Best regards,<br class="">
Andrii<br class="">
<br class="">
<br class="">
On 06/05/2016 16:11, Josh Moore wrote:<br class="">
<blockquote type="cite" class="">On Fri, May 6, 2016 at 12:49 PM, Andrii Iudin <<a href="mailto:andrii@ebi.ac.uk" class="">andrii@ebi.ac.uk</a>> wrote:<br class="">
<blockquote type="cite" class="">Dear Josh,<br class="">
</blockquote>
Hi Andrii,<br class="">
<br class="">
<br class="">
<blockquote type="cite" class="">I have added a logout after to the script after each import call. This<br class="">
time<br class="">
more than 2000 entries have been imported, however an error happened.<br class="">
Please<br class="">
could you check the attached log? Is this the same issue with NFS or<br class="">
something different? Is it possible that using sessions might help?<br class="">
</blockquote>
It does look like you're still running in the session/service<br class="">
exhaustion as you were seeing earlier. If using a single session<br class="">
doesn't solve the problem, the only other thing I can think to try at<br class="">
this point is a forcible closing of services. What version of OMERO<br class="">
are you using?<br class="">
<br class="">
Cheers,<br class="">
~Josh.<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
<blockquote type="cite" class="">Thank you and best regards,<br class="">
Andrii<br class="">
<br class="">
On 02/05/2016 06:44, Josh Moore wrote:<br class="">
<blockquote type="cite" class="">On Fri, Apr 29, 2016 at 12:41 PM, Andrii Iudin <<a href="mailto:andrii@ebi.ac.uk" class="">andrii@ebi.ac.uk</a>><br class="">
wrote:<br class="">
<blockquote type="cite" class="">Dear Josh,<br class="">
</blockquote>
Hi Andrii,<br class="">
<br class="">
<br class="">
<blockquote type="cite" class="">Thank you for providing the possible solution to our problem. We will<br class="">
test<br class="">
the session usage and get back with the results. Please could you<br class="">
clarify<br class="">
a<br class="">
few things about your propositions?<br class="">
<br class="">
Is it possible to add a wait time somewhere in the code to compensate<br class="">
for<br class="">
the slower NFS locking?<br class="">
</blockquote>
Cconceivably, but considering the state the serve could possibly be in<br class="">
at that point (shutdown, etc) it's difficult to know. One option is to<br class="">
put your /OMERO directory on a non-NFS filesystem and then symlink in<br class="">
individual directories from NFS. Ultimately, though, this points to an<br class="">
issue with the remote fileshare that needs to be looked into.<br class="">
<br class="">
<br class="">
<blockquote type="cite" class="">As far as I can see we do not call<br class="">
bin/omero login<br class="">
</blockquote>
`bin/omero import` calls `login` if no login is present.<br class="">
<br class="">
<br class="">
<blockquote type="cite" class="">explicitly at this moment. Is it an integral part of the import?<br class="">
There<br class="">
is<br class="">
also BlitzGateway.connect() call before the script goes into the loop<br class="">
over<br class="">
all images.<br class="">
</blockquote>
Agreed. There are a couple of different logins in play here which<br class="">
makes it all a bit complicated. One option would be to get everything<br class="">
into the same process with no subprocess calls to `bin/omero import`.<br class="">
<br class="">
<br class="">
<blockquote type="cite" class="">Does this mean then that we should call logout after each import?<br class="">
</blockquote>
That's probably the easiest thing to test. Longer-term, it'd be better<br class="">
to use a session key.<br class="">
<br class="">
<br class="">
<blockquote type="cite" class="">Thank you and best regards,<br class="">
Andrii<br class="">
</blockquote>
Cheers,<br class="">
~Josh.<br class="">
<br class="">
<br class="">
<br class="">
<blockquote type="cite" class="">On 28/04/2016 10:17, Josh Moore wrote:<br class="">
<blockquote type="cite" class="">On Wed, Apr 27, 2016 at 11:40 AM, Andrii Iudin <<a href="mailto:andrii@ebi.ac.uk" class="">andrii@ebi.ac.uk</a>><br class="">
wrote:<br class="">
<blockquote type="cite" class="">Dear Josh,<br class="">
</blockquote>
Hi Andrii,<br class="">
<br class="">
<br class="">
<blockquote type="cite" class="">Thank you for pointing to the documentation on the remote shares.<br class="">
Those<br class="">
.lock files usually appear if we stop the server after one of the<br class="">
"crashes".<br class="">
When stopping and starting the server during its normal functioning<br class="">
they<br class="">
seem to be not created.<br class="">
</blockquote>
It sounds like a race condition. When the server is under pressure,<br class="">
etc., then there's no time for the slower NFS locking implementation<br class="">
to do what it should. This is what makes the remote share not behave<br class="">
as a posix filesystem should. There has been some success with other<br class="">
versions of NFS and lockd tuning.<br class="">
<br class="">
<br class="">
<blockquote type="cite" class="">The run_command definition is following:<br class="">
def run_command(self, command, logFile=None):<br class="">
</blockquote>
Thanks for the definition. I don't see anything off-hand in your<br class="">
code.<br class="">
If there's a keep alive bug in the import code itself, you might<br class="">
trying running a separate process with:<br class="">
<br class="">
bin/omero sessions keepalive<br class="">
<br class="">
You can either do that in a console for testing, or via your Python<br class="">
driver itself. If that fixes the problem, then we can help you<br class="">
integrate that code into your main script without the need for a<br class="">
subprocess. Additionally, the session UUID that is created by that<br class="">
method could be used in all of your import subprocesses which would<br class="">
1)<br class="">
protect the use of the password and 2) lower the overhead on the<br class="">
server.<br class="">
<br class="">
(In fact, now that I think of it, if you don't have a call to<br class="">
`bin/omero logout` anywhere in your code, this may be exactly the<br class="">
problem that you are running into. Each call to `bin/omero login`<br class="">
creates a new session which is kept alive for the default session<br class="">
timeout.)<br class="">
<br class="">
Cheers,<br class="">
~Josh.<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
<blockquote type="cite" class="">Best regards,<br class="">
Andrii<br class="">
<br class="">
<br class="">
On 26/04/2016 21:00, Josh Moore wrote:<br class="">
<blockquote type="cite" class="">Hi Andrii,<br class="">
<br class="">
On Tue, Apr 26, 2016 at 10:56 AM, Andrii Iudin <<a href="mailto:andrii@ebi.ac.uk" class="">andrii@ebi.ac.uk</a>><br class="">
wrote:<br class="">
<blockquote type="cite" class="">Dear Josh,<br class="">
<br class="">
Please find attached the import script. For each EMDB entry it<br class="">
performs<br class="">
an<br class="">
import of six images - three sides and their thumbnails.<br class="">
</blockquote>
Thanks for this. And where's the definition of `run_command`?<br class="">
<br class="">
<br class="">
<blockquote type="cite" class="">To stop OMERO we use "omero web stop" and then "omero admin stop"<br class="">
commands.<br class="">
After this it is necessary to remove<br class="">
var/OMERO.data/.omero/repository/*/.lock files before starting<br class="">
OMERO<br class="">
again.<br class="">
The system is NFS.<br class="">
</blockquote>
I'd assume then that disconnections & the .lock files are<br class="">
unrelated.<br class="">
Please see<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
<a href="https://www.openmicroscopy.org/site/support/omero5.2/sysadmins/unix/server-binary-repository.html#locking-and-remote-shares" class="">https://www.openmicroscopy.org/site/support/omero5.2/sysadmins/unix/server-binary-repository.html#locking-and-remote-shares</a><br class="">
regarding using remote shares.<br class="">
<br class="">
Cheers,<br class="">
~Josh.<br class="">
<br class="">
<br class="">
<br class="">
<blockquote type="cite" class="">Best regards,<br class="">
Andrii<br class="">
<br class="">
<br class="">
On 25/04/2016 16:21, Josh Moore wrote:<br class="">
<blockquote type="cite" class="">On Fri, Apr 22, 2016 at 12:41 PM, Andrii Iudin<br class="">
<andrii@ebi.ac.uk><br class="">
wrote:<br class="">
<blockquote type="cite" class="">Dear OMERO developers,<br class="">
<br class="">
We are experiencing an issue when importing a large number of<br class="">
images<br class="">
in<br class="">
a<br class="">
single consequent go. This usually happens after importing more<br class="">
than<br class="">
a<br class="">
thousand images. Please see below excerpts from the logs.<br class="">
Increasing<br class="">
a<br class="">
time<br class="">
period between each import seemed to helped a bit, however this<br class="">
issue<br class="">
ultimately happened anyway.<br class="">
</blockquote>
Is this script available publicly? It would be useful to see how<br class="">
it's<br class="">
working.<br class="">
<br class="">
<br class="">
<blockquote type="cite" class="">To get OMERO server working after this happens,<br class="">
it is necessary to stop it, remove .lock files and start the<br class="">
server<br class="">
again.<br class="">
It would be much appreciated if you could point out to a<br class="">
possible<br class="">
way<br class="">
to<br class="">
solve this issue.<br class="">
</blockquote>
How did you stop OMERO? Is your file system on NFS or another<br class="">
remote<br class="">
share?<br class="">
<br class="">
<br class="">
<blockquote type="cite" class="">Thank you and with best regards,<br class="">
Andrii<br class="">
</blockquote>
Cheers,<br class="">
~Josh.<br class="">
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
_______________________________________________<br class="">
ome-devel mailing list<br class="">
<a href="mailto:ome-devel@lists.openmicroscopy.org.uk" class="">ome-devel@lists.openmicroscopy.org.uk</a><br class="">
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel<br class="">
</blockquote>
<br class="">
_______________________________________________<br class="">
ome-devel mailing list<br class="">
<a href="mailto:ome-devel@lists.openmicroscopy.org.uk" class="">ome-devel@lists.openmicroscopy.org.uk</a><br class="">
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel<br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
<br>
<span style="font-size:10pt;">The University of Dundee is a registered Scottish Charity, No: SC015096</span>
</body>
</html>