[ome-devel] Problem when importing >1000 images

[Josh Moore]

On Wed, Apr 27, 2016 at 11:40 AM, Andrii Iudin <andrii at ebi.ac.uk> wrote:
> Dear Josh,

Hi Andrii,


> Thank you for pointing to the documentation on the remote shares. Those
> .lock files usually appear if we stop the server after one of the "crashes".
> When stopping and starting the server during its normal functioning they
> seem to be not created.

It sounds like a race condition. When the server is under pressure,
etc., then there's no time for the slower NFS locking implementation
to do what it should. This is what makes the remote share not behave
as a posix filesystem should. There has been some success with other
versions of NFS and lockd tuning.


> The run_command definition is following:
>     def run_command(self, command, logFile=None):

Thanks for the definition. I don't see anything off-hand in your code.
If there's a keep alive bug in the import code itself, you might
trying running a separate process with:

    bin/omero sessions keepalive

You can either do that in a console for testing, or via your Python
driver itself. If that fixes the problem, then we can help you
integrate that code into your main script without the need for a
subprocess. Additionally, the session UUID that is created by that
method could be used in all of your import subprocesses which would 1)
protect the use of the password and 2) lower the overhead on the
server.

(In fact, now that I think of it, if you don't have a call to
`bin/omero logout` anywhere in your code, this may be exactly the
problem that you are running into. Each call to `bin/omero login`
creates a new session which is kept alive for the default session
timeout.)

Cheers,
~Josh.




> Best regards,
> Andrii
>
>
> On 26/04/2016 21:00, Josh Moore wrote:
>>
>> Hi Andrii,
>>
>> On Tue, Apr 26, 2016 at 10:56 AM, Andrii Iudin <andrii at ebi.ac.uk> wrote:
>>>
>>> Dear Josh,
>>>
>>> Please find attached the import script. For each EMDB entry it performs
>>> an
>>> import of six images - three sides and their thumbnails.
>>
>> Thanks for this. And where's the definition of `run_command`?
>>
>>
>>> To stop OMERO we use "omero web stop" and then "omero admin stop"
>>> commands.
>>> After this it is necessary to remove
>>> var/OMERO.data/.omero/repository/*/.lock files before starting OMERO
>>> again.
>>> The system is NFS.
>>
>> I'd assume then that disconnections & the .lock files are unrelated.
>> Please see
>> https://www.openmicroscopy.org/site/support/omero5.2/sysadmins/unix/server-binary-repository.html#locking-and-remote-shares
>> regarding using remote shares.
>>
>> Cheers,
>> ~Josh.
>>
>>
>>
>>> Best regards,
>>> Andrii
>>>
>>>
>>> On 25/04/2016 16:21, Josh Moore wrote:
>>>>
>>>> On Fri, Apr 22, 2016 at 12:41 PM, Andrii Iudin <andrii at ebi.ac.uk> wrote:
>>>>>
>>>>> Dear OMERO developers,
>>>>>
>>>>> We are experiencing an issue when importing a large number of images in
>>>>> a
>>>>> single consequent go. This usually happens after importing more than a
>>>>> thousand images. Please see below excerpts from the logs. Increasing a
>>>>> time
>>>>> period between each import seemed to helped a bit, however this issue
>>>>> ultimately happened anyway.
>>>>
>>>> Is this script available publicly? It would be useful to see how it's
>>>> working.
>>>>
>>>>
>>>>> To get OMERO server working after this happens,
>>>>> it is necessary to stop it, remove .lock files and start the server
>>>>> again.
>>>>> It would be much appreciated if you could point out to a possible way
>>>>> to
>>>>> solve this issue.
>>>>
>>>> How did you stop OMERO? Is your file system on NFS or another remote
>>>> share?
>>>>
>>>>
>>>>> Thank you and with best regards,
>>>>> Andrii
>>>>
>>>> Cheers,
>>>> ~Josh.