[ome-devel] Trouble with OMERO.Dropbox

[Colin Blackburn]

Hi Niko,

> Sure, we'd love to try this. In the current situation, DropBox is entirely unusable for our scenario :-(

I’ll push a commit as soon as I can and let you know. I assume your server is built using our 5.0.x line?

> Is there no way to configure the behaviour of the inotify subsystem to trigger (or pass on) modification with a lower frequency, e.g. only once per second instead of once per millisecond?

No, not as far as I can see from the python library we us, pyinotify. The latest version of this library, later than the one one we bundle, allows events to be ignored but they are still propagated from inotify to pyinotify but just passed no further.

Cheers,

Colin


> On 5 Feb 2015, at 10:45, Niko Ehrenfeuchter <nikolaus.ehrenfeuchter at unibas.ch> wrote:
>
> Hi Colin,
>
> On 05.02.2015 11:28, Colin Blackburn wrote:
>> Hi Niko,
>>
>>> I finally managed to do so, log files were uploaded in QA ticket
>>> 10456.
>>
>> Thanks.
>>
>>> The relevant part in those logs is from January 29th, copying a
>>> large Leica LIF file (~32 GB), probably containing a multi-position
>>> timelapse. I can ask for more details if this is important (or
>>> provide the file).
>>
>> The same effect is occurring with the two smaller LSM files following
>> this LIF file copy, though with a smaller impact on the logging. The
>> underlying events are being produced by the inotify system so if this
>> is a new effect might something have changed with your OS/filesystem?
>> I assume from the logs that the imports are then proceeding as
>> normal?
>
> This is not new, there were no changes done to the system. It is just that smaller files have a much less serious impact on the system.
>
> Although I have to say that we've previously seen it stall for pretty long periods (30-120 minutes) when many (~15-20) people copied several smaller files to the DropBox at the same time.
>
> Most of the files get imported correctly at the end, but I wouldn't place a bet on the very large ones. However, we've seen system load going up to more than 50 (!), rendering the whole machine basically dead.
>
>> One thing we could certainly fix at our end is to reduce much of this
>> logging to DEBUG level, though I don’t know how much impact that
>> would have on cpu usage. But if you were happy to patch your code I
>> could make this available for testing.
>
> Sure, we'd love to try this. In the current situation, DropBox is entirely unusable for our scenario :-(
>
>> It is also possible, again via a change in our codebase, to not
>> actively ask for modify events, however if a create event is missed
>> for a file (notification systems aren’t perfect) then first modify
>> event for a file serves in lieu of a create. This would be a more
>> significant change that would require more extensive testing from
>> us.
>
> Is there no way to configure the behaviour of the inotify subsystem to trigger (or pass on) modification with a lower frequency, e.g. only once per second instead of once per millisecond?
>
> Cheers,
> ~Niko
> _______________________________________________
> ome-devel mailing list
> ome-devel at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel


The University of Dundee is a registered Scottish Charity, No: SC015096