[ome-devel] Trouble with OMERO.Dropbox

[Niko Ehrenfeuchter]

Hi Colin,

On 05.02.2015 11:28, Colin Blackburn wrote:
> Hi Niko,
>
>> I finally managed to do so, log files were uploaded in QA ticket
>> 10456.
>
> Thanks.
>
>> The relevant part in those logs is from January 29th, copying a
>> large Leica LIF file (~32 GB), probably containing a multi-position
>> timelapse. I can ask for more details if this is important (or
>> provide the file).
>
> The same effect is occurring with the two smaller LSM files following
> this LIF file copy, though with a smaller impact on the logging. The
> underlying events are being produced by the inotify system so if this
> is a new effect might something have changed with your OS/filesystem?
> I assume from the logs that the imports are then proceeding as
> normal?

This is not new, there were no changes done to the system. It is just 
that smaller files have a much less serious impact on the system.

Although I have to say that we've previously seen it stall for pretty 
long periods (30-120 minutes) when many (~15-20) people copied several 
smaller files to the DropBox at the same time.

Most of the files get imported correctly at the end, but I wouldn't 
place a bet on the very large ones. However, we've seen system load 
going up to more than 50 (!), rendering the whole machine basically dead.

> One thing we could certainly fix at our end is to reduce much of this
> logging to DEBUG level, though I don’t know how much impact that
> would have on cpu usage. But if you were happy to patch your code I
> could make this available for testing.

Sure, we'd love to try this. In the current situation, DropBox is 
entirely unusable for our scenario :-(

> It is also possible, again via a change in our codebase, to not
> actively ask for modify events, however if a create event is missed
> for a file (notification systems aren’t perfect) then first modify
> event for a file serves in lieu of a create. This would be a more
> significant change that would require more extensive testing from
> us.

Is there no way to configure the behaviour of the inotify subsystem to 
trigger (or pass on) modification with a lower frequency, e.g. only once 
per second instead of once per millisecond?

Cheers,
~Niko