[ome-devel] OMERO 5 deletion performance

[Josh Moore]

Hi Yanling,

On Jun 3, 2014, at 7:28 PM, Yanling Liu wrote:

> Made mistake in outputting Blitz-0.log file. Here is what I found:
> 
> 2014-06-03 08:43:19,924 WARN  [ o.h.h.ast.exec.MultiTableDeleteExecutor]
> (2-thread-3) unable to drop temporary id table after use [ERROR: current
> transaction is aborted, commands ignored until end of transaction block]
> 2014-06-03 08:43:19,927 ERROR [            ome.services.delete.Deletion]
> (2-thread-3) Failure during DeleteHandle.steps :
> ...
> 
> From my limited knowledge about java and postgres, this looks like a
> postgres issue. Shall I change max_locks_per_transaction? If so what would
> be optimal value?
> 
...

> On Tue, Jun 3, 2014 at 9:29 AM, Yanling Liu <vrnova at gmail.com> wrote:
>> ...
>> But I do found something in postgres log file:
>> 
>> 2014-06-03 08:43:19 EDT WARNING:  out of shared memory
>> 2014-06-03 08:43:19 EDT ERROR:  out of shared memory
>> 2014-06-03 08:43:19 EDT HINT:  You might need to increase
>> max_locks_per_transaction.
>> 
...

>> In postgresql.conf I have following settings:
>> 
>> max_connections = 100
>> #max_locks_per_transaction = 64         # min 10
>> 
>> Mote the max_locks_per_transaction is commented out by default. I am not
>> an expert on postgres and I would appreciate if you may provide some hints
>> on how to set postgres parameters properly for OMERO.

Since you likely still have available resources on your system with
8 GB, one solution would be to give increase the shared memory
setting of your system. See the "Linux" section of
http://www.postgresql.org/docs/9.1/static/kernel-resources.html

Cheers,
~Josh.




>> On Tue, Jun 3, 2014 at 6:28 AM, Josh Moore <josh at glencoesoftware.com>
>> wrote:
>> 
>>> Hi Yanling,
>>> 
>>>>> On Mon, Jun 2, 2014 at 10:02 AM, Yanling Liu <vrnova at gmail.com> wrote:
>>>>> 
>>>>>> I have OMERO 5 installed on Ubuntu 12.04 64bit 8GB Mem, 4 Cores @
>>> 3.47GHz
>>>>>> and NFS mounted storage for OMERO data storage.
>>>>>> 
>>>>>> When deleting couple hundred images (each at 1k x 1k pixels) through
>>>>>> OMERO web, it took long time (> 10 minutes) to finish. In general
>>> project
>>>>>> and dataset deletion are very slow in OMERO web. I wonder if there's
>>> any
>>>>>> hint to tune the performance of OMERO.
>>>>>> 
>>>>>> I have increased JVM memory usage based on OMERO documentation.
>>> 
>>> Though there are further tunings you can try, Deletion performance is
>>> quite poor at the moment, but it's something we're actively working on.
>>> 
>>> 
>>>> On Mon, Jun 2, 2014 at 11:09 AM, Yanling Liu <vrnova at gmail.com> wrote:
>>>> 
>>>>> By looking OMEROweb.log, deleting single file takes 5 good seconds. Is
>>>>> this normal and expected? How could I find out if this because OMERO
>>>>> itself, or postgres, or NFS storage access speed?
>>>>> 
>>>>> Thanks ahead.....
>>> 
>>> Could you send us the var/log/Blitz-0.log file for the time of the
>>> deletion?
>>> 
>>> 
>>> 
>>> On Jun 2, 2014, at 5:12 PM, Yanling Liu wrote:
>>> 
>>>> Also I am getting this error:
>>>> 
>>>> 2014-06-02 16:07:37,242 ERROR [      omeroweb.webclient.webclient_utils]
>>>> (proc.20168) _formatReport:38 Format report: {'warning': '', 'error':
>>>> 'Failure during DeleteHandle.steps
>>>> :org.hibernate.exception.GenericJDBCException: could not insert/select
>>> ids
>>>> for bulk delete'}
>>>> 
>>>> Anything wrong with postgres? Or is it because I selected too many files
>>>> for bulk deletion?
>>> 
>>> Possibly. Again, the log file would be the most helpful. There may be an
>>> issue with your PG installation.
>>> 
>>> Cheers,
>>> ~Josh.
>> 
>> 
>>