<div dir="ltr"><div><div><div><div><div><div><div>Hello Josh,<br><br></div>A little update on this issue:<br><br></div>After bumping kernel.shmmax to 8GB and Postgresql max_locks_per_transaction to 256, I can now do bulk deletion for 910 images in 25 datasets at one time.<br>
<br></div>However, I have got the same error message when trying to delete a lager set of images after import failure. Tried to increase max_locks_per_transaction to 2048 and waiting for deletion to finish, which took couple hours...<br>
<br></div>My system has 8GB total memory, I would think it doesn't make sense to increase kernel.shmmax to beyond 8GB, right?<br><br></div>What I am also interested in knowing the common practice for OMERO memory configuration. What would be recommend memory configuration for OME if we have couple thousand images per project?<br>
<br></div>I'll start a new thread regarding my import errors...<br><br></div>Thanks,<br>Yanling<br></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Jun 5, 2014 at 7:39 PM, Josh Moore <span dir="ltr"><<a href="mailto:josh@glencoesoftware.com" target="_blank">josh@glencoesoftware.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Yanling,<br>
<div class=""><br>
On Jun 4, 2014, at 2:14 PM, Yanling Liu wrote:<br>
<br>
> I am using postgres 9.1. And I got same error message after increasing<br>
> kernel.shmmax to 128MB.<br>
<br>
</div>Sorry that you're still having troubles. Unfortunately, the entire OME team<br>
is currently at the annual users' meeting:<br>
<br>
<a href="http://www.openmicroscopy.org/site/community/minutes/meetings/9th-annual-users-meeting-june-2014" target="_blank">http://www.openmicroscopy.org/site/community/minutes/meetings/9th-annual-users-meeting-june-2014</a><br>
<br>
so our responses will be somewhat slower than usual. Until your PostgreSQL installation<br>
is no longer showing the error, though, comparing deletion times won't be useful.<br>
<br>
Have you restarted PostgreSQL after the change? If that didn't work, your host?<br>
<br>
You might want to read through <a href="https://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server" target="_blank">https://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server</a><br>
to find out more about changing these settings.<br>
<br>
> ...<br>
<div class="">> In case you might wonder why I need to delete files. It is simply because<br>
> there's import error so I need to delete whatever imported into OMERO<br>
> before doing import again.<br>
<br>
</div>Makes complete sense.<br>
<div class=""><br>
<br>
> Not to mention import performance in OMERO is also low...<br>
<br>
</div>Can you quantify this for us? What file format are you trying to import? How many individual files and GB per fileset?<br>
How long did import take? Could you send us the import log?<br>
<br>
Also note that part of the import times will be dependent on the performance of your PostgreSQL installation, so until we've handled the first issue, we may not want to compare import times).<br>
<br>
<br>
> ...<br>
> Thanks,<br>
> Yanling<br>
<br>
<br>
Cheers,<br>
~Josh.<br>
<div class="HOEnZb"><div class="h5"><br>
<br>
> On Tue, Jun 3, 2014 at 6:48 PM, Roger Leigh <<a href="mailto:rleigh@dundee.ac.uk">rleigh@dundee.ac.uk</a>> wrote:<br>
><br>
>> On 03/06/14 19:25, Josh Moore wrote:<br>
>><br>
>>> Hi Yanling,<br>
>>><br>
>>> On Jun 3, 2014, at 7:28 PM, Yanling Liu wrote:<br>
>>><br>
>>> Made mistake in outputting Blitz-0.log file. Here is what I found:<br>
>>>><br>
>>>> 2014-06-03 08:43:19,924 WARN [ o.h.h.ast.exec.MultiTableDeleteExecutor]<br>
>>>> (2-thread-3) unable to drop temporary id table after use [ERROR: current<br>
>>>> transaction is aborted, commands ignored until end of transaction block]<br>
>>>> 2014-06-03 08:43:19,927 ERROR [ ome.services.delete.Deletion]<br>
>>>> (2-thread-3) Failure during DeleteHandle.steps :<br>
>>>> ...<br>
>>>><br>
>>>> From my limited knowledge about java and postgres, this looks like a<br>
>>>> postgres issue. Shall I change max_locks_per_transaction? If so what<br>
>>>> would<br>
>>>> be optimal value?<br>
>>>><br>
>>>> ...<br>
>>><br>
>>> On Tue, Jun 3, 2014 at 9:29 AM, Yanling Liu <<a href="mailto:vrnova@gmail.com">vrnova@gmail.com</a>> wrote:<br>
>>>><br>
>>>>> ...<br>
>>>>> But I do found something in postgres log file:<br>
>>>>><br>
>>>>> 2014-06-03 08:43:19 EDT WARNING: out of shared memory<br>
>>>>> 2014-06-03 08:43:19 EDT ERROR: out of shared memory<br>
>>>>> 2014-06-03 08:43:19 EDT HINT: You might need to increase<br>
>>>>> max_locks_per_transaction.<br>
>>>>><br>
>>>>> ...<br>
>>><br>
>>> In postgresql.conf I have following settings:<br>
>>>>><br>
>>>>> max_connections = 100<br>
>>>>> #max_locks_per_transaction = 64 # min 10<br>
>>>>><br>
>>>>> Mote the max_locks_per_transaction is commented out by default. I am not<br>
>>>>> an expert on postgres and I would appreciate if you may provide some<br>
>>>>> hints<br>
>>>>> on how to set postgres parameters properly for OMERO.<br>
>>>>><br>
>>>><br>
>> <a href="http://www.postgresql.org/docs/9.3/static/runtime-config-locks.html" target="_blank">http://www.postgresql.org/docs/9.3/static/runtime-config-locks.html</a><br>
>> There shouldn't be a need to fiddle with these settings unless you see<br>
>> problems specifically relating to them.<br>
>><br>
>><br>
>> Since you likely still have available resources on your system with<br>
>>> 8 GB, one solution would be to give increase the shared memory<br>
>>> setting of your system. See the "Linux" section of<br>
>>> <a href="http://www.postgresql.org/docs/9.1/static/kernel-resources.html" target="_blank">http://www.postgresql.org/docs/9.1/static/kernel-resources.html</a><br>
>>><br>
>><br>
>> As a caveat, do make sure to check the documentation for the version of<br>
>> PostgreSQL you are using. More recent versions of PostgreSQL (>= 9.3)<br>
>> no longer use SYSV SHM significantly and so these settings might not<br>
>> apply to you. See e.g.<br>
>> <a href="http://www.postgresql.org/docs/9.3/static/kernel-resources.html" target="_blank">http://www.postgresql.org/docs/9.3/static/kernel-resources.html</a> and<br>
>> <a href="http://rhaas.blogspot.co.uk/2012/06/absurd-shared-memory-limits.html" target="_blank">http://rhaas.blogspot.co.uk/2012/06/absurd-shared-memory-limits.html</a>.<br>
>> In the case of PostgreSQL 9.3 and later you can increase shared_buffers<br>
>> as high as you like providing you have the system memory (and swap<br>
>> space) to back it.<br>
>><br>
>><br>
>> Regards,<br>
>> Roger<br>
<br>
</div></div></blockquote></div><br></div>