[ome-users] excessive time to upgrade database / cleanse
Olivier Miquel
olivier.miquel at mri.cnrs.fr
Mon May 2 13:04:28 BST 2016
Hi Josh,
You could find the full Blitz logfile here :
https://cloud.mri.cnrs.fr/public.php?service=files&t=5c3bbf76142d4f41358d6b5a976c4f42
Here's what appears at the same time as the script dies :
2016-05-02 11:39:57,396 ERROR [o.s.t.interceptor.TransactionInterceptor]
(Server-112) Application exception overridden by rollback exception
2016-05-02 11:39:57,404 INFO [ org.perf4j.TimingLogger] (Server-112)
start[1462181755632] time[241772] tag[omero.call.exception]
2016-05-02 11:39:57,406 WARN [ ome.services.util.ServiceHandler]
(Server-112) Unknown exception thrown.
2016-05-02 11:39:57,406 INFO [ ome.services.util.ServiceHandler]
(Server-112) Excp:
org.springframework.transaction.TransactionSystemException: Could not
roll back JDBC transaction; nested exception is
org.postgresql.util.PSQLException: This connection has been closed.
2016-05-02 11:39:57,406 ERROR [ ome.services.util.ServiceHandler]
(Server-112) Method interface ome.services.util.Executor$Work.doWork
invocation took 241772
Thanks for your support,
O.
On 02/05/2016 07:50, Josh Moore wrote:
> On Fri, Apr 29, 2016 at 3:30 PM, Olivier Miquel
> <olivier.miquel at mri.cnrs.fr> wrote:
>> Dear Mark,
>>
>> I had tried to cleanse the binary repository, with no success. The script
>> died after several minutes of silence.
> Hi Olivier,
>
> this actually looks like something went side-ways on the server-side.
> Could you send us your the Blitz log from var/log for when this
> happened?
>
> Cheers,
> ~Josh.
>
>
>> Is there an alternate way to achieve that?
>>
>> Log :
>> omero at kansas:~$ /home/omero/apps/OMERO/OMERO.server/bin/omero admin cleanse
>> /home/omero/apps/OMERO/OMERO.data
>> Previous session expired for root on localhost:4064
>> Server: [localhost:4064]
>> Username: [root]
>> Password:
>> Created session 50852b9f-4f40-4dc2-ad14-b900a8527386 (root at localhost:4064).
>> Idle timeout: 10 min. Current group: system
>> Traceback (most recent call last):
>> File "/home/omero/apps/OMERO/OMERO.server/bin/omero", line 125, in
>> <module>
>> rv = omero.cli.argv()
>> File "/home/omero/apps/OMERO/OMERO.server/lib/python/omero/cli.py", line
>> 1438, in argv
>> cli.invoke(args[1:])
>> File "/home/omero/apps/OMERO/OMERO.server/lib/python/omero/cli.py", line
>> 952, in invoke
>> stop = self.onecmd(line, previous_args)
>> File "/home/omero/apps/OMERO/OMERO.server/lib/python/omero/cli.py", line
>> 1029, in onecmd
>> self.execute(line, previous_args)
>> File "/home/omero/apps/OMERO/OMERO.server/lib/python/omero/cli.py", line
>> 1111, in execute
>> args.func(args)
>> File
>> "/home/omero/apps/OMERO/OMERO.server/lib/python/omero/plugins/admin.py",
>> line 1776, in cleanse
>> dry_run=args.dry_run)
>> File
>> "/home/omero/apps/OMERO/OMERO.server/lib/python/omero/util/cleanse.py", line
>> 242, in cleanse
>> delete_empty_dirs(proxy, root, client, dry_run)
>> File
>> "/home/omero/apps/OMERO/OMERO.server/lib/python/omero/util/cleanse.py", line
>> 250, in delete_empty_dirs
>> is_empty_dir(repo, '/', False, to_delete)
>> File
>> "/home/omero/apps/OMERO/OMERO.server/lib/python/omero/util/cleanse.py", line
>> 282, in is_empty_dir
>> is_empty_dir(repo, subdirectory, may_delete_subdir, empty_subdirs):
>> File
>> "/home/omero/apps/OMERO/OMERO.server/lib/python/omero/util/cleanse.py", line
>> 282, in is_empty_dir
>> is_empty_dir(repo, subdirectory, may_delete_subdir, empty_subdirs):
>> File
>> "/home/omero/apps/OMERO/OMERO.server/lib/python/omero/util/cleanse.py", line
>> 282, in is_empty_dir
>> is_empty_dir(repo, subdirectory, may_delete_subdir, empty_subdirs):
>> File
>> "/home/omero/apps/OMERO/OMERO.server/lib/python/omero/util/cleanse.py", line
>> 277, in is_empty_dir
>> for entry in repo.listFiles(directory):
>> File
>> "/home/omero/apps/OMERO/OMERO.server/lib/python/omero_Repositories_ice.py",
>> line 329, in listFiles
>> return _M_omero.grid.Repository._op_listFiles.invoke(self, ((path, ),
>> _ctx))
>> Ice.UnknownException: exception ::Ice::UnknownException
>> {
>> unknown = ome.conditions.InternalException: Wrapped Exception:
>> (org.springframework.transaction.TransactionSystemException):
>> Could not roll back JDBC transaction; nested exception is
>> org.postgresql.util.PSQLException: This connection has been closed.
>> at
>> org.springframework.jdbc.datasource.DataSourceTransactionManager.doRollback(DataSourceTransactionManager.java:286)
>> at
>> org.springframework.transaction.support.AbstractPlatformTransactionManager.processRollback(AbstractPlatformTransactionManager.java:845)
>> at
>> org.springframework.transaction.support.AbstractPlatformTransactionManager.rollback(AbstractPlatformTransactionManager.java:822)
>> at
>> org.springframework.transaction.interceptor.TransactionAspectSupport.completeTransactionAfterThrowing(TransactionAspectSupport.java:430)
>> at
>> org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:112)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>> at
>> ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:249)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>> at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>> at
>> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
>> at com.sun.proxy.$Proxy70.doWork(Unknown Source)
>> at ome.services.util.Executor$Impl.execute(Executor.java:443)
>> at
>> ome.services.blitz.repo.RepositoryDaoImpl.getOriginalFiles(RepositoryDaoImpl.java:439)
>> at
>> ome.services.blitz.repo.PublicRepositoryI.listFiles(PublicRepositoryI.java:263)
>> at
>> omero.grid._ManagedRepositoryTie.listFiles(_ManagedRepositoryTie.java:136)
>> at omero.grid._RepositoryDisp.___listFiles(_RepositoryDisp.java:324)
>> at
>> omero.grid._ManagedRepositoryDisp.__dispatch(_ManagedRepositoryDisp.java:585)
>> at IceInternal.Incoming.invoke(Incoming.java:222)
>> at Ice.ConnectionI.invokeAll(ConnectionI.java:2482)
>> at Ice.ConnectionI.dispatch(ConnectionI.java:1258)
>> at Ice.ConnectionI.message(ConnectionI.java:1213)
>> at IceInternal.ThreadPool.run(ThreadPool.java:321)
>> at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
>> at
>> IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:693)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> }
>>
>>
>> On 08/03/2016 10:19, Mark Carroll wrote:
>>> Dear Olivier,
>>>
>>>> Is there somewhere a script to check and clean the database?
>>>> Our omero.server is in production for about five years now and perhaps
>>>> we now need to clean some unused stuff in the database?
>>>
>>> What I'd been mentioning to Carnë in this thread about periodic VACUUMs
>>> and suchlike probably suffices for PostgreSQL; their documentation is
>>> good on this point. If we notice that our server has wrongly been
>>> leaving some kind of debris in the database then we usually include
>>> cleanup in upgrade scripts so it gets done for our users anyway in time;
>>> for instance, the upgrade that we've been discussing included some
>>> removal of uselessly duplicate rows from a large table.
>>>
>>> However, separately from the database, there is also the binary
>>> repository: it is probably worth glancing at the first part of
>>>
>>> https://www.openmicroscopy.org/site/support/omero/sysadmins/server-backup-and-restore.html
>>> which talks about cleaning files up before doing a backup. If you
>>> haven't run a "bin/omero admin cleanse" for a while then that might be
>>> something that's worth adding to your infrequent maintenance checklist.
>>>
>>> -- Mark
>>>
>>> The University of Dundee is a registered Scottish Charity, No: SC015096
>>> _______________________________________________
>>> ome-users mailing list
>>> ome-users at lists.openmicroscopy.org.uk
>>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
> _______________________________________________
> ome-users mailing list
> ome-users at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
--
Olivier Miquel
Administrateur Système et Réseau
Plate-forme technologique MRI
Tel : 06.50.19.27.43
Mail:olivier.miquel at mri.cnrs.fr
More information about the ome-users
mailing list