[ome-users] excessive time to upgrade database / cleanse
Josh Moore
josh at glencoesoftware.com
Tue May 3 07:45:52 BST 2016
Hi Olivier,
On Mon, May 2, 2016 at 2:04 PM, Olivier Miquel
<olivier.miquel at mri.cnrs.fr> wrote:
> Hi Josh,
>
> You could find the full Blitz logfile here :
>
> https://cloud.mri.cnrs.fr/public.php?service=files&t=5c3bbf76142d4f41358d6b5a976c4f42
Thanks.
> Here's what appears at the same time as the script dies :
>
> 2016-05-02 11:39:57,396 ERROR [o.s.t.interceptor.TransactionInterceptor]
> (Server-112) Application exception overridden by rollback exception
> 2016-05-02 11:39:57,404 INFO [ org.perf4j.TimingLogger] (Server-112)
> start[1462181755632] time[241772] tag[omero.call.exception]
> 2016-05-02 11:39:57,406 WARN [ ome.services.util.ServiceHandler]
> (Server-112) Unknown exception thrown.
> 2016-05-02 11:39:57,406 INFO [ ome.services.util.ServiceHandler]
> (Server-112) Excp:
> org.springframework.transaction.TransactionSystemException: Could not roll
> back JDBC transaction; nested exception is
> org.postgresql.util.PSQLException: This connection has been closed.
> 2016-05-02 11:39:57,406 ERROR [ ome.services.util.ServiceHandler]
> (Server-112) Method interface ome.services.util.Executor$Work.doWork
> invocation took 241772
And the start of this method call is:
2016-05-02 11:35:55,632 INFO [
ome.services.util.ServiceHandler] (Server-112) Executor.doWork --
ome.services.blitz.repo.RepositoryDaoImpl.getOriginalFiles[...,
CheckedPath(bellisj_454/2015-10/01)]
2016-05-02 11:35:55,632 INFO [
ome.services.util.ServiceHandler] (Server-112) Args: [null,
InternalSF at 945268165]
Can you tell us: how large is the directory
/OMERO/ManagedRepository/bellisj_454/2015-10/01
?
~Josh.
> Thanks for your support,
>
>
> O.
>
>
> On 02/05/2016 07:50, Josh Moore wrote:
>>
>> On Fri, Apr 29, 2016 at 3:30 PM, Olivier Miquel
>> <olivier.miquel at mri.cnrs.fr> wrote:
>>>
>>> Dear Mark,
>>>
>>> I had tried to cleanse the binary repository, with no success. The script
>>> died after several minutes of silence.
>>
>> Hi Olivier,
>>
>> this actually looks like something went side-ways on the server-side.
>> Could you send us your the Blitz log from var/log for when this
>> happened?
>>
>> Cheers,
>> ~Josh.
>>
>>
>>> Is there an alternate way to achieve that?
>>>
>>> Log :
>>> omero at kansas:~$ /home/omero/apps/OMERO/OMERO.server/bin/omero admin
>>> cleanse
>>> /home/omero/apps/OMERO/OMERO.data
>>> Previous session expired for root on localhost:4064
>>> Server: [localhost:4064]
>>> Username: [root]
>>> Password:
>>> Created session 50852b9f-4f40-4dc2-ad14-b900a8527386
>>> (root at localhost:4064).
>>> Idle timeout: 10 min. Current group: system
>>> Traceback (most recent call last):
>>> File "/home/omero/apps/OMERO/OMERO.server/bin/omero", line 125, in
>>> <module>
>>> rv = omero.cli.argv()
>>> File "/home/omero/apps/OMERO/OMERO.server/lib/python/omero/cli.py",
>>> line
>>> 1438, in argv
>>> cli.invoke(args[1:])
>>> File "/home/omero/apps/OMERO/OMERO.server/lib/python/omero/cli.py",
>>> line
>>> 952, in invoke
>>> stop = self.onecmd(line, previous_args)
>>> File "/home/omero/apps/OMERO/OMERO.server/lib/python/omero/cli.py",
>>> line
>>> 1029, in onecmd
>>> self.execute(line, previous_args)
>>> File "/home/omero/apps/OMERO/OMERO.server/lib/python/omero/cli.py",
>>> line
>>> 1111, in execute
>>> args.func(args)
>>> File
>>> "/home/omero/apps/OMERO/OMERO.server/lib/python/omero/plugins/admin.py",
>>> line 1776, in cleanse
>>> dry_run=args.dry_run)
>>> File
>>> "/home/omero/apps/OMERO/OMERO.server/lib/python/omero/util/cleanse.py",
>>> line
>>> 242, in cleanse
>>> delete_empty_dirs(proxy, root, client, dry_run)
>>> File
>>> "/home/omero/apps/OMERO/OMERO.server/lib/python/omero/util/cleanse.py",
>>> line
>>> 250, in delete_empty_dirs
>>> is_empty_dir(repo, '/', False, to_delete)
>>> File
>>> "/home/omero/apps/OMERO/OMERO.server/lib/python/omero/util/cleanse.py",
>>> line
>>> 282, in is_empty_dir
>>> is_empty_dir(repo, subdirectory, may_delete_subdir, empty_subdirs):
>>> File
>>> "/home/omero/apps/OMERO/OMERO.server/lib/python/omero/util/cleanse.py",
>>> line
>>> 282, in is_empty_dir
>>> is_empty_dir(repo, subdirectory, may_delete_subdir, empty_subdirs):
>>> File
>>> "/home/omero/apps/OMERO/OMERO.server/lib/python/omero/util/cleanse.py",
>>> line
>>> 282, in is_empty_dir
>>> is_empty_dir(repo, subdirectory, may_delete_subdir, empty_subdirs):
>>> File
>>> "/home/omero/apps/OMERO/OMERO.server/lib/python/omero/util/cleanse.py",
>>> line
>>> 277, in is_empty_dir
>>> for entry in repo.listFiles(directory):
>>> File
>>>
>>> "/home/omero/apps/OMERO/OMERO.server/lib/python/omero_Repositories_ice.py",
>>> line 329, in listFiles
>>> return _M_omero.grid.Repository._op_listFiles.invoke(self, ((path,
>>> ),
>>> _ctx))
>>> Ice.UnknownException: exception ::Ice::UnknownException
>>> {
>>> unknown = ome.conditions.InternalException: Wrapped Exception:
>>> (org.springframework.transaction.TransactionSystemException):
>>> Could not roll back JDBC transaction; nested exception is
>>> org.postgresql.util.PSQLException: This connection has been closed.
>>> at
>>>
>>> org.springframework.jdbc.datasource.DataSourceTransactionManager.doRollback(DataSourceTransactionManager.java:286)
>>> at
>>>
>>> org.springframework.transaction.support.AbstractPlatformTransactionManager.processRollback(AbstractPlatformTransactionManager.java:845)
>>> at
>>>
>>> org.springframework.transaction.support.AbstractPlatformTransactionManager.rollback(AbstractPlatformTransactionManager.java:822)
>>> at
>>>
>>> org.springframework.transaction.interceptor.TransactionAspectSupport.completeTransactionAfterThrowing(TransactionAspectSupport.java:430)
>>> at
>>>
>>> org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:112)
>>> at
>>>
>>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>>> at
>>>
>>> ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:249)
>>> at
>>>
>>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>>> at
>>> ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121)
>>> at
>>>
>>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>>> at
>>>
>>> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
>>> at com.sun.proxy.$Proxy70.doWork(Unknown Source)
>>> at ome.services.util.Executor$Impl.execute(Executor.java:443)
>>> at
>>>
>>> ome.services.blitz.repo.RepositoryDaoImpl.getOriginalFiles(RepositoryDaoImpl.java:439)
>>> at
>>>
>>> ome.services.blitz.repo.PublicRepositoryI.listFiles(PublicRepositoryI.java:263)
>>> at
>>>
>>> omero.grid._ManagedRepositoryTie.listFiles(_ManagedRepositoryTie.java:136)
>>> at
>>> omero.grid._RepositoryDisp.___listFiles(_RepositoryDisp.java:324)
>>> at
>>>
>>> omero.grid._ManagedRepositoryDisp.__dispatch(_ManagedRepositoryDisp.java:585)
>>> at IceInternal.Incoming.invoke(Incoming.java:222)
>>> at Ice.ConnectionI.invokeAll(ConnectionI.java:2482)
>>> at Ice.ConnectionI.dispatch(ConnectionI.java:1258)
>>> at Ice.ConnectionI.message(ConnectionI.java:1213)
>>> at IceInternal.ThreadPool.run(ThreadPool.java:321)
>>> at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
>>> at
>>> IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:693)
>>> at java.lang.Thread.run(Thread.java:745)
>>>
>>> }
>>>
>>>
>>> On 08/03/2016 10:19, Mark Carroll wrote:
>>>>
>>>> Dear Olivier,
>>>>
>>>>> Is there somewhere a script to check and clean the database?
>>>>> Our omero.server is in production for about five years now and perhaps
>>>>> we now need to clean some unused stuff in the database?
>>>>
>>>>
>>>> What I'd been mentioning to Carnë in this thread about periodic VACUUMs
>>>> and suchlike probably suffices for PostgreSQL; their documentation is
>>>> good on this point. If we notice that our server has wrongly been
>>>> leaving some kind of debris in the database then we usually include
>>>> cleanup in upgrade scripts so it gets done for our users anyway in time;
>>>> for instance, the upgrade that we've been discussing included some
>>>> removal of uselessly duplicate rows from a large table.
>>>>
>>>> However, separately from the database, there is also the binary
>>>> repository: it is probably worth glancing at the first part of
>>>>
>>>>
>>>> https://www.openmicroscopy.org/site/support/omero/sysadmins/server-backup-and-restore.html
>>>> which talks about cleaning files up before doing a backup. If you
>>>> haven't run a "bin/omero admin cleanse" for a while then that might be
>>>> something that's worth adding to your infrequent maintenance checklist.
>>>>
>>>> -- Mark
>>>>
>>>> The University of Dundee is a registered Scottish Charity, No: SC015096
>>>> _______________________________________________
>>>> ome-users mailing list
>>>> ome-users at lists.openmicroscopy.org.uk
>>>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>>
>> _______________________________________________
>> ome-users mailing list
>> ome-users at lists.openmicroscopy.org.uk
>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>
>
>
> --
> Olivier Miquel
> Administrateur Système et Réseau
> Plate-forme technologique MRI
> Tel : 06.50.19.27.43
> Mail:olivier.miquel at mri.cnrs.fr
>
> _______________________________________________
> ome-users mailing list
> ome-users at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
More information about the ome-users
mailing list