[ome-users] excessive time to upgrade database / cleanse
Olivier Miquel
olivier.miquel at mri.cnrs.fr
Tue May 3 14:05:04 BST 2016
Hi Josh,
It's a 25G directory containing 54 436 files.
root at kansas:/home/omero/apps/OMERO/OMERO.data/ManagedRepository# du -hs
bellisj_454/2015-10/01
25G bellisj_454/2015-10/01
root at kansas:/home/omero/apps/OMERO/OMERO.data/ManagedRepository# find
./bellisj_454/2015-10/01 -type f | wc -l
54436
Thanks,
O.
On 03/05/2016 08:45, Josh Moore wrote:
> Hi Olivier,
>
> On Mon, May 2, 2016 at 2:04 PM, Olivier Miquel
> <olivier.miquel at mri.cnrs.fr> wrote:
>> Hi Josh,
>>
>> You could find the full Blitz logfile here :
>>
>> https://cloud.mri.cnrs.fr/public.php?service=files&t=5c3bbf76142d4f41358d6b5a976c4f42
> Thanks.
>
>
>> Here's what appears at the same time as the script dies :
>>
>> 2016-05-02 11:39:57,396 ERROR [o.s.t.interceptor.TransactionInterceptor]
>> (Server-112) Application exception overridden by rollback exception
>> 2016-05-02 11:39:57,404 INFO [ org.perf4j.TimingLogger] (Server-112)
>> start[1462181755632] time[241772] tag[omero.call.exception]
>> 2016-05-02 11:39:57,406 WARN [ ome.services.util.ServiceHandler]
>> (Server-112) Unknown exception thrown.
>> 2016-05-02 11:39:57,406 INFO [ ome.services.util.ServiceHandler]
>> (Server-112) Excp:
>> org.springframework.transaction.TransactionSystemException: Could not roll
>> back JDBC transaction; nested exception is
>> org.postgresql.util.PSQLException: This connection has been closed.
>> 2016-05-02 11:39:57,406 ERROR [ ome.services.util.ServiceHandler]
>> (Server-112) Method interface ome.services.util.Executor$Work.doWork
>> invocation took 241772
> And the start of this method call is:
>
> 2016-05-02 11:35:55,632 INFO [
> ome.services.util.ServiceHandler] (Server-112) Executor.doWork --
> ome.services.blitz.repo.RepositoryDaoImpl.getOriginalFiles[...,
> CheckedPath(bellisj_454/2015-10/01)]
> 2016-05-02 11:35:55,632 INFO [
> ome.services.util.ServiceHandler] (Server-112) Args: [null,
> InternalSF at 945268165]
>
> Can you tell us: how large is the directory
>
> /OMERO/ManagedRepository/bellisj_454/2015-10/01
>
> ?
>
> ~Josh.
>
>
>> Thanks for your support,
>>
>>
>> O.
>>
>>
>> On 02/05/2016 07:50, Josh Moore wrote:
>>> On Fri, Apr 29, 2016 at 3:30 PM, Olivier Miquel
>>> <olivier.miquel at mri.cnrs.fr> wrote:
>>>> Dear Mark,
>>>>
>>>> I had tried to cleanse the binary repository, with no success. The script
>>>> died after several minutes of silence.
>>> Hi Olivier,
>>>
>>> this actually looks like something went side-ways on the server-side.
>>> Could you send us your the Blitz log from var/log for when this
>>> happened?
>>>
>>> Cheers,
>>> ~Josh.
>>>
>>>
>>>> Is there an alternate way to achieve that?
>>>>
>>>> Log :
>>>> omero at kansas:~$ /home/omero/apps/OMERO/OMERO.server/bin/omero admin
>>>> cleanse
>>>> /home/omero/apps/OMERO/OMERO.data
>>>> Previous session expired for root on localhost:4064
>>>> Server: [localhost:4064]
>>>> Username: [root]
>>>> Password:
>>>> Created session 50852b9f-4f40-4dc2-ad14-b900a8527386
>>>> (root at localhost:4064).
>>>> Idle timeout: 10 min. Current group: system
>>>> Traceback (most recent call last):
>>>> File "/home/omero/apps/OMERO/OMERO.server/bin/omero", line 125, in
>>>> <module>
>>>> rv = omero.cli.argv()
>>>> File "/home/omero/apps/OMERO/OMERO.server/lib/python/omero/cli.py",
>>>> line
>>>> 1438, in argv
>>>> cli.invoke(args[1:])
>>>> File "/home/omero/apps/OMERO/OMERO.server/lib/python/omero/cli.py",
>>>> line
>>>> 952, in invoke
>>>> stop = self.onecmd(line, previous_args)
>>>> File "/home/omero/apps/OMERO/OMERO.server/lib/python/omero/cli.py",
>>>> line
>>>> 1029, in onecmd
>>>> self.execute(line, previous_args)
>>>> File "/home/omero/apps/OMERO/OMERO.server/lib/python/omero/cli.py",
>>>> line
>>>> 1111, in execute
>>>> args.func(args)
>>>> File
>>>> "/home/omero/apps/OMERO/OMERO.server/lib/python/omero/plugins/admin.py",
>>>> line 1776, in cleanse
>>>> dry_run=args.dry_run)
>>>> File
>>>> "/home/omero/apps/OMERO/OMERO.server/lib/python/omero/util/cleanse.py",
>>>> line
>>>> 242, in cleanse
>>>> delete_empty_dirs(proxy, root, client, dry_run)
>>>> File
>>>> "/home/omero/apps/OMERO/OMERO.server/lib/python/omero/util/cleanse.py",
>>>> line
>>>> 250, in delete_empty_dirs
>>>> is_empty_dir(repo, '/', False, to_delete)
>>>> File
>>>> "/home/omero/apps/OMERO/OMERO.server/lib/python/omero/util/cleanse.py",
>>>> line
>>>> 282, in is_empty_dir
>>>> is_empty_dir(repo, subdirectory, may_delete_subdir, empty_subdirs):
>>>> File
>>>> "/home/omero/apps/OMERO/OMERO.server/lib/python/omero/util/cleanse.py",
>>>> line
>>>> 282, in is_empty_dir
>>>> is_empty_dir(repo, subdirectory, may_delete_subdir, empty_subdirs):
>>>> File
>>>> "/home/omero/apps/OMERO/OMERO.server/lib/python/omero/util/cleanse.py",
>>>> line
>>>> 282, in is_empty_dir
>>>> is_empty_dir(repo, subdirectory, may_delete_subdir, empty_subdirs):
>>>> File
>>>> "/home/omero/apps/OMERO/OMERO.server/lib/python/omero/util/cleanse.py",
>>>> line
>>>> 277, in is_empty_dir
>>>> for entry in repo.listFiles(directory):
>>>> File
>>>>
>>>> "/home/omero/apps/OMERO/OMERO.server/lib/python/omero_Repositories_ice.py",
>>>> line 329, in listFiles
>>>> return _M_omero.grid.Repository._op_listFiles.invoke(self, ((path,
>>>> ),
>>>> _ctx))
>>>> Ice.UnknownException: exception ::Ice::UnknownException
>>>> {
>>>> unknown = ome.conditions.InternalException: Wrapped Exception:
>>>> (org.springframework.transaction.TransactionSystemException):
>>>> Could not roll back JDBC transaction; nested exception is
>>>> org.postgresql.util.PSQLException: This connection has been closed.
>>>> at
>>>>
>>>> org.springframework.jdbc.datasource.DataSourceTransactionManager.doRollback(DataSourceTransactionManager.java:286)
>>>> at
>>>>
>>>> org.springframework.transaction.support.AbstractPlatformTransactionManager.processRollback(AbstractPlatformTransactionManager.java:845)
>>>> at
>>>>
>>>> org.springframework.transaction.support.AbstractPlatformTransactionManager.rollback(AbstractPlatformTransactionManager.java:822)
>>>> at
>>>>
>>>> org.springframework.transaction.interceptor.TransactionAspectSupport.completeTransactionAfterThrowing(TransactionAspectSupport.java:430)
>>>> at
>>>>
>>>> org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:112)
>>>> at
>>>>
>>>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>>>> at
>>>>
>>>> ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:249)
>>>> at
>>>>
>>>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>>>> at
>>>> ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121)
>>>> at
>>>>
>>>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>>>> at
>>>>
>>>> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
>>>> at com.sun.proxy.$Proxy70.doWork(Unknown Source)
>>>> at ome.services.util.Executor$Impl.execute(Executor.java:443)
>>>> at
>>>>
>>>> ome.services.blitz.repo.RepositoryDaoImpl.getOriginalFiles(RepositoryDaoImpl.java:439)
>>>> at
>>>>
>>>> ome.services.blitz.repo.PublicRepositoryI.listFiles(PublicRepositoryI.java:263)
>>>> at
>>>>
>>>> omero.grid._ManagedRepositoryTie.listFiles(_ManagedRepositoryTie.java:136)
>>>> at
>>>> omero.grid._RepositoryDisp.___listFiles(_RepositoryDisp.java:324)
>>>> at
>>>>
>>>> omero.grid._ManagedRepositoryDisp.__dispatch(_ManagedRepositoryDisp.java:585)
>>>> at IceInternal.Incoming.invoke(Incoming.java:222)
>>>> at Ice.ConnectionI.invokeAll(ConnectionI.java:2482)
>>>> at Ice.ConnectionI.dispatch(ConnectionI.java:1258)
>>>> at Ice.ConnectionI.message(ConnectionI.java:1213)
>>>> at IceInternal.ThreadPool.run(ThreadPool.java:321)
>>>> at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
>>>> at
>>>> IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:693)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>>
>>>> }
>>>>
>>>>
>>>> On 08/03/2016 10:19, Mark Carroll wrote:
>>>>> Dear Olivier,
>>>>>
>>>>>> Is there somewhere a script to check and clean the database?
>>>>>> Our omero.server is in production for about five years now and perhaps
>>>>>> we now need to clean some unused stuff in the database?
>>>>>
>>>>> What I'd been mentioning to Carnë in this thread about periodic VACUUMs
>>>>> and suchlike probably suffices for PostgreSQL; their documentation is
>>>>> good on this point. If we notice that our server has wrongly been
>>>>> leaving some kind of debris in the database then we usually include
>>>>> cleanup in upgrade scripts so it gets done for our users anyway in time;
>>>>> for instance, the upgrade that we've been discussing included some
>>>>> removal of uselessly duplicate rows from a large table.
>>>>>
>>>>> However, separately from the database, there is also the binary
>>>>> repository: it is probably worth glancing at the first part of
>>>>>
>>>>>
>>>>> https://www.openmicroscopy.org/site/support/omero/sysadmins/server-backup-and-restore.html
>>>>> which talks about cleaning files up before doing a backup. If you
>>>>> haven't run a "bin/omero admin cleanse" for a while then that might be
>>>>> something that's worth adding to your infrequent maintenance checklist.
>>>>>
>>>>> -- Mark
>>>>>
>>>>> The University of Dundee is a registered Scottish Charity, No: SC015096
>>>>> _______________________________________________
>>>>> ome-users mailing list
>>>>> ome-users at lists.openmicroscopy.org.uk
>>>>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>>> _______________________________________________
>>> ome-users mailing list
>>> ome-users at lists.openmicroscopy.org.uk
>>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>>
>>
>> --
>> Olivier Miquel
>> Administrateur Système et Réseau
>> Plate-forme technologique MRI
>> Tel : 06.50.19.27.43
>> Mail:olivier.miquel at mri.cnrs.fr
>>
>> _______________________________________________
>> ome-users mailing list
>> ome-users at lists.openmicroscopy.org.uk
>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
> _______________________________________________
> ome-users mailing list
> ome-users at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
--
Olivier Miquel
Administrateur Système et Réseau
Plate-forme technologique MRI
Tel : 06.50.19.27.43
Mail:olivier.miquel at mri.cnrs.fr
More information about the ome-users
mailing list