[ome-users] excessive time to upgrade database / cleanse

Josh Moore josh at glencoesoftware.com
Tue May 3 07:45:52 BST 2016


Hi Olivier,

On Mon, May 2, 2016 at 2:04 PM, Olivier Miquel
<olivier.miquel at mri.cnrs.fr> wrote:
> Hi Josh,
>
> You could find the full Blitz logfile here :
>
> https://cloud.mri.cnrs.fr/public.php?service=files&t=5c3bbf76142d4f41358d6b5a976c4f42

Thanks.


> Here's what appears at the same time as the script dies  :
>
> 2016-05-02 11:39:57,396 ERROR [o.s.t.interceptor.TransactionInterceptor]
> (Server-112) Application exception overridden by rollback exception
> 2016-05-02 11:39:57,404 INFO  [ org.perf4j.TimingLogger] (Server-112)
> start[1462181755632] time[241772] tag[omero.call.exception]
> 2016-05-02 11:39:57,406 WARN  [ ome.services.util.ServiceHandler]
> (Server-112) Unknown exception thrown.
> 2016-05-02 11:39:57,406 INFO  [ ome.services.util.ServiceHandler]
> (Server-112)  Excp:
> org.springframework.transaction.TransactionSystemException: Could not roll
> back JDBC transaction; nested exception is
> org.postgresql.util.PSQLException: This connection has been closed.
> 2016-05-02 11:39:57,406 ERROR [ ome.services.util.ServiceHandler]
> (Server-112) Method interface ome.services.util.Executor$Work.doWork
> invocation took 241772

And the start of this method call is:

2016-05-02 11:35:55,632 INFO  [
ome.services.util.ServiceHandler] (Server-112)  Executor.doWork --
ome.services.blitz.repo.RepositoryDaoImpl.getOriginalFiles[...,
CheckedPath(bellisj_454/2015-10/01)]
2016-05-02 11:35:55,632 INFO  [
ome.services.util.ServiceHandler] (Server-112)  Args:    [null,
InternalSF at 945268165]

Can you tell us: how large is the directory

  /OMERO/ManagedRepository/bellisj_454/2015-10/01

?

~Josh.


> Thanks for your support,
>
>
> O.
>
>
> On 02/05/2016 07:50, Josh Moore wrote:
>>
>> On Fri, Apr 29, 2016 at 3:30 PM, Olivier Miquel
>> <olivier.miquel at mri.cnrs.fr> wrote:
>>>
>>> Dear Mark,
>>>
>>> I had tried to cleanse the binary repository, with no success. The script
>>> died after several minutes of silence.
>>
>> Hi Olivier,
>>
>> this actually looks like something went side-ways on the server-side.
>> Could you send us your the Blitz log from var/log for when this
>> happened?
>>
>> Cheers,
>> ~Josh.
>>
>>
>>> Is there an alternate way to achieve that?
>>>
>>> Log :
>>> omero at kansas:~$ /home/omero/apps/OMERO/OMERO.server/bin/omero admin
>>> cleanse
>>> /home/omero/apps/OMERO/OMERO.data
>>> Previous session expired for root on localhost:4064
>>> Server: [localhost:4064]
>>> Username: [root]
>>> Password:
>>> Created session 50852b9f-4f40-4dc2-ad14-b900a8527386
>>> (root at localhost:4064).
>>> Idle timeout: 10 min. Current group: system
>>> Traceback (most recent call last):
>>>    File "/home/omero/apps/OMERO/OMERO.server/bin/omero", line 125, in
>>> <module>
>>>      rv = omero.cli.argv()
>>>    File "/home/omero/apps/OMERO/OMERO.server/lib/python/omero/cli.py",
>>> line
>>> 1438, in argv
>>>      cli.invoke(args[1:])
>>>    File "/home/omero/apps/OMERO/OMERO.server/lib/python/omero/cli.py",
>>> line
>>> 952, in invoke
>>>      stop = self.onecmd(line, previous_args)
>>>    File "/home/omero/apps/OMERO/OMERO.server/lib/python/omero/cli.py",
>>> line
>>> 1029, in onecmd
>>>      self.execute(line, previous_args)
>>>    File "/home/omero/apps/OMERO/OMERO.server/lib/python/omero/cli.py",
>>> line
>>> 1111, in execute
>>>      args.func(args)
>>>    File
>>> "/home/omero/apps/OMERO/OMERO.server/lib/python/omero/plugins/admin.py",
>>> line 1776, in cleanse
>>>      dry_run=args.dry_run)
>>>    File
>>> "/home/omero/apps/OMERO/OMERO.server/lib/python/omero/util/cleanse.py",
>>> line
>>> 242, in cleanse
>>>      delete_empty_dirs(proxy, root, client, dry_run)
>>>    File
>>> "/home/omero/apps/OMERO/OMERO.server/lib/python/omero/util/cleanse.py",
>>> line
>>> 250, in delete_empty_dirs
>>>      is_empty_dir(repo, '/', False, to_delete)
>>>    File
>>> "/home/omero/apps/OMERO/OMERO.server/lib/python/omero/util/cleanse.py",
>>> line
>>> 282, in is_empty_dir
>>>      is_empty_dir(repo, subdirectory, may_delete_subdir, empty_subdirs):
>>>    File
>>> "/home/omero/apps/OMERO/OMERO.server/lib/python/omero/util/cleanse.py",
>>> line
>>> 282, in is_empty_dir
>>>      is_empty_dir(repo, subdirectory, may_delete_subdir, empty_subdirs):
>>>    File
>>> "/home/omero/apps/OMERO/OMERO.server/lib/python/omero/util/cleanse.py",
>>> line
>>> 282, in is_empty_dir
>>>      is_empty_dir(repo, subdirectory, may_delete_subdir, empty_subdirs):
>>>    File
>>> "/home/omero/apps/OMERO/OMERO.server/lib/python/omero/util/cleanse.py",
>>> line
>>> 277, in is_empty_dir
>>>      for entry in repo.listFiles(directory):
>>>    File
>>>
>>> "/home/omero/apps/OMERO/OMERO.server/lib/python/omero_Repositories_ice.py",
>>> line 329, in listFiles
>>>      return _M_omero.grid.Repository._op_listFiles.invoke(self, ((path,
>>> ),
>>> _ctx))
>>> Ice.UnknownException: exception ::Ice::UnknownException
>>> {
>>>      unknown = ome.conditions.InternalException:  Wrapped Exception:
>>> (org.springframework.transaction.TransactionSystemException):
>>> Could not roll back JDBC transaction; nested exception is
>>> org.postgresql.util.PSQLException: This connection has been closed.
>>>          at
>>>
>>> org.springframework.jdbc.datasource.DataSourceTransactionManager.doRollback(DataSourceTransactionManager.java:286)
>>>          at
>>>
>>> org.springframework.transaction.support.AbstractPlatformTransactionManager.processRollback(AbstractPlatformTransactionManager.java:845)
>>>          at
>>>
>>> org.springframework.transaction.support.AbstractPlatformTransactionManager.rollback(AbstractPlatformTransactionManager.java:822)
>>>          at
>>>
>>> org.springframework.transaction.interceptor.TransactionAspectSupport.completeTransactionAfterThrowing(TransactionAspectSupport.java:430)
>>>          at
>>>
>>> org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:112)
>>>          at
>>>
>>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>>>          at
>>>
>>> ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:249)
>>>          at
>>>
>>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>>>          at
>>> ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121)
>>>          at
>>>
>>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>>>          at
>>>
>>> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
>>>          at com.sun.proxy.$Proxy70.doWork(Unknown Source)
>>>          at ome.services.util.Executor$Impl.execute(Executor.java:443)
>>>          at
>>>
>>> ome.services.blitz.repo.RepositoryDaoImpl.getOriginalFiles(RepositoryDaoImpl.java:439)
>>>          at
>>>
>>> ome.services.blitz.repo.PublicRepositoryI.listFiles(PublicRepositoryI.java:263)
>>>          at
>>>
>>> omero.grid._ManagedRepositoryTie.listFiles(_ManagedRepositoryTie.java:136)
>>>          at
>>> omero.grid._RepositoryDisp.___listFiles(_RepositoryDisp.java:324)
>>>          at
>>>
>>> omero.grid._ManagedRepositoryDisp.__dispatch(_ManagedRepositoryDisp.java:585)
>>>          at IceInternal.Incoming.invoke(Incoming.java:222)
>>>          at Ice.ConnectionI.invokeAll(ConnectionI.java:2482)
>>>          at Ice.ConnectionI.dispatch(ConnectionI.java:1258)
>>>          at Ice.ConnectionI.message(ConnectionI.java:1213)
>>>          at IceInternal.ThreadPool.run(ThreadPool.java:321)
>>>          at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
>>>          at
>>> IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:693)
>>>          at java.lang.Thread.run(Thread.java:745)
>>>
>>> }
>>>
>>>
>>> On 08/03/2016 10:19, Mark Carroll wrote:
>>>>
>>>> Dear Olivier,
>>>>
>>>>> Is there somewhere a script to check and clean the database?
>>>>> Our omero.server  is in production for about five years now and perhaps
>>>>> we now need to clean some unused stuff in the database?
>>>>
>>>>
>>>> What I'd been mentioning to Carnë in this thread about periodic VACUUMs
>>>> and suchlike probably suffices for PostgreSQL; their documentation is
>>>> good on this point. If we notice that our server has wrongly been
>>>> leaving some kind of debris in the database then we usually include
>>>> cleanup in upgrade scripts so it gets done for our users anyway in time;
>>>> for instance, the upgrade that we've been discussing included some
>>>> removal of uselessly duplicate rows from a large table.
>>>>
>>>> However, separately from the database, there is also the binary
>>>> repository: it is probably worth glancing at the first part of
>>>>
>>>>
>>>> https://www.openmicroscopy.org/site/support/omero/sysadmins/server-backup-and-restore.html
>>>> which talks about cleaning files up before doing a backup. If you
>>>> haven't run a "bin/omero admin cleanse" for a while then that might be
>>>> something that's worth adding to your infrequent maintenance checklist.
>>>>
>>>> -- Mark
>>>>
>>>> The University of Dundee is a registered Scottish Charity, No: SC015096
>>>> _______________________________________________
>>>> ome-users mailing list
>>>> ome-users at lists.openmicroscopy.org.uk
>>>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>>
>> _______________________________________________
>> ome-users mailing list
>> ome-users at lists.openmicroscopy.org.uk
>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>
>
>
> --
> Olivier Miquel
> Administrateur Système et Réseau
> Plate-forme technologique MRI
> Tel : 06.50.19.27.43
> Mail:olivier.miquel at mri.cnrs.fr
>
> _______________________________________________
> ome-users mailing list
> ome-users at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users


More information about the ome-users mailing list