[ome-users] NullPointerException in FileMaker

Chris Allan callan at lifesci.dundee.ac.uk
Tue Aug 7 08:59:14 BST 2012 

Hi Harri,

On 6 Aug 2012, at 15:42, Harri Jäälinoja wrote:

> Hi all,
> I think I've solved this. Some googling points to NFS locking issues:
> https://forums.oracle.com/forums/thread.jspa?threadID=2232128&tstart=0&messageID=9907997
> 
> This makes sense, because in our new environment, the big data drive is a NFS share.
> 
> I did this:
> service nfslock start
> 
> and after that, I haven't seen the errors anymore.

We'll certainly try and add that to the startup script tests in the future so that it's obvious when there is a complete lack of locking.

> 
> Do you have any comments in general about having OMERO store data on NFS (speed for example)? We have also the PostgreSQL data directory there. In our case we got this 4T storage area for free from a national computing center, and I don't think we have much say about how we receive it, but it would be good to know if there are any possible issues.

Really that heavily depends on the performance of your NFS data store. One of the classical problems with NFS is locking semantics, and really this applies to any distributed or network filesystem. OMERO makes fairly heavy use of POSIX locking, especially on the Files and Index portions of the binary repository. You may wish to store your Index on local disk, it will be faster, less prone to corruption and should be small anyway. Storing your PostgreSQL data directory on NFS is probably a very bad idea unless you have a really good understanding of the semantics of the NFS data store you're using.

Over the next few days I'll try and put up some of the specific dos and don'ts on the troubleshooting page on the OME website.

> 
> Cheers,
> Harri

-Chris


> On 06/08/12 17:08, Harri Jäälinoja wrote:
>> Hi,
>> I added some more logging to
>> components/blitz/src/ome/services/blitz/repo/AbstractRepositoryI.java,
>> in the helper GetOrGreateRepo:
>> 
>>             } catch (Exception e) {
>>                 log.error("Error in GetOrCreateRepo.doWork",e);
>>                 e.printStackTrace();
>>                 fileMaker.close(); // If anything goes awry, we release
>> for
>>                 // others!
>>                 return e;
>>             }
>> 
>> This is what it prints:
>> 
>> 2012-08-06 17:01:03,182 ERROR [.services.blitz.repo.AbstractRepositoryI]
>> (2-thread-1) Error in GetOrCreateRepo.doWork
>> java.io.IOException: No locks available
>>     at sun.nio.ch.FileChannelImpl.lock0(Native Method)
>>     at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:832)
>>     at java.nio.channels.FileChannel.lock(FileChannel.java:860)
>>     at ome.services.blitz.repo.FileMaker.getLine(FileMaker.java:95)
>>     at
>> ome.services.blitz.repo.AbstractRepositoryI$GetOrCreateRepo.doWork(AbstractRepositoryI.java:233)
>> 
>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> 
>> 
>> What does this mean, why are there no locks available?
>> 
>> Thanks,
>> Harri
>> 
>> 
>> On 06/08/12 12:33, Harri Jäälinoja wrote:
>>> Hi again,
>>> 
>>> I forgot to mention that this error is repeated once a minute:
>>> 
>>> [root at omerovm-1 ~]# grep ERROR /var/log/OMERO/Blitz-0.log |tail
>>> 2012-08-06 12:21:16,970 ERROR [.services.blitz.repo.AbstractRepositoryI]
>>> (2-thread-5) Unexpected error in called executor on takeover
>>> 2012-08-06 12:22:16,988 ERROR [.services.blitz.repo.AbstractRepositoryI]
>>> (2-thread-3) Unexpected error in called executor on takeover
>>> 2012-08-06 12:23:16,971 ERROR [.services.blitz.repo.AbstractRepositoryI]
>>> (2-thread-2) Unexpected error in called executor on takeover
>>> 2012-08-06 12:24:16,951 ERROR [.services.blitz.repo.AbstractRepositoryI]
>>> (2-thread-3) Unexpected error in called executor on takeover
>>> 2012-08-06 12:25:16,964 ERROR [.services.blitz.repo.AbstractRepositoryI]
>>> (2-thread-2) Unexpected error in called executor on takeover
>>> 
>>> Regards,
>>> Harri
>>> 
>>> On 03/08/12 14:37, harri.jaalinoja at helsinki.fi wrote:
>>>> Hello,
>>>> 
>>>> looking at
>>>> https://github.com/openmicroscopy/openmicroscopy/blob/master/components/blitz/src/ome/services/blitz/repo/FileMaker.java,
>>>> 
>>>> 
>>>> 
>>>> 
>>>> the NPE is thrown because there is no lock to release:
>>>>         try {
>>>>                 lock.release(); // line 132
>>>>             } catch (IOException e) {
>>>>                 log.warn("Failed to release lock");
>>>>             }
>>>> 
>>>> It seems lock is only initialized if getLine() is called, otherwise it
>>>> is null:
>>>> 
>>>>     public String getLine() throws Exception {
>>>>         synchronized (mutex) {
>>>> 
>>>>             if (dbUuid == null) {
>>>>                 throw new InternalException("Not initialized");
>>>>             }
>>>> 
>>>>             lock = dotLockRaf.getChannel().lock();
>>>> 
>>>> 
>>>> So somehow the getLine call was omitted now?
>>>> 
>>>> Cheers,
>>>> Harri
>>>> 
>>>> 
>>>> Quoting harri.jaalinoja at helsinki.fi:
>>>> 
>>>>> Hello All,
>>>>> I noticed an error in the OMERO 4.4.1 logs (on RedHat5.8). I just
>>>>> restarted OMERO server, so this in not initiated by user action
>>>>> (unless OMERO stores jobs over shutdown and continues them after
>>>>> restart: yesterday I left "combine images" script running).
>>>>> 
>>>>> The stack trace is below. Any ideas what is wrong?
>>>>> 
>>>>> Thanks,
>>>>> Harri
>>>>> 
>>>>> ----
>>>>> 2012-08-03 13:28:16,981 WARN  [
>>>>> ome.services.util.ServiceHandler] (2-thread-4) Method interface
>>>>> ome.services.util.Executor$Work.doWork invocation took 6043
>>>>> 2012-08-03 13:28:16,981 ERROR
>>>>> [.services.blitz.repo.AbstractRepositoryI] (2-thread-4) Unexpected
>>>>> error in called executor on takeover
>>>>> ome.conditions.InternalException:  Wrapped Exception:
>>>>> (java.lang.NullPointerException):
>>>>> null
>>>>>    at ome.services.blitz.repo.FileMaker.close(FileMaker.java:132)
>>>>>    at
>>>>> ome.services.blitz.repo.AbstractRepositoryI$GetOrCreateRepo.doWork(AbstractRepositoryI.java:301)
>>>>> 
>>>>> 
>>>>> 
>>>>>    at sun.reflect.GeneratedMethodAccessor225.invoke(Unknown Source)
>>>>>    at
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>> 
>>>>> 
>>>>> 
>>>>>    at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>    at
>>>>> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
>>>>> 
>>>>> 
>>>>> 
>>>>>    at
>>>>> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
>>>>> 
>>>>> 
>>>>> 
>>>>>    at
>>>>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
>>>>> 
>>>>> 
>>>>> 
>>>>>    at
>>>>> ome.services.util.Executor$Impl$Interceptor.invoke(Executor.java:508)
>>>>>    at
>>>>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>>>>> 
>>>>> 
>>>>> 
>>>>>    at ome.security.basic.EventHandler.invoke(EventHandler.java:154)
>>>>>    at
>>>>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>>>>> 
>>>>> 
>>>>> 
>>>>>    at
>>>>> org.springframework.orm.hibernate3.HibernateInterceptor.invoke(HibernateInterceptor.java:111)
>>>>> 
>>>>> 
>>>>> 
>>>>>    at
>>>>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>>>>> 
>>>>> 
>>>>> 
>>>>>    at
>>>>> org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:108)
>>>>> 
>>>>> 
>>>>> 
>>>>>    at
>>>>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>>>>> 
>>>>> 
>>>>> 
>>>>>    at
>>>>> ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:241)
>>>>> 
>>>>> 
>>>>> 
>>>>>    at
>>>>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>>>>> 
>>>>> 
>>>>> 
>>>>>    at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:116)
>>>>>    at
>>>>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>>>>> 
>>>>> 
>>>>> 
>>>>>    at
>>>>> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
>>>>> 
>>>>> 
>>>>> 
>>>>>    at $Proxy65.doWork(Unknown Source)
>>>>>    at ome.services.util.Executor$Impl.execute(Executor.java:406)
>>>>>    at ome.services.util.Executor$Impl.execute(Executor.java:350)
>>>>>    at
>>>>> ome.services.blitz.repo.AbstractRepositoryI.takeover(AbstractRepositoryI.java:120)
>>>>> 
>>>>> 
>>>>> 
>>>>>    at sun.reflect.GeneratedMethodAccessor240.invoke(Unknown Source)
>>>>>    at
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>> 
>>>>> 
>>>>> 
>>>>>    at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>    at
>>>>> org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:273)
>>>>>    at
>>>>> org.springframework.scheduling.quartz.MethodInvokingJobDetailFactoryBean$MethodInvokingJob.executeInternal(MethodInvokingJobDetailFactoryBean.java:264)
>>>>> 
>>>>> 
>>>>> 
>>>>>    at
>>>>> org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:86)
>>>>> 
>>>>> 
>>>>> 
>>>>>    at org.quartz.core.JobRunShell.run(JobRunShell.java:203)
>>>>>    at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>> 
>>>>> 
>>>>> 
>>>>>    at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>> 
>>>>> 
>>>>> 
>>>>>    at java.lang.Thread.run(Thread.java:662)
>>>>> 
>>>>> _______________________________________________
>>>>> ome-users mailing list
>>>>> ome-users at lists.openmicroscopy.org.uk
>>>>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>>>>> 
>>>> 
>>>> 
>>>> _______________________________________________
>>>> ome-users mailing list
>>>> ome-users at lists.openmicroscopy.org.uk
>>>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>>> 
>>> 
>>> _______________________________________________
>>> ome-users mailing list
>>> ome-users at lists.openmicroscopy.org.uk
>>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>> 
>> 
>> _______________________________________________
>> ome-users mailing list
>> ome-users at lists.openmicroscopy.org.uk
>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
> 
> 
> _______________________________________________
> ome-users mailing list
> ome-users at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users

More information about the ome-users mailing list