[ome-users] OMERO Import issues...
Björn Quast
bquast at evolution.uni-bonn.de
Wed Sep 23 22:15:52 BST 2009
Hi,
I just tried it with a number of grayscale tif images, all about 30 Mb, one
layer. When I use the importer-gui the import hangs after 1 image (with java -
Xmx256m). After changing to -Xmx2000m the importer stops after three files, but
several test show me that the number of loaded files do not depend on the
memory given by the -Xmx flag. The importer.log is attached.
The "KeepAlive ping" looks, as if the importer looses the connection, but the
client was started on the same machine with the server.
Sometimes error messages of the Ice.Marshal occur:
Ice.MarshalException
reason = "OutOfMemoryError occurred while allocating a ByteBuffer"
at IceInternal.Buffer.reserve(Buffer.java:163)
at IceInternal.Buffer.resize(Buffer.java:72)
at IceInternal.Buffer.expand(Buffer.java:59)
at IceInternal.BasicStream.expand(BasicStream.java:2031)
at IceInternal.BasicStream.writeInt(BasicStream.java:872)
at omero.api._RawPixelsStoreDelM.setPlane(_RawPixelsStoreDelM.java:1128)
at
omero.api.RawPixelsStorePrxHelper.setPlane(RawPixelsStorePrxHelper.java:1546)
at
omero.api.RawPixelsStorePrxHelper.setPlane(RawPixelsStorePrxHelper.java:1518)
at
ome.formats.OMEROMetadataStoreClient.setPlane(OMEROMetadataStoreClient.java:2469)
at ome.formats.importer.ImportLibrary.importData(ImportLibrary.java:450)
at ome.formats.importer.ImportLibrary.importImage(ImportLibrary.java:354)
at ome.formats.importer.ImportHandler.importImages(ImportHandler.java:192)
at ome.formats.importer.ImportHandler.access$0(ImportHandler.java:110)
at ome.formats.importer.ImportHandler$1.run(ImportHandler.java:91)
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:329)
at IceInternal.Buffer.reserve(Buffer.java:141)
... 13 more
After connecting to the database with OMEROinsight_unix.sh and trying to open
one of the loaded images another error message occurs: (again attached). Most
of the large single layer images are shown by an X as thumbnail, but their
metadata are viewable. Images, which are shown as thumbnails can also be shown
in detail.
I hope this help to find the problem.
Björn
> On Wed, Sep 23, 2009 at 11:25 AM, Curtis Rueden <ctrueden at wisc.edu> wrote:
> > Hi Björn & David,
> >
> > Thanks for the feedback, and sorry for your trouble importing large image
> > planes. As of Beta 4.0.3, the OMERO.importer defaults to a maximum heap
> > size of 1024 MB. So any image planes of 1 GB or more are certainly too
> > large to process successfully, and even image planes over 512 MB may have
> > problems due to temporary overhead or garbage collection issues.
> >
> > One workaround is to increase the maximum heap size. With a 32-bit JVM,
> > this limit can often be 1536 or even ~1800 GB depending on your operating
> > system. The process for increasing the heap size is platform-dependent,
> > however. For example, on Mac OS X, right click the OMERO.importer icon,
> > choose "Show package contents," navigate into Contents, and edit the
> > Info.plist file's line "-Xmx1024M" to read "-Xmx1700M" or whatever value
> > you wish to try.
> >
> > That said, we have plans to improve the support for large image planes by
> > processing them in tiles, negating the need for such a large amount of
> > RAM at once. Unfortunately, this support is not planned until at least
> > Beta 4.2. The Bio-Formats readers already support processing an image
> > plane in tiles, but the writers do not, so the bfconvert command line
> > tool does not yet support this either.
> >
> > -Curtis
> >
> >
> > On Wed, Sep 23, 2009 at 3:28 AM, Björn Quast
> > <bquast at evolution.uni-bonn.de
> >
> > > wrote:
> >>
> >> Hi David,
> >>
> >> I have had the same importing problems with one layer tif images, when
> >> they
> >> exceed a specific image size . There is no problem to import very large
> >> multi-
> >> layer tifs (about 500 Mb), but if one layer is to large the import
> >> stopped and
> >> I got a message on the console, that the java heap memory was exceeded.
> >> I can only guess, that the importer fails to read in large strips of
> >> image data. tif files may store all image data in one chunk or line by
> >> line or in
> >> smaller pieces. But even with line by line stored images I got the
> >> problems,
> >> if one layer is to large.
> >>
> >> It also happens, that a few images were imported but than the importer
> >> stops.
> >> Again, I can only guess that there is a memory issue.
> >>
> >> In the moment I can not reproduce it, because I have no omero running,
> >> but perhaps I can give more details when my server is up again.
> >>
> >> Björn
> >>
> >> David Gutman wrote:
> >> > I am having a lot of problems trying to import images into OMERO. I
> >> > am using the 4.0.3beta code and having issues trying to import any of
> >> > my images.....
> >> >
> >> > The images are initially acquired as Aperio SVS files ....they are
> >> > basically a TIFF container with JPEG2000 tiles embedded in them.
> >> > I've tried the following scenarios and still can't get all of my
> >> > images to consistently import, and get all sorts of errors, so wanted
> >> > to see if the community had any thoughts...
> >> >
> >> > 1) Split 6 channel SVS file into individual channels using
> >> > tiffsplit--- these images are just standard tiff's, and I can even
> >> > view them with imagemagick, and other graphics viewers (At least the
> >> > small ones)
> >> > This produces 6 images, the really small images seem to import okay
> >> > sometimes, but not always (like 1024x512 or whatever-- these generally
> >> > are just an image of the label on the slide)...
> >> >
> >> > So some of the frames, but of course not the large frames (22090x27696
> >> > pixels) will not load either through the GUI or through the command
> >> > line interpreter--- I've restricted this now where I don't even try to
> >> > upload images greater than 250 MegaPixels (so well under 1Gigapixel
> >> > even considering all three channels) ......but I've had issues with
> >> > even much smaller images....
> >> >
> >> > Has anyone had any similar issues trying to load in large files with
> >> > the current beta of OMERO I've even tried to transcode some of these
> >> > using the bioformats tools to OME-XML.TIFF files with similar
> >> > disappointing results
> >> >
> >> > Oddly-- some of my images upload OK, and some do not...
> >> >
> >> > 2966455 2009-09-16 13:57 TCGA-06-0214-01B-03-BS3aad.tif
> >> > 158051 2009-09-16 13:57 TCGA-06-0214-01B-03-BS3aab.tif
> >> >
> >> > The "b" image loaded fine, the "d" image did not-- it's not a terribly
> >> > big image, but I keep getting weird errors when I try and import it--
> >> > has anyone had any issues lately with the OMEROIMPORTER function/gui?
> >>
> >> --
> >> Dr. Björn Quast
> >> Universität Bonn
> >> Institut für Evolutionsbiologie und Ökologie
> >> 53121 Bonn
> >> Tel.: 0228/735758 email: bquast at evolution.uni-bonn.de
> >> _______________________________________________
> >> ome-users mailing list
> >> ome-users at lists.openmicroscopy.org.uk
> >> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
--
Dr. Björn Quast
Universität Bonn
Institut für Evolutionsbiologie und Ökologie
53121 Bonn
Tel.: 0228/735758 email: bquast at evolution.uni-bonn.de
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-users/attachments/20090923/0b5741a3/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: importer.log
Type: text/x-log
Size: 12916 bytes
Desc: not available
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-users/attachments/20090923/0b5741a3/attachment.bin>
-------------- next part --------------
java.lang.Exception: omero.InternalException
serverStackTrace = "ome.conditions.InternalException: Wrapped Exception: (java.lang.OutOfMemoryError):
Java heap space
"
serverExceptionClass = "ome.conditions.InternalException"
message = " Wrapped Exception: (java.lang.OutOfMemoryError):
Java heap space"
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
at java.lang.Class.newInstance0(Class.java:372)
at java.lang.Class.newInstance(Class.java:325)
at IceInternal.BasicStream$DynamicUserExceptionFactory.createAndThrow(BasicStream.java:2127)
at IceInternal.BasicStream.throwException(BasicStream.java:1541)
at IceInternal.Outgoing.throwUserException(Outgoing.java:442)
at omero.api._RenderingEngineDelM.renderAsPackedInt(_RenderingEngineDelM.java:1324)
at omero.api.RenderingEnginePrxHelper.renderAsPackedInt(RenderingEnginePrxHelper.java:1821)
at omero.api.RenderingEnginePrxHelper.renderAsPackedInt(RenderingEnginePrxHelper.java:1793)
at org.openmicroscopy.shoola.env.rnd.RenderingControlProxy.renderPlaneUncompressed(RenderingControlProxy.java:425)
at org.openmicroscopy.shoola.env.rnd.RenderingControlProxy.renderPlane(RenderingControlProxy.java:1287)
at org.openmicroscopy.shoola.agents.imviewer.view.ImViewerModel.fireImageRetrieval(ImViewerModel.java:647)
at org.openmicroscopy.shoola.agents.imviewer.view.ImViewerComponent.renderXYPlane(ImViewerComponent.java:948)
at org.openmicroscopy.shoola.agents.imviewer.view.ImViewerComponent.setRenderingControl(ImViewerComponent.java:914)
at org.openmicroscopy.shoola.agents.imviewer.RenderingControlLoader.handleResult(RenderingControlLoader.java:157)
at org.openmicroscopy.shoola.env.data.events.DSCallAdapter.eventFired(DSCallAdapter.java:90)
at org.openmicroscopy.shoola.env.data.views.BatchCallMonitor$1.run(BatchCallMonitor.java:124)
at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:226)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:602)
at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:275)
at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:200)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:190)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:185)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:177)
at java.awt.EventDispatchThread.run(EventDispatchThread.java:138)
at org.openmicroscopy.shoola.env.ui.UserNotifierImpl.showErrorDialog(UserNotifierImpl.java:128)
at org.openmicroscopy.shoola.env.ui.UserNotifierImpl.notifyError(UserNotifierImpl.java:179)
at org.openmicroscopy.shoola.env.ui.UserNotifierImpl.notifyError(UserNotifierImpl.java:168)
at org.openmicroscopy.shoola.agents.imviewer.view.ImViewerComponent.handleException(ImViewerComponent.java:1747)
at org.openmicroscopy.shoola.agents.imviewer.view.ImViewerModel.fireImageRetrieval(ImViewerModel.java:649)
at org.openmicroscopy.shoola.agents.imviewer.view.ImViewerComponent.renderXYPlane(ImViewerComponent.java:948)
at org.openmicroscopy.shoola.agents.imviewer.view.ImViewerComponent.setRenderingControl(ImViewerComponent.java:914)
at org.openmicroscopy.shoola.agents.imviewer.RenderingControlLoader.handleResult(RenderingControlLoader.java:157)
at org.openmicroscopy.shoola.env.data.events.DSCallAdapter.eventFired(DSCallAdapter.java:90)
at org.openmicroscopy.shoola.env.data.views.BatchCallMonitor$1.run(BatchCallMonitor.java:124)
at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:226)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:602)
at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:275)
at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:200)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:190)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:185)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:177)
at java.awt.EventDispatchThread.run(EventDispatchThread.java:138)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: omeroinsight.log
Type: text/x-log
Size: 4823 bytes
Desc: not available
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-users/attachments/20090923/0b5741a3/attachment-0001.bin>
More information about the ome-users
mailing list