[ome-users] OMERO Import issues...

Thu Sep 24 02:25:01 BST 2009

That's essentially the exact same issue I had-- metadata was visible, but
displayed an X when trying to look at the image...

On Wed, Sep 23, 2009 at 5:15 PM, Björn Quast
<bquast at evolution.uni-bonn.de>wrote:

> Hi,
>
>
> I just tried it with a number of grayscale tif images, all about 30 Mb, one
> layer. When I use the importer-gui the import hangs after 1 image (with java
> -Xmx256m). After changing to -Xmx2000m the importer stops after three files,
> but several test show me that the number of loaded files do not depend on
> the memory given by the -Xmx flag. The importer.log is attached.
> The "KeepAlive ping" looks, as if the importer looses the connection, but
> the client was started on the same machine with the server.
>
>
> Sometimes error messages of the Ice.Marshal occur:
>
>
> Ice.MarshalException
> reason = "OutOfMemoryError occurred while allocating a ByteBuffer"
> at IceInternal.Buffer.reserve(Buffer.java:163)
> at IceInternal.Buffer.resize(Buffer.java:72)
> at IceInternal.Buffer.expand(Buffer.java:59)
> at IceInternal.BasicStream.expand(BasicStream.java:2031)
> at IceInternal.BasicStream.writeInt(BasicStream.java:872)
> at omero.api._RawPixelsStoreDelM.setPlane(_RawPixelsStoreDelM.java:1128)
> at
> omero.api.RawPixelsStorePrxHelper.setPlane(RawPixelsStorePrxHelper.java:1546)
> at
> omero.api.RawPixelsStorePrxHelper.setPlane(RawPixelsStorePrxHelper.java:1518)
> at
> ome.formats.OMEROMetadataStoreClient.setPlane(OMEROMetadataStoreClient.java:2469)
> at ome.formats.importer.ImportLibrary.importData(ImportLibrary.java:450)
> at ome.formats.importer.ImportLibrary.importImage(ImportLibrary.java:354)
> at ome.formats.importer.ImportHandler.importImages(ImportHandler.java:192)
> at ome.formats.importer.ImportHandler.access$0(ImportHandler.java:110)
> at ome.formats.importer.ImportHandler$1.run(ImportHandler.java:91)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
> at java.nio.ByteBuffer.allocate(ByteBuffer.java:329)
> at IceInternal.Buffer.reserve(Buffer.java:141)
> ... 13 more
>
>
>
>
>
> After connecting to the database with OMEROinsight_unix.sh and trying to
> open one of the loaded images another error message occurs: (again
> attached). Most of the large single layer images are shown by an X as
> thumbnail, but their metadata are viewable. Images, which are shown as
> thumbnails can also be shown in detail.
>
>
>
> I hope this help to find the problem.
>
>
> Björn
>
>
>
>
> > On Wed, Sep 23, 2009 at 11:25 AM, Curtis Rueden <ctrueden at wisc.edu>
> wrote:
> > > Hi Björn & David,
> > >
> > > Thanks for the feedback, and sorry for your trouble importing large
> image
> > > planes. As of Beta 4.0.3, the OMERO.importer defaults to a maximum heap
> > > size of 1024 MB. So any image planes of 1 GB or more are certainly too
> > > large to process successfully, and even image planes over 512 MB may
> have
> > > problems due to temporary overhead or garbage collection issues.
> > >
> > > One workaround is to increase the maximum heap size. With a 32-bit JVM,
> > > this limit can often be 1536 or even ~1800 GB depending on your
> operating
> > > system. The process for increasing the heap size is platform-dependent,
> > > however. For example, on Mac OS X, right click the OMERO.importer icon,
> > > choose "Show package contents," navigate into Contents, and edit the
> > > Info.plist file's line "-Xmx1024M" to read "-Xmx1700M" or whatever
> value
> > > you wish to try.
> > >
> > > That said, we have plans to improve the support for large image planes
> by
> > > processing them in tiles, negating the need for such a large amount of
> > > RAM at once. Unfortunately, this support is not planned until at least
> > > Beta 4.2. The Bio-Formats readers already support processing an image
> > > plane in tiles, but the writers do not, so the bfconvert command line
> > > tool does not yet support this either.
> > >
> > > -Curtis
> > >
> > >
> > > On Wed, Sep 23, 2009 at 3:28 AM, Björn Quast
> > > <bquast at evolution.uni-bonn.de
> > >
> > > > wrote:
> > >>
> > >> Hi David,
> > >>
> > >> I have had the same importing problems with one layer tif images, when
> > >> they
> > >> exceed a specific image size . There is no problem to import very
> large
> > >> multi-
> > >> layer tifs (about 500 Mb), but if one layer is to large the import
> > >> stopped and
> > >> I got a message on the console, that the java heap memory was
> exceeded.
> > >> I can only guess, that the importer fails to read in large strips of
> > >> image data. tif files may store all image data in one chunk or line by
> > >> line or in
> > >> smaller pieces. But even with line by line stored images I got the
> > >> problems,
> > >> if one layer is to large.
> > >>
> > >> It also happens, that a few images were imported but than the importer
> > >> stops.
> > >> Again, I can only guess that there is a memory issue.
> > >>
> > >> In the moment I can not reproduce it, because I have no omero running,
> > >> but perhaps I can give more details when my server is up again.
> > >>
> > >> Björn
> > >>
> > >> David Gutman wrote:
> > >> > I am having a lot of problems trying to import images into OMERO. I
> > >> > am using the 4.0.3beta code and having issues trying to import any
> of
> > >> > my images.....
> > >> >
> > >> > The images are initially acquired as Aperio SVS files ....they are
> > >> > basically a TIFF container with JPEG2000 tiles embedded in them.
> > >> > I've tried the following scenarios and still can't get all of my
> > >> > images to consistently import, and get all sorts of errors, so
> wanted
> > >> > to see if the community had any thoughts...
> > >> >
> > >> > 1) Split 6 channel SVS file into individual channels using
> > >> > tiffsplit--- these images are just standard tiff's, and I can even
> > >> > view them with imagemagick, and other graphics viewers (At least the
> > >> > small ones)
> > >> > This produces 6 images, the really small images seem to import okay
> > >> > sometimes, but not always (like 1024x512 or whatever-- these
> generally
> > >> > are just an image of the label on the slide)...
> > >> >
> > >> > So some of the frames, but of course not the large frames
> (22090x27696
> > >> > pixels) will not load either through the GUI or through the command
> > >> > line interpreter--- I've restricted this now where I don't even try
> to
> > >> > upload images greater than 250 MegaPixels (so well under 1Gigapixel
> > >> > even considering all three channels) ......but I've had issues with
> > >> > even much smaller images....
> > >> >
> > >> > Has anyone had any similar issues trying to load in large files with
> > >> > the current beta of OMERO I've even tried to transcode some of these
> > >> > using the bioformats tools to OME-XML.TIFF files with similar
> > >> > disappointing results
> > >> >
> > >> > Oddly-- some of my images upload OK, and some do not...
> > >> >
> > >> > 2966455 2009-09-16 13:57 TCGA-06-0214-01B-03-BS3aad.tif
> > >> > 158051 2009-09-16 13:57 TCGA-06-0214-01B-03-BS3aab.tif
> > >> >
> > >> > The "b" image loaded fine, the "d" image did not-- it's not a
> terribly
> > >> > big image, but I keep getting weird errors when I try and import
> it--
> > >> > has anyone had any issues lately with the OMEROIMPORTER
> function/gui?
> > >>
> > >> --
> > >> Dr. Björn Quast
> > >> Universität Bonn
> > >> Institut für Evolutionsbiologie und Ökologie
> > >> 53121 Bonn
> > >> Tel.: 0228/735758 email: bquast at evolution.uni-bonn.de
> > >> _______________________________________________
> > >> ome-users mailing list
> > >> ome-users at lists.openmicroscopy.org.uk
> > >> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>
>
>
> --
> Dr. Björn Quast
> Universität Bonn
> Institut für Evolutionsbiologie und Ökologie
> 53121 Bonn
> Tel.: 0228/735758 email: bquast at evolution.uni-bonn.de
>
> _______________________________________________
> ome-users mailing list
> ome-users at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>
>

-- 
David A Gutman, M.D. Ph.D.
Center for Comprehensive Informatics
Emory University School of Medicine
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-users/attachments/20090923/7b53c677/attachment.html>