[ome-users] [Software Feedback] Bioformat cpp withitk
Curtis Rueden
ctrueden at wisc.edu
Thu Jul 12 22:14:57 BST 2012
Hi Matthias,
Thanks for the additional information. The stack trace was helpful in
quickly finding the problem.
It is a bug in ITKBridgePipes (the Java side of BF-ITK), which does not
call the method openBytes(no, x, y, w, h) but rather always openBytes(no)
and then crops after the fact, as you suspected.
I have a pushed a proposed (untested) fix:
https://github.com/ctrueden/bioformats/commit/eb83ace0f0a121c8c44e3fb8b95ec5040f029c55
I have uploaded a build for you to test at:
http://dev.loci.wisc.edu/curtis/2012-07-12-itk-crop/loci_tools.jar
Just replace your old loci_tools.jar and let us know how it goes! If all is
well, I will file a PR to merge the fix into the mainline.
Regards,
Curtis
On Thu, Jul 12, 2012 at 3:52 PM, Matthias Noll <
matthias.noll at igd.fraunhofer.de> wrote:
> Hi Curtis,
> I used the IORegion to set a subimage region for reading e.g.
>
> ImageType2D::SizeType smallSize;
>> smallSize.Fill(500);
>> ImageType2D::IndexType index;
>> index.Fill(0);
>> ImageType2D::RegionType region(index, smallSize);
>> try{
>> m_reader->GetOutput()->**SetRequestedRegion(region);
>> m_reader->Update();
>> }catch(itk::ExceptionObject &e){return;}
>>
>
>
> The debug window shows that the region is set correctly. Also the read
> command consists of filename and selected image region.
> So I would say the itk side of the reader is well in order. At least as
> far as I can tell right now.
>
> Debug: In j:\itk4_1_bio\itk4.1.0\src\**modules\io\imagebase\include\**itkImageIOBase.h,
>> line 161
>> BioFormatsImageIO (0000000003689390): setting IORegion to ImageIORegion
>> (0000000000B1F2D8)
>> Dimension: 2
>> Index: 0 0
>> Size: 500 500
>>
>> Debug: In J:\ITK4_1_Bio\bioformats\src\**components\native\bf-itk-pipe\**itkBioFormatsImageIO.cxx,
>> line 647
>> BioFormatsImageIO (0000000003689390): BioFormatsImageIO::Read
>>
>> Debug: In J:\ITK4_1_Bio\bioformats\src\**components\native\bf-itk-pipe\**itkBioFormatsImageIO.cxx,
>> line 668
>> BioFormatsImageIO (0000000003689390): BioFormatsImageIO::Read command:
>> read D:/CD45ROdk_c03H1665-8_n1_RS2 - 2010-11-30-largest.ome.tif 0
>> 500 0 500 0 1 0 1 0 1
>>
>> Debug: In J:\ITK4_1_Bio\bioformats\src\**components\native\bf-itk-pipe\**itkBioFormatsImageIO.cxx,
>> line 284
>> BioFormatsImageIO (0000000003689390): BioFormatsImageIO::**DestroyJavaProcess
>> killing java process
>>
>> Debug: In J:\ITK4_1_Bio\bioformats\src\**components\native\bf-itk-pipe\**itkBioFormatsImageIO.cxx,
>> line 289
>> BioFormatsImageIO (0000000003689390): BioFormatsImageIO::**DestroyJavaProcess
>> destroying java process
>>
>
> But looking at the following error message of the piped java exception,
> the openBytes() function in FormatReader.java tries to allocate an array of
> the dimension 69632 x 37888 x 3 x 1 whereas is should only allocate the
> requested 500x500x3x1.
>
> itk::ERROR: BioFormatsImageIO(**0000000003689390): BioFormatsImageIO:
>> 'ITKBridgePipe read' exited abnormally. Exception in thread "main"
>> loci.formats.FormatException: Image plane too large. Only 2GB of data can
>> be extracted at one time. You can workaround the problem by opening the
>> plane in tiles; for further details, see: http://www.openmicroscopy.org/*
>> *site/support/faq/bio-formats/**i-see-an-outofmemory-or-**
>> negativearraysize-error-**message-when-attempting-to-**
>> open-an-svs-or-jpeg-2000-file.**-what-does-this-mean<http://www.openmicroscopy.org/site/support/faq/bio-formats/i-see-an-outofmemory-or-negativearraysize-error-message-when-attempting-to-open-an-svs-or-jpeg-2000-file.-what-does-this-mean>
>> at loci.formats.FormatReader.**openBytes(FormatReader.java:**760)
>> at loci.formats.FormatReader.**openBytes(FormatReader.java:**739)
>> at loci.formats.ImageReader.**openBytes(ImageReader.java:**393)
>> at loci.formats.itk.**ITKBridgePipes.read(**
>> ITKBridgePipes.java:325)
>> at loci.formats.itk.**ITKBridgePipes.executeCommand(**
>> ITKBridgePipes.java:133)
>> at loci.formats.itk.**ITKBridgePipes.waitForInput(**
>> ITKBridgePipes.java:94)
>> at loci.formats.itk.**ITKBridgePipes.main(**
>> ITKBridgePipes.java:554)
>> Caused by: java.lang.**IllegalArgumentException: Array size too large:
>> 69632 x 37888 x 3 x 1
>> at loci.common.DataTools.**safeMultiply32(DataTools.java:**888)
>> at loci.common.DataTools.**allocate(DataTools.java:862)
>> at loci.formats.FormatReader.**openBytes(FormatReader.java:**757)
>> ... 6 more
>>
>
> I'm not sure where this problem arises, since I can't debug the java part,
> at least I'm not sure how to. Maybe the java part of the reader resets the
> image region with the actual image header information, reads out everything
> and crops the result later, if the reading worked.
> I tested this with a smaller image of the dimensions 19000x8000 and the
> same requested region. This worked but the read took a while. And also I
> had to change the java vm memory from 265MB to 1024MB which points also to
> a large amount of data allocation.
>
> Best regards,
> Matthias
>
>
> Am 12/07/2012 21:44, schrieb Curtis Rueden:
>
>> Hi Matthias,
>>
>>
>> Now I encountered another problem. With normal sized images everything
>>> works fine. But since I pretty much only have images that are larger than
>>> 2GB, the Java machine has not enough memory to "load" the whole image
>>> plane. However this is not really what I want to do anyway. It seems that
>>> itk has a way to manage large files by opening them using streams. An
>>> image
>>> is split into pieces on read so that large images can be processes
>>> sequentially. I'm not sure but this functionality seems not to be
>>> implemented down to the wrapping bioformat reader for itk.
>>>
>>
>> As you observed, Bio-Formats has the capability to read in only a subset
>> of
>> image data, including a cropped region of each XY plane. The BF-ITK plugin
>> is supposed to provide this capability as well, using the "IORegion"
>> feature of ITK ImageIO. That said, we have not really tested this feature,
>> and there may be a problem with it. I am actually not even sure how the
>> calling code should go about specifying the IORegion. But I suggest you
>> give it a try (ask on ITK-users if you need assistance) and then report
>> back if you find any problems with it.
>>
>> HTH,
>> Curtis
>>
>>
>> On Wed, Jul 11, 2012 at 3:54 AM, Matthias Noll <
>> matthias.noll at igd.fraunhofer.**de <matthias.noll at igd.fraunhofer.de>>
>> wrote:
>>
>> Hi Churtis,
>>>
>>> I managed to apply the reader after deregistration of other readers that
>>> were interfering in the reader selection.
>>>
>>> Now I encountered another problem. With normal sized images everything
>>> works fine. But since I pretty much only have images that are larger than
>>> 2GB, the Java machine has not enough memory to "load" the whole image
>>> plane. However this is not really what I want to do anyway. It seems that
>>> itk has a way to manage large files by opening them using streams. An
>>> image
>>> is split into pieces on read so that large images can be processes
>>> sequentially. I'm not sure but this functionality seems not to be
>>> implemented down to the wrapping bioformat reader for itk.
>>>
>>> itk::ERROR: BioFormatsImageIO(****0000000003815B60): BioFormatsImageIO:
>>>
>>> 'ITKBridgePipe read' exited abnormally. Exception in thread "main"
>>> loci.formats.FormatException: Image plane too large. Only 2GB of data can
>>> be extracted at one time.
>>>
>>> Looking at the error message, the Java wrapped code obviously wants to
>>> read in the whole image. The cropping seems to be done after the complete
>>> image is read.
>>>
>>> I also found this page using the error output.
>>> http://www.openmicroscopy.org/****site/support/faq/bio-**formats/**<http://www.openmicroscopy.org/**site/support/faq/bio-formats/**>
>>> i-see-an-outofmemory-or-****negativearraysize-error-**
>>> message-when-attempting-to-****open-an-svs-or-jpeg-2000-file.****
>>> -what-does-this-mean<http://**www.openmicroscopy.org/site/**
>>> support/faq/bio-formats/i-see-**an-outofmemory-or-**
>>> negativearraysize-error-**message-when-attempting-to-**
>>> open-an-svs-or-jpeg-2000-file.**-what-does-this-mean<http://www.openmicroscopy.org/site/support/faq/bio-formats/i-see-an-outofmemory-or-negativearraysize-error-message-when-attempting-to-open-an-svs-or-jpeg-2000-file.-what-does-this-mean>
>>> >
>>>
>>>
>>> I could use the option to increase the JVM memory space but this would
>>> only help me so far. With really large images that exceed my memory I
>>> would
>>> still have a problem. Is there a way to get around the complete read-in
>>> of
>>> the input with the itk reader that I've misses so far?
>>>
>>> Best Regards,
>>> Matthias
>>>
>>> Am 7/2/2012 7:25 PM, schrieb Curtis Rueden:
>>>
>>> Hi Matthias,
>>>>
>>>> I want to use the cpp wrapped bioformat reader with itk to read
>>>> ome.tif files. The interfacing with itk however is kind of not
>>>> working properly. First I have to unregister the base readers (jpg,
>>>> png etc. )using the command itk::ObjectFactoryBase::__****ReHash();
>>>> If
>>>>
>>>> I
>>>>
>>>> don't do this, one of base readers will try to load the tiff image.
>>>> Namely the TiffImageIO. Since the image in question is tiled, this
>>>> read operation will fail. And therfore the bioformat reader will
>>>> never be activated. If I use the ReHash() function, the bioformat
>>>> reading works, but all other reader/writers are unregistered in the
>>>> ObjectFactoryBase. In registering them manually for e.g. png saving
>>>> a output region of the read ome.tiff image. The bioformat reader
>>>> will always tell itk that he can write the input image. But he
>>>> sould
>>>> not, since I choose .png as the output format. I'm not quite shure
>>>> what I'm doing wrong concerning the reader. I thought the plugin
>>>> should behave as a normal itk reader/writer.
>>>>
>>>>
>>>> I am CCing the ITK-users list, since I think your question is a more
>>>> general one about the ITK ImageIO plugin mechanism. It seems to me that
>>>> all you really need is a simple way to reshuffle the order of the
>>>> available ImageIO plugins—in this case, to give the Bio-Formats ITK
>>>> plugin a higher priority than the built-in TIFF ImageIO plugin. Does
>>>> anyone know a good way to do this?
>>>>
>>>> Thanks,
>>>> Curtis
>>>>
>>>> P.S. The ImageIO plugin Matthias is talking about is here:
>>>> http://loci.wisc.edu/bio-****formats/itk<http://loci.wisc.edu/bio-**formats/itk>
>>>> <http://loci.wisc.**edu/bio-formats/itk<http://loci.wisc.edu/bio-formats/itk>
>>>> >
>>>>
>>>>
>>>> On Mon, Jul 2, 2012 at 11:09 AM, <matthias.noll at igd.fraunhofer.****de<
>>>> matthias.noll at igd.**fraunhofer.de <matthias.noll at igd.fraunhofer.de>>
>>>> <mailto:matthias.noll at igd.**fr**aunhofer.de <http://fraunhofer.de><
>>>> matthias.noll at igd.**fraunhofer.de <matthias.noll at igd.fraunhofer.de>>>>
>>>>
>>>> wrote:
>>>>
>>>> Matthias Noll sent a message using the contact form at
>>>> http://loci.wisc.edu/contact.
>>>>
>>>> Hello,
>>>> I want to use the cpp wrapped bioformat reader with itk to read
>>>> ome.tif files. The interfacing with itk however is kind of not
>>>> working properly. First I have to unregister the base readers (jpg,
>>>> png etc. )using the command itk::ObjectFactoryBase::__****ReHash();
>>>> If
>>>>
>>>> I
>>>>
>>>> don't do this, one of base readers will try to load the tiff image.
>>>> Namely the TiffImageIO. Since the image in question is tiled, this
>>>> read operation will fail. And therfore the bioformat reader will
>>>> never be activated. If I use the ReHash() function, the bioformat
>>>> reading works, but all other reader/writers are unregistered in the
>>>> ObjectFactoryBase. In registering them manually for e.g. png saving
>>>> a output region of the read ome.tiff image. The bioformat reader
>>>> will always tell itk that he can write the input image. But he
>>>> sould
>>>> not, since I choose .png as the output format. I'm not quite shure
>>>> what I'm doing wrong concerning the reader. I thought the plugin
>>>> should behave as a normal itk reader/writer.
>>>> I'm greatful for any kind of help.
>>>> Thanks Matthias
>>>>
>>>>
>>>>
>>>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-users/attachments/20120712/9bf8131a/attachment.html>
More information about the ome-users
mailing list