[ome-users] Error on max intensity for large volumes

[ingvar]

Hi Josh,

I would not expect a backport for something like this.  More that this 
is an incentive for me to move forward.

At some point users will start to create single slices approaching 2^31 
data points.  Think that is when you need to look into the direct memory 
management options...

Cheers,
Ingvar

On 2013-06-14 09:20, Josh Moore wrote:
> On Jun 13, 2013, at 10:04 PM, Josh Moore wrote:
> 
>> 
>> On Jun 12, 2013, at 1:07 PM, ingvar wrote:
>> 
>>> Dear OMERO,
>> 
>> Hi Ingvar,
> 
> Hi again,
> 
> 
>>> On an EM volume that contains more than 2^31 data points, I get an 
>>> error when attempting to do "Max Intensity" in the OMERO.web image 
>>> viewer.  The error message in OMEROweb.log and OMEROweb_request.log 
>>> indicate that the cause is an overflow in an integer index in Java.  
>>> Assuming that the index points into a Java array, then there may be a 
>>> bigger problem, as Java arrays in the standard Oracle implementation 
>>> is limited to MAX_INTEGER - 8 (or some other small offset depending 
>>> on who you believe).
>>> 
>>> There are several possible solutions here:
>>> direct memory management 
>>> (http://www.omsn.de/blog/big-arrays-in-java)
>>> use a 3rd party library such as fastutil 
>>> (http://fastutil.di.unimi.it)
>>> rewrite the code, I do not think the whole data array has to be in 
>>> memory for max intensity
>> 
>> It's unlikely we'd try any of the more advanced workarounds in the 
>> short-term, but certainly loading sections of the data should work.
>> 
>> 
>>> Steps to reproduce:
>>> Fetch the file 
>>> ftp://ftp.ebi.ac.uk/pub/databases/emdb/structures/EMD-2362/map/emd_2362.map.gz 
>>> (2.6 GB)
>>> Upload into OMERO
>>> In OMERO.web's image viewer select "Max Intensity"
>> 
>> Thanks for the steps and the data. I'll download and try to 
>> reproduce.
> 
> Against, 4.4.8 there is an exception. See
> https://trac.openmicroscopy.org.uk/ome/ticket/11126
> 
> The web client is left in the "Loading" state. It's highly unlikely a
> fix will be backported to 4.3.3, but we'll certainly look into fixing
> it for 4.4.
> 
> 
>>> I am on a pretty old version, 4.3.3, if I remember correctly, so if 
>>> this has been addressed since then, I guess this is the incentive for 
>>> me to upgrade.
>> 
>> Let's hope so!
>> 
>>> Kind Regards,
>>> Ingvar Lagerstedt
>> 
>> Cheers,
>> ~Josh
> 
> Thanks again,
> ~Josh.