[ome-users] Can somebody explain the pyramid building concept to me?

[Jake Carroll]

Hi Josh,



On 20/09/12 4:46 AM, "Josh Moore" <josh at glencoesoftware.com> wrote:

>Hi Jake & Luke,
>
>>> I personally think the server has hung, but I can't be sure. We can't
>>>see
>>> anything "wrong" on the client side, because it does progress - but
>>>surely
>>> it should not take a day to ingest a 9.35GB image?
>> 
>> Agreed. 
>
>Looks like it'll take even longer.
>
>
>>> Is there a place I can send you one of these files, so you could try
>>>and
>>> replicate the problem yourself? I have numerous ways I can send you a
>>>big
>>> file..
>> 
>> I'll send you connection information off-list, but my guess is still
>>that the main process is struggling. There's not a OutOfMemory exception
>>in the log you sent me, which would point to Blitz-0 needing more memory
>>(though I didn't have all the logs). But the issues with the postgresql
>>connection point to perhaps running into SWAP.
>> 
>> When uploading your files, you might also upload a tarball of var/log.
>
>Thanks for uploading the file. As I mentioned, I'm currently importing it
>locally and I haven't yet seen any signs of a memory leak or similar.
>I've using the command-line importer with the arguments:
>
>  bin/omero import -- --debug=ALL *.ims
>
>so that I can see the upload times for each tile. At the moment, only
>about 4 tiles are going in per second. Since the largest resolution (0)
>is 165K x 63K, this leads to an almost 2 day import.


OK. Cool. So we're not insane. Didn't think I'd forgotten how to set up
virtualised high performance imaging processing boxes ;). It's good to
know what you see is what we see.


>Though it won't be possible to significantly increase this for 4.4.4,
>we'd be very interested in working with you to bring these times down if
>you have time to try any of the following out:

Yep. We're indeed keen on this. Basically, we want to try and use OMERO
for our entire institutional imaging curation needs. It's high time the
organisation organised, catalog-induced, curated and sorted it's gigantic
imaging-data glut out - and we really do believe OMERO is the answer. We
just need it to be strong/gutsy enough to deal with the big end of town
stuff like the Imaris Bitplane IMS format, which is traditionally massive,
as you can see.

>
> * Change the OMERO tile size. One problem is likely that the IMS file
>uses 128x128 tiles while OMERO is configured by default ufor 256x256. You
>could try changing the OMERO default and see if the import times improve.
>Rendering times may be reduced.

OK. Points to documentation where I do this? Or do I need to recompile
from source to achieve?

>
> * Resize the file (very adventurous). There are tools for repacking an
>HDF5 file which would allow you to change the IMS chunk size. You should
>only attempt this on a copy, since it's possible this would break the
>original software.

Hrrrm yeah. That's sort of a last ditch effort.

>
> * (If you're willing to build your own server from source) Enable
>"fs-lite" for the IMS file so that it's simply transferred to the server
>and read on the fly. The underlying performance issues of netcdf.jar will
>still come into play, but may be more manageable. An example of enabling
>a file format for fs-lite is available under [1].

Yep. Am keen. I'll get looking around.

>
>
>Longer-term we plan on reworking the HDF5-based readers and/or the
>HDF5-library we use. Once this is available, you'd of course be welcome
>to try that out as well.

Sounds good.

I'll start building a "Dev" OMERO server shortly. We'll be in touch. I'm
sure I'll have questions about the sourceŠ

Cheers.

--JC

>
>Cheers,
>~Josh
>
>[1] 
>https://github.com/melissalinkert/openmicroscopy/commit/fc8e8b4de8e38268b2
>dfa749422082b10732124b