[ome-users] Can somebody explain the pyramid building concept to me?

[Josh Moore]

Hi Jake & Luke,

>> I personally think the server has hung, but I can't be sure. We can't see
>> anything "wrong" on the client side, because it does progress - but surely
>> it should not take a day to ingest a 9.35GB image?
> 
> Agreed. 

Looks like it'll take even longer.


>> Is there a place I can send you one of these files, so you could try and
>> replicate the problem yourself? I have numerous ways I can send you a big
>> file..
> 
> I'll send you connection information off-list, but my guess is still that the main process is struggling. There's not a OutOfMemory exception in the log you sent me, which would point to Blitz-0 needing more memory (though I didn't have all the logs). But the issues with the postgresql connection point to perhaps running into SWAP.
> 
> When uploading your files, you might also upload a tarball of var/log.

Thanks for uploading the file. As I mentioned, I'm currently importing it locally and I haven't yet seen any signs of a memory leak or similar. I've using the command-line importer with the arguments:

  bin/omero import -- --debug=ALL *.ims

so that I can see the upload times for each tile. At the moment, only about 4 tiles are going in per second. Since the largest resolution (0) is 165K x 63K, this leads to an almost 2 day import. Though it won't be possible to significantly increase this for 4.4.4, we'd be very interested in working with you to bring these times down if you have time to try any of the following out:

 * Change the OMERO tile size. One problem is likely that the IMS file uses 128x128 tiles while OMERO is configured by default ufor 256x256. You could try changing the OMERO default and see if the import times improve. Rendering times may be reduced.

 * Resize the file (very adventurous). There are tools for repacking an HDF5 file which would allow you to change the IMS chunk size. You should only attempt this on a copy, since it's possible this would break the original software.

 * (If you're willing to build your own server from source) Enable "fs-lite" for the IMS file so that it's simply transferred to the server and read on the fly. The underlying performance issues of netcdf.jar will still come into play, but may be more manageable. An example of enabling a file format for fs-lite is available under [1].


Longer-term we plan on reworking the HDF5-based readers and/or the HDF5-library we use. Once this is available, you'd of course be welcome to try that out as well.

Cheers,
~Josh

[1] https://github.com/melissalinkert/openmicroscopy/commit/fc8e8b4de8e38268b2dfa749422082b10732124b