[ome-devel] Import again...

[Stefan Frank]

Hi Curtis,

this is great news! It's my last day before my holidays, so if you
didn't find time for that, I can start writing the command-line-tool
based on the v3.00alpha3 when I'm back(mid-september). Do you have an
open cvs, so that we can make sure that we code against the head-revision?

But this already brings up another question: uploading images from
Visbio into OME is only one part of the story - it would also be
nice to get images out of OME into Visbio to do volume-rendering,
measurements etc. - or the other way round, having the Visbio-features
inside Shoola. Of course, it shouldn't be too hard to do to integrate
a project-browser into Visbio, but this is obviuosly a general pattern
we have encountered elsewhere: We are currently trying to stitch 
together different tools for Visualization(Shoola and Tikal and now 
Visbio), Annotation(Protege), and Workflow(Taverna) - in our project we 
need all the pieces, but every software is lacking some functionality, 
making it hard to build an integrated workflow (what the marketing-guys 
would call "seamless") - we've been discussing several architectures for 
an integration-platform on which we can throw all this together: Some of 
the projects already have some plug-in-features - unfortunately every 
project here has a different approach. We are obviously not the first 
people who want to integrate all these different pieces together, but 
maybe its this great, beautiful beast of a project we have(mitocheck), 
that makes this problem worse for us.

We've got terabytes of 5D mitosis videos on the way, and we're 
responsible for getting the infrastructure for storage and computation, 
as well as the coordination of the feature-extraction, classification, 
and annotation of these images up and running.

To handle all this we've got a got a stack of different clients to 
integrate:
- ome (of course...)
- shoola (For looking at pictures and organising projects)
- Tikal (has some of the algorithms we need)
- Taverna (handles the distributed workflows)
- Protege (for annotating the samples and handling the constraints
against existing and yet-to-develop ontologies)
- VisBio (Visualization)

Each of these programs do a good job at what they are build for - but
none of the programs can do the whole thing alone.

The question still remains: How can we make all this things work
together? How do we integrate all that? Do we have to develop the
integration platform on our own or can use one that's already existing? 
Are these common problems, that other people have encountered when 
working with OME? And if so, what were your solutions?

Sorry for this lengthy mail, but I think it's important, that we get
this "Integration"-Discussion started...

Cheers
Stefan



Curtis Rueden wrote:
> Hi Stefan,
> 
> Regarding the "semantic_type" bug, I reported it a while ago and Doug 
> has already fixed it. There is currently another TestImport issue 
> currently under investigation, which will hopefully be cleared up soon.
> 
> I am nearing completion of VisBio v3.00alpha3. The timing seems good 
> because I'll probably end up releasing near the same time as OME v2.2.1. 
> My hope is that these two releases will play nicely together again (with 
> respect to client-side import).
> 
> As for doing command-line client-side imports, VisBio3's code is pretty 
> clean for the most part, and writing a small command-line Java 
> application to do client-side upload shouldn't be too hard. The file 
> pattern detection logic is completely modular 
> (loci.visbio.data.FilePattern), as is the data object infrastructure 
> (loci.visbio.data.Dataset). The only thing that's not as of alpha2 is 
> the actual OME upload code (loci.visbio.ome.OMEManager). But I'll clean 
> it up for alpha3, providing an OME upload method for use with external 
> applications (if I have time, I may even code a simple command-line 
> client-side OME upload tool and include it with the alpha3 release).
> 
> The goal from the beginning has been for VisBio to function not only as 
> an end-user application, but also as a Java visualization toolkit for 
> multidimensional biological image data. Please let me know if you have 
> any comments, questions or suggestions.
> 
> Cheers,
> Curtis
> 
> Stefan Frank wrote:
> 
>> Hi folks,
>>
>> we're still struggling with the import of images and are pursuing 
>> different paths for importing files - apart from hacking the 
>> perl-import-scripts(glad to have put this on josh's list, not mine), I 
>> went through the different approaches using the XML-RPC from java: As 
>> far as I can see, we have several duplicate Codebases here that do 
>> (part of) the Job: We have the TestImport from OME-JAVA, there is the 
>> code in Visbio3 that started out with copying TestImport-Code and I 
>> even remember to have seen first Steps towards an import from inside 
>> Shoola.
>>
>> On the 
>> docs-site(http://www.openmicroscopy.org/system-admin/import.html), 
>> there is already a marriage between visbio and the Import-Module 
>> announced - what is the state of this marriage? VisBio is pretty cool 
>> on the import- side, with a clever way of determining the files that 
>> belong together, however, we need a Command-Line-Interface, preferably 
>> tweakable via parameters to set the z/time/channel-patterns. We'd like 
>> to put this into a cron-job and upload files from a given directory 
>> over night. Or is this already possible with Visbio3?!
>>
>> Is anybody already doing this?! I do not want to add another 
>> code-base, but I think refactoring the TestImport.java from OME-JAVA, 
>> adding cli to it,  and then taking this together with the 
>> VisBio-Classes would do the trick - maybe we can come up with classes 
>> that can be used inside Shoola for the import, too?!
>>
>> cheers
>> stefan
> 
> 
> 

-- 
Stefan Frank
iBioS - Intelligent BioInformatics Systems
http://www.dkfz-heidelberg.de/ibios
DKFZ - German Cancer Research Center
Im Neuenheimer Feld 580
69120 Heidelberg
Tel.: +49 (0) 6221 42-3612
Mail: s.frank at dkfz-heidelberg.de