[ome-devel] Bio-Formats build system and repository structure [was: Re: Error building Bio-Formats develop]

Mon Jun 10 17:24:59 BST 2013

Hi all,

 Thanks for sending this out, Curtis.

 If anyone's interested, I filed this as a github
issue<https://github.com/scifio/scifio/issues/3>under the SCIFIO
repo... so you can post comments there as well.

- Mark

On Mon, Jun 10, 2013 at 10:50 AM, Curtis Rueden <ctrueden at wisc.edu> wrote:

> Hi Roger & everyone,
>
> Sorry for the delay in reply. After spending the last couple of weeks on
> ImageJ build issues related to native code components (specifically, the
> ImageJ launcher in C), I have some new perspective on the new code
> generation of the Bio-Formats build system.
>
> First of all, I want to say thanks to Roger for solving the build for both
> Ant and Maven. I know maintaining the dual build systems can be substantial
> extra work. But I think the Maven system has many advantages, so I am happy
> it is being maintained.
>
> That said, I think Bio-Formats would greatly benefit from substantial
> modularization of components. We are realizing this with SCIFIO, and I
> think it applies to the OME-XML component as well.
>
> Below, I will lay out what I think is a better structure for the build
> system, which would result in more advantages and less pain than with the
> current structure.
>
> > One thing which might be an issue is that while xsd-fu generates the
> > ome-xml model code, which could potentially be downloaded, it also
> > generates all the MetadataStore, MetadateRetrieve and all the other
> > Metadata-related classes in scifio, including OMEXMLMetadataImpl.
> > Given that these are paired with the generated model code, generating
> > one and downloading the other may result in breakage on model changes,
> > or changes in xsd-fu or the templates which change the generated code.
>
> MetadataStore, MetadataRetrieve, etc., would move to the ome-xml
> component, keeping all generated code together.
>
> One Git repository for each of:
>
> - SCIFIO (https://github.com/scifio/scifio)
> - OME-XML (https://github.com/openmicroscopy/ome-xml)
> - Bio-Formats (https://github.com/openmicroscopy/bioformats)
> - Fork: Apache POI (https://github.com/openmicroscopy/ome-poi)
>  -- change package prefix to avoid third party code collisions
> - Fork: MDB Tools Java (https://github.com/openmicroscopy/ome-mdb-tools)
>  -- change package prefix to avoid third party code collisions
> - Fork: JAI Image I/O (https://github.com/scifio/scifio-jai-image-io)
>   -- change package prefix to avoid third party code collisions
> - Stub: LWF (https://github.com/scifio/lwf-stubs)
>
> In other words, OME-XML gets its own Git repository, which includes all
> the code generated code. Each fork and stub also has its own repository in
> the relevant namespace.
>
> Dependencies between repositories would be done by release version
> coupling. For Maven projects (i.e., SCIFIO), simply making releases and
> using release dependencies would be sufficient to facilitate repeatable
> builds. For Ant-based projects (i.e., stuff in openmicroscopy namespace),
> release JARs would continue to be committed to the repository as they are
> now, or they could be resolved remotely via Ivy or similar.
>
> This would making building Bio-Formats much simpler and faster. As Roger
> pointed out, we do not really need to code generate the OME-XML stuff on
> every build, but rather only when the schema changes. Of course, the
> OME-XML component contains other code which would be subject to change
> between schema releases, but that's fine.
>
> This more modular structure would also facilitate these components being
> developed on separate release cycles. The forks and stubs rarely change and
> do not need to be released with every OME release. And the OME-XML project
> could be released along side schema changes (i.e., twice a year) rather
> than with every OME release.
>
> Comments welcome.
>
> Regards,
> Curtis
>
>
> On Thu, May 2, 2013 at 12:25 PM, Roger Leigh <r.leigh at dundee.ac.uk> wrote:
>
>> On 02/05/2013 16:52, Curtis Rueden wrote:
>>
>>   > If so, the build is completely identical--the sources which get
>>>  > generated on the fly from the templates by xsd-fu are identical bar a
>>>  > few lines comments in the top  boilerplate.
>>>
>>> OK, good to know.
>>>
>>> One more question/concern: presumably, the Bio-Formats build no longer
>>> functions on Windows, due to the Python + Genshi dependency. With the
>>> Ant build, this might be non-trivial to solve. But solving the issue
>>> with Maven is very straightforward: include the "ome-xml" module in the
>>> reactor only within a profile. Then, when that profile is not enabled,
>>> Maven will resolve the ome-xml dependency from the remote repository
>>> rather than regenerating and rebuilding the code. This would eliminate
>>> the need to install Genshi, and make it easier to build on Windows
>>> again. What do you think?
>>>
>>
>> I'm afraid I'm no authority on Maven, so I'm not sure.  Maybe Melissa or
>> Josh have a better take on this than me.  I assume that this will work
>> correctly on Windows if python is installed?
>>
>> One thing which might be an issue is that while xsd-fu generates the
>> ome-xml model code, which could potentially be downloaded, it also
>> generates all the MetadataStore, MetadateRetrieve and all the other
>> Metadata-related classes in scifio, including OMEXMLMetadataImpl.  Given
>> that these are paired with the generated model code, generating one and
>> downloading the other may result in breakage on model changes,
>> or changes in xsd-fu or the templates which change the generated code.
>>
>> While it's not all enabled yet, I'd like to have the model selectable as
>> an ant properly (it's xsdfu.schemaver), so that it's possible to change
>> to a different model when building.  There's currently some hardcoded
>> "2012-06" versions which need to be switched to change to use the
>> property value.
>>
>>
>> Regards,
>> Roger
>>
>> --
>> Dr Roger Leigh -- Open Microscopy Environment
>> Wellcome Trust Centre for Gene Regulation and Expression,
>> College of Life Sciences, University of Dundee, Dow Street,
>> Dundee DD1 5EH Scotland UK   Tel: (01382) 386364
>>
>>
>> The University of Dundee is a registered Scottish Charity, No: SC015096
>>
>> ______________________________**_________________
>> ome-devel mailing list
>> ome-devel at lists.**openmicroscopy.org.uk<ome-devel at lists.openmicroscopy.org.uk>
>> http://lists.openmicroscopy.**org.uk/mailman/listinfo/ome-**devel<http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel>
>>
>
>
> _______________________________________________
> ome-devel mailing list
> ome-devel at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-devel/attachments/20130610/0ab5cb33/attachment-0001.html>