<div dir="ltr">Hi Melissa & everyone,<br><div class="gmail_extra"><br></div><div class="gmail_extra">Thank you for replying to my suggestion. I appreciate the discussion.</div><div class="gmail_extra"><br></div><div class="gmail_extra">
<div class="gmail_extra">> Note that we've previously (and relatively recently) put quite a bit<br></div><div><div>> of effort into getting the OME-XML code and specification into</div><div>> bioformats.git.</div>
</div><div><br></div><div>Yes, you might recall that I am the one who did most of that work. ;-) [1]</div><div><br></div><div>And I would be willing to do it again, if it meant a cleaner build system. (Though splitting subtrees is much less complicated.)</div>
<div class="gmail_extra"><br></div><div class="gmail_extra"><div class="gmail_extra"><div class="gmail_extra">> If we were to follow this proposal, and unless I misunderstand, then</div><div class="gmail_extra">> making one simple change in the OME-XML schema (e.g. changing a single</div>
<div class="gmail_extra">> field type) would require:</div><div><br></div><div>Well, I think specification and ome-xml should live in the same repository.</div><div><br></div><div>So it would require:</div>
<div>- filing a PR into the specification repo (ome-xml.git in my proposal)</div><div>- cutting a new ome-xml release artifact</div><div>- filing a PR into bioformats.git to update the version of ome-xml</div>
<div><br></div><div>So, two PRs instead of one. (As Mark said, scifio.git doesn't depend on ome-xml.) But the schema and ome-xml code doesn't change nearly as often as Bio-Formats does, of course.</div>
<div><br></div><div>Still, I can appreciate how two PRs might be undesirable (believe me; you know how much I hate the develop/dev_4_4 split :-). To avoid that, you could keep specification & ome-xml & bio-formats in the same git repository as they are now, while still using separate versioning per component. This would provide on target for PRs, while making it much simpler for most people to build Bio-Formats thanks to remote release dependency resolution of Maven/Ivy/etc.</div>
<div><br></div><div><div>> Our policy so far has been to release everything at once; I think it<br></div><div><div>> would make more sense to agree first whether that should be changed,</div><div>> and then consider solutions. I personally do favor having everything</div>
<div>> be released at once, as it makes keeping track of version numbers</div><div>> (mentally and when supporting users) much easier.</div><div><br></div><div>Indeed, my proposal hinges on the idea that everything should actually not be released at once -- that many of these are separate projects, which are A) useful in their own right, outside an OMERO-specific context; and B) developed at a much different pace than one another, resulting in gratuitous/unnecessary/confusing releases when all continue to be versioned together. [2]</div>
<div><br></div><div><div>> I understand the desire for smarter autogeneration, but I think that</div><div>> would be much better accomplished within our existing build systems,</div><div>> rather than fragmenting the codebase.</div>
<div><br></div><div>I do not see it as one codebase, but rather several projects which all currently happen to be lumped into one repository with a single release schedule imposed upon them. Right now, all of these project releases are dictated by the OMERO release schedule [3]. The OME team is ramping up for the OMERO 5 release, and presumably a Bio-Formats 5 release will go along with that. But what is really so radically new in Bio-Formats 5? Nothing, I would argue. Now, e.g., an extensible SCIFIO-based Bio-Formats would justify a 5.0.0 release. But instead, I am sad to see Bio-Formats versions that have little semantic meaning [4] with respect to Bio-Formats itself, all because of the simultaneous top-down release schedule driven by the OMERO project.</div>
<div><br></div><div>Instead, my proposal emphasizes the individual projects as useful to the community and the world in their own right. For example, MDB Tools Java is not available anywhere else to my knowledge [5], since we rescued it from an obscure forum post. Wouldn't it be a wonderful service to the community to post it to its own Git repository on GitHub, so that it is easy to find, and the greater community (beyond just those interested in OME) might contribute back?</div>
<div><br></div><div>I understand I am proposing a substantial change. It is certainly about more than just improving the code generation process. That's why I changed the subject. And I understand the reluctance to change anything that might disrupt current development processes. But in this case, I think the change would be well worth it to make the OME projects into even better members of the global FOSS community.</div>
<div><br></div><div></div></div><div>Regards,</div><div>Curtis</div></div></div></div><div><br></div><div>[1] <a href="http://trac.openmicroscopy.org.uk/ome/ticket/10435#comment:8" target="_blank">http://trac.openmicroscopy.org.uk/ome/ticket/10435#comment:8</a></div>
</div><div class="gmail_extra"><br></div><div class="gmail_extra" style>[2] One quick example of how the gratuitous releases cause problems: whenever I update Bio-Formats using the ImageJ updater, it notices that several of the JARs actually have no functional changes, and does not alter the version number of the JAR files. Yes, we could potentially fix this in the updater, but wouldn't it make more sense to simply not make vacuous releases in the first place?</div>
<div class="gmail_extra"><br></div><div class="gmail_extra" style>[3] Note that we are not even that consistent, since there are several other projects we develop like native-lib-loader which are *not* synchronized with the OMERO/Bio-Formats version. This works just fine, since we treat those projects as external dependencies. The same thing would work fine for the components/forks, components/stubs and ome-xml.</div>
<div class="gmail_extra"><br></div><div class="gmail_extra" style>[4] ImageJ2 and SCIFIO are now using SemVer, which conveys useful information in the version number: <a href="http://semver.org/">http://semver.org/</a></div>
<div class="gmail_extra"><br></div><div class="gmail_extra">[5] Well, potentially this: <a href="http://jackcess.sourceforge.net/" target="_blank">http://jackcess.sourceforge.net/</a></div><div class="gmail_extra"><br></div>
<div class="gmail_extra"><br></div><div class="gmail_quote">
On Tue, Jun 11, 2013 at 12:52 PM, Melissa Linkert <span dir="ltr"><<a href="mailto:melissa@glencoesoftware.com" target="_blank">melissa@glencoesoftware.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Hi Curtis,<br>
<div><br>
> That said, I think Bio-Formats would greatly benefit from substantial<br>
> modularization of components. We are realizing this with SCIFIO, and I<br>
> think it applies to the OME-XML component as well.<br>
><br>
> Below, I will lay out what I think is a better structure for the build<br>
> system, which would result in more advantages and less pain than with the<br>
> current structure.<br>
</div>...<br>
<div>> MetadataStore, MetadataRetrieve, etc., would move to the ome-xml component,<br>
> keeping all generated code together.<br>
><br>
> One Git repository for each of:<br>
><br>
> - SCIFIO (<a href="https://github.com/scifio/scifio" target="_blank">https://github.com/scifio/scifio</a>)<br>
> - OME-XML (<a href="https://github.com/openmicroscopy/ome-xml" target="_blank">https://github.com/openmicroscopy/ome-xml</a>)<br>
> - Bio-Formats (<a href="https://github.com/openmicroscopy/bioformats" target="_blank">https://github.com/openmicroscopy/bioformats</a>)<br>
> - Fork: Apache POI (<a href="https://github.com/openmicroscopy/ome-poi" target="_blank">https://github.com/openmicroscopy/ome-poi</a>)<br>
> -- change package prefix to avoid third party code collisions<br>
> - Fork: MDB Tools Java (<a href="https://github.com/openmicroscopy/ome-mdb-tools" target="_blank">https://github.com/openmicroscopy/ome-mdb-tools</a>)<br>
> -- change package prefix to avoid third party code collisions<br>
> - Fork: JAI Image I/O (<a href="https://github.com/scifio/scifio-jai-image-io" target="_blank">https://github.com/scifio/scifio-jai-image-io</a>)<br>
> -- change package prefix to avoid third party code collisions<br>
> - Stub: LWF (<a href="https://github.com/scifio/lwf-stubs" target="_blank">https://github.com/scifio/lwf-stubs</a>)<br>
<br>
</div>What you and Mark do for your work on SCIFIO is up to you, but I would<br>
be extremely hesitant to do something like this for Bio-Formats itself.<br>
Spreading one codebase across 7 different repositories is at best<br>
invasive, and would have a substantial impact upon anyone who routinely<br>
works on Bio-Formats.<br>
<div><br>
> In other words, OME-XML gets its own Git repository, which includes all the<br>
> code generated code. Each fork and stub also has its own repository in the<br>
> relevant namespace.<br>
<br>
</div>Note that we've previously (and relatively recently) put quite a bit of<br>
effort into getting the OME-XML code and specification into bioformats.git.<br>
<div><br>
> Dependencies between repositories would be done by release version<br>
> coupling. For Maven projects (i.e., SCIFIO), simply making releases and<br>
> using release dependencies would be sufficient to facilitate repeatable<br>
> builds. For Ant-based projects (i.e., stuff in openmicroscopy namespace),<br>
> release JARs would continue to be committed to the repository as they are<br>
> now, or they could be resolved remotely via Ivy or similar.<br>
<br>
</div>I don't agree that doing that makes things easier. If we were to follow<br>
this proposal, and unless I misunderstand, then making one simple change in<br>
the OME-XML schema (e.g. changing a single field type) would require:<br>
<br>
- a pull request into whichever repository houses the specification<br>
(currently bioformats.git)<br>
- creation of "release" artifacts from whichever repository houses the<br>
specification<br>
- a pull request into ome-xml.git (to update the autogenerated code)<br>
- creation of release JARs from ome-xml.git<br>
- a pull request into scifio.git (to update SCIFIO readers)<br>
- creation of release JARs from scifio.git<br>
- a pull request into bioformats.git (to update Bio-Formats readers)<br>
<br>
...instead of what we have now, which is a single pull request into<br>
bioformats.git.<br>
<div><br>
> This would making building Bio-Formats much simpler and faster. As Roger<br>
> pointed out, we do not really need to code generate the OME-XML stuff on<br>
> every build, but rather only when the schema changes. Of course, the<br>
> OME-XML component contains other code which would be subject to change<br>
> between schema releases, but that's fine.<br>
<br>
</div>I understand the desire for smarter autogeneration, but I think that<br>
would be much better accomplished within our existing build systems,<br>
rather than fragmenting the codebase.<br>
<div><br>
> This more modular structure would also facilitate these components being<br>
> developed on separate release cycles. The forks and stubs rarely change and<br>
> do not need to be released with every OME release. And the OME-XML project<br>
> could be released along side schema changes (i.e., twice a year) rather<br>
> than with every OME release.<br>
<br>
</div>Our policy so far has been to release everything at once; I think it<br>
would make more sense to agree first whether that should be changed, and<br>
then consider solutions. I personally do favor having everything be<br>
released at once, as it makes keeping track of version numbers (mentally<br>
and when supporting users) much easier.<br>
<br>
Again, what you do with respect to <a href="http://github.com/scifio/scifio" target="_blank">http://github.com/scifio/scifio</a> is up<br>
to you. Doing this for Bio-Formats itself would have a non-trivial impact<br>
on every single OME team member and a large portion of the developer community,<br>
and as such I think it would be better to consider other options for<br>
making autogeneration easier.<br>
<br>
Regards,<br>
-Melissa<br>
<div><div><br>
On Mon, Jun 10, 2013 at 10:50:25AM -0500, Curtis Rueden wrote:<br>
> Hi Roger & everyone,<br>
><br>
> Sorry for the delay in reply. After spending the last couple of weeks on<br>
> ImageJ build issues related to native code components (specifically, the<br>
> ImageJ launcher in C), I have some new perspective on the new code<br>
> generation of the Bio-Formats build system.<br>
><br>
> First of all, I want to say thanks to Roger for solving the build for both<br>
> Ant and Maven. I know maintaining the dual build systems can be substantial<br>
> extra work. But I think the Maven system has many advantages, so I am happy<br>
> it is being maintained.<br>
><br>
> That said, I think Bio-Formats would greatly benefit from substantial<br>
> modularization of components. We are realizing this with SCIFIO, and I<br>
> think it applies to the OME-XML component as well.<br>
><br>
> Below, I will lay out what I think is a better structure for the build<br>
> system, which would result in more advantages and less pain than with the<br>
> current structure.<br>
><br>
> > One thing which might be an issue is that while xsd-fu generates the<br>
> > ome-xml model code, which could potentially be downloaded, it also<br>
> > generates all the MetadataStore, MetadateRetrieve and all the other<br>
> > Metadata-related classes in scifio, including OMEXMLMetadataImpl.<br>
> > Given that these are paired with the generated model code, generating<br>
> > one and downloading the other may result in breakage on model changes,<br>
> > or changes in xsd-fu or the templates which change the generated code.<br>
><br>
> MetadataStore, MetadataRetrieve, etc., would move to the ome-xml component,<br>
> keeping all generated code together.<br>
><br>
> One Git repository for each of:<br>
><br>
> - SCIFIO (<a href="https://github.com/scifio/scifio" target="_blank">https://github.com/scifio/scifio</a>)<br>
> - OME-XML (<a href="https://github.com/openmicroscopy/ome-xml" target="_blank">https://github.com/openmicroscopy/ome-xml</a>)<br>
> - Bio-Formats (<a href="https://github.com/openmicroscopy/bioformats" target="_blank">https://github.com/openmicroscopy/bioformats</a>)<br>
> - Fork: Apache POI (<a href="https://github.com/openmicroscopy/ome-poi" target="_blank">https://github.com/openmicroscopy/ome-poi</a>)<br>
> -- change package prefix to avoid third party code collisions<br>
> - Fork: MDB Tools Java (<a href="https://github.com/openmicroscopy/ome-mdb-tools" target="_blank">https://github.com/openmicroscopy/ome-mdb-tools</a>)<br>
> -- change package prefix to avoid third party code collisions<br>
> - Fork: JAI Image I/O (<a href="https://github.com/scifio/scifio-jai-image-io" target="_blank">https://github.com/scifio/scifio-jai-image-io</a>)<br>
> -- change package prefix to avoid third party code collisions<br>
> - Stub: LWF (<a href="https://github.com/scifio/lwf-stubs" target="_blank">https://github.com/scifio/lwf-stubs</a>)<br>
><br>
> In other words, OME-XML gets its own Git repository, which includes all the<br>
> code generated code. Each fork and stub also has its own repository in the<br>
> relevant namespace.<br>
><br>
> Dependencies between repositories would be done by release version<br>
> coupling. For Maven projects (i.e., SCIFIO), simply making releases and<br>
> using release dependencies would be sufficient to facilitate repeatable<br>
> builds. For Ant-based projects (i.e., stuff in openmicroscopy namespace),<br>
> release JARs would continue to be committed to the repository as they are<br>
> now, or they could be resolved remotely via Ivy or similar.<br>
><br>
> This would making building Bio-Formats much simpler and faster. As Roger<br>
> pointed out, we do not really need to code generate the OME-XML stuff on<br>
> every build, but rather only when the schema changes. Of course, the<br>
> OME-XML component contains other code which would be subject to change<br>
> between schema releases, but that's fine.<br>
><br>
> This more modular structure would also facilitate these components being<br>
> developed on separate release cycles. The forks and stubs rarely change and<br>
> do not need to be released with every OME release. And the OME-XML project<br>
> could be released along side schema changes (i.e., twice a year) rather<br>
> than with every OME release.<br>
><br>
> Comments welcome.<br>
><br>
> Regards,<br>
> Curtis<br>
><br>
><br>
> On Thu, May 2, 2013 at 12:25 PM, Roger Leigh <<a href="mailto:r.leigh@dundee.ac.uk" target="_blank">r.leigh@dundee.ac.uk</a>> wrote:<br>
><br>
> > On 02/05/2013 16:52, Curtis Rueden wrote:<br>
> ><br>
> > > If so, the build is completely identical--the sources which get<br>
> >> > generated on the fly from the templates by xsd-fu are identical bar a<br>
> >> > few lines comments in the top boilerplate.<br>
> >><br>
> >> OK, good to know.<br>
> >><br>
> >> One more question/concern: presumably, the Bio-Formats build no longer<br>
> >> functions on Windows, due to the Python + Genshi dependency. With the<br>
> >> Ant build, this might be non-trivial to solve. But solving the issue<br>
> >> with Maven is very straightforward: include the "ome-xml" module in the<br>
> >> reactor only within a profile. Then, when that profile is not enabled,<br>
> >> Maven will resolve the ome-xml dependency from the remote repository<br>
> >> rather than regenerating and rebuilding the code. This would eliminate<br>
> >> the need to install Genshi, and make it easier to build on Windows<br>
> >> again. What do you think?<br>
> >><br>
> ><br>
> > I'm afraid I'm no authority on Maven, so I'm not sure. Maybe Melissa or<br>
> > Josh have a better take on this than me. I assume that this will work<br>
> > correctly on Windows if python is installed?<br>
> ><br>
> > One thing which might be an issue is that while xsd-fu generates the<br>
> > ome-xml model code, which could potentially be downloaded, it also<br>
> > generates all the MetadataStore, MetadateRetrieve and all the other<br>
> > Metadata-related classes in scifio, including OMEXMLMetadataImpl. Given<br>
> > that these are paired with the generated model code, generating one and<br>
> > downloading the other may result in breakage on model changes,<br>
> > or changes in xsd-fu or the templates which change the generated code.<br>
> ><br>
> > While it's not all enabled yet, I'd like to have the model selectable as<br>
> > an ant properly (it's xsdfu.schemaver), so that it's possible to change<br>
> > to a different model when building. There's currently some hardcoded<br>
> > "2012-06" versions which need to be switched to change to use the<br>
> > property value.<br>
> ><br>
> ><br>
> > Regards,<br>
> > Roger<br>
> ><br>
> > --<br>
> > Dr Roger Leigh -- Open Microscopy Environment<br>
> > Wellcome Trust Centre for Gene Regulation and Expression,<br>
> > College of Life Sciences, University of Dundee, Dow Street,<br>
> > Dundee DD1 5EH Scotland UK Tel: (01382) 386364<br>
> ><br>
> ><br>
> > The University of Dundee is a registered Scottish Charity, No: SC015096<br>
> ><br>
</div></div>> > ______________________________**_________________<br>
> > ome-devel mailing list<br>
> > ome-devel@lists.**<a href="http://openmicroscopy.org.uk" target="_blank">openmicroscopy.org.uk</a><<a href="mailto:ome-devel@lists.openmicroscopy.org.uk" target="_blank">ome-devel@lists.openmicroscopy.org.uk</a>><br>
> > <a href="http://lists.openmicroscopy." target="_blank">http://lists.openmicroscopy.</a>**<a href="http://org.uk/mailman/listinfo/ome-**devel" target="_blank">org.uk/mailman/listinfo/ome-**devel</a><<a href="http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel" target="_blank">http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel</a>><br>
<div><div>> ><br>
<br>
> _______________________________________________<br>
> ome-devel mailing list<br>
> <a href="mailto:ome-devel@lists.openmicroscopy.org.uk" target="_blank">ome-devel@lists.openmicroscopy.org.uk</a><br>
> <a href="http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel" target="_blank">http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel</a><br>
<br>
</div></div></blockquote></div><br></div></div>