<div dir="ltr">Hi Melissa,<div><br></div><div><div>> I have opened a story to investigate general build system and</div><div>> versioning changes and potentially adding e.g. mdbtools.git to</div><div>> <a href="http://github.com/openmicroscopy">github.com/openmicroscopy</a>:</div>
<div>> </div><div>> <a href="http://trac.openmicroscopy.org.uk/ome/ticket/11228">http://trac.openmicroscopy.org.uk/ome/ticket/11228</a></div></div><div><br></div><div style>Thank you for creating these tickets! I have written a few comments. For anyone else interested: any further discussion on these issues can be found by following that link.</div>
<div style><br></div><div style>Regards,</div><div style>Curtis</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Jul 4, 2013 at 12:06 PM, Melissa Linkert <span dir="ltr"><<a href="mailto:melissa@glencoesoftware.com" target="_blank">melissa@glencoesoftware.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Curtis,<br>
<div class="im"><br>
> I do not see it as one codebase, but rather several projects which all<br>
> currently happen to be lumped into one repository with a single release<br>
> schedule imposed upon them. Right now, all of these project releases are<br>
> dictated by the OMERO release schedule [3]. The OME team is ramping up for<br>
> the OMERO 5 release, and presumably a Bio-Formats 5 release will go along<br>
> with that. But what is really so radically new in Bio-Formats 5? Nothing, I<br>
> would argue. Now, e.g., an extensible SCIFIO-based Bio-Formats would<br>
> justify a 5.0.0 release. But instead, I am sad to see Bio-Formats versions<br>
> that have little semantic meaning [4] with respect to Bio-Formats itself,<br>
> all because of the simultaneous top-down release schedule driven by the<br>
> OMERO project.<br>
><br>
> Instead, my proposal emphasizes the individual projects as useful to the<br>
> community and the world in their own right. For example, MDB Tools Java is<br>
> not available anywhere else to my knowledge [5], since we rescued it from<br>
> an obscure forum post. Wouldn't it be a wonderful service to the community<br>
> to post it to its own Git repository on GitHub, so that it is easy to find,<br>
> and the greater community (beyond just those interested in OME) might<br>
> contribute back?<br>
><br>
> I understand I am proposing a substantial change. It is certainly about<br>
> more than just improving the code generation process. That's why I changed<br>
> the subject. And I understand the reluctance to change anything that might<br>
> disrupt current development processes. But in this case, I think the change<br>
> would be well worth it to make the OME projects into even better members of<br>
> the global FOSS community.<br>
<br>
</div>I can see the value in doing this for some of the components (such as<br>
components/forks/*), but doing this for every single component would have<br>
a negative impact on the workflow of the OME team without a clear benefit.<br>
<br>
I have opened a story to investigate general build system and versioning<br>
changes and potentially adding e.g. mdbtools.git to <a href="http://github.com/openmicroscopy" target="_blank">github.com/openmicroscopy</a>:<br>
<br>
<a href="http://trac.openmicroscopy.org.uk/ome/ticket/11228" target="_blank">http://trac.openmicroscopy.org.uk/ome/ticket/11228</a><br>
<br>
All of the tasks under that story are fairly significant undertakings<br>
though, and as such I have not yet assigned a milestone.<br>
<br>
Regards,<br>
-Melissa<br>
<div class="HOEnZb"><div class="h5"><br>
On Fri, Jun 14, 2013 at 04:52:08PM -0500, Curtis Rueden wrote:<br>
> Hi Melissa & everyone,<br>
><br>
> Thank you for replying to my suggestion. I appreciate the discussion.<br>
><br>
> > Note that we've previously (and relatively recently) put quite a bit<br>
> > of effort into getting the OME-XML code and specification into<br>
> > bioformats.git.<br>
><br>
> Yes, you might recall that I am the one who did most of that work. ;-) [1]<br>
><br>
> And I would be willing to do it again, if it meant a cleaner build system.<br>
> (Though splitting subtrees is much less complicated.)<br>
><br>
> > If we were to follow this proposal, and unless I misunderstand, then<br>
> > making one simple change in the OME-XML schema (e.g. changing a single<br>
> > field type) would require:<br>
><br>
> Well, I think specification and ome-xml should live in the same repository.<br>
><br>
> So it would require:<br>
> - filing a PR into the specification repo (ome-xml.git in my proposal)<br>
> - cutting a new ome-xml release artifact<br>
> - filing a PR into bioformats.git to update the version of ome-xml<br>
><br>
> So, two PRs instead of one. (As Mark said, scifio.git doesn't depend on<br>
> ome-xml.) But the schema and ome-xml code doesn't change nearly as often as<br>
> Bio-Formats does, of course.<br>
><br>
> Still, I can appreciate how two PRs might be undesirable (believe me; you<br>
> know how much I hate the develop/dev_4_4 split :-). To avoid that, you<br>
> could keep specification & ome-xml & bio-formats in the same git repository<br>
> as they are now, while still using separate versioning per component. This<br>
> would provide on target for PRs, while making it much simpler for most<br>
> people to build Bio-Formats thanks to remote release dependency resolution<br>
> of Maven/Ivy/etc.<br>
><br>
> > Our policy so far has been to release everything at once; I think it<br>
> > would make more sense to agree first whether that should be changed,<br>
> > and then consider solutions. I personally do favor having everything<br>
> > be released at once, as it makes keeping track of version numbers<br>
> > (mentally and when supporting users) much easier.<br>
><br>
> Indeed, my proposal hinges on the idea that everything should actually not<br>
> be released at once -- that many of these are separate projects, which are<br>
> A) useful in their own right, outside an OMERO-specific context; and B)<br>
> developed at a much different pace than one another, resulting in<br>
> gratuitous/unnecessary/confusing releases when all continue to be versioned<br>
> together. [2]<br>
><br>
> > I understand the desire for smarter autogeneration, but I think that<br>
> > would be much better accomplished within our existing build systems,<br>
> > rather than fragmenting the codebase.<br>
><br>
> I do not see it as one codebase, but rather several projects which all<br>
> currently happen to be lumped into one repository with a single release<br>
> schedule imposed upon them. Right now, all of these project releases are<br>
> dictated by the OMERO release schedule [3]. The OME team is ramping up for<br>
> the OMERO 5 release, and presumably a Bio-Formats 5 release will go along<br>
> with that. But what is really so radically new in Bio-Formats 5? Nothing, I<br>
> would argue. Now, e.g., an extensible SCIFIO-based Bio-Formats would<br>
> justify a 5.0.0 release. But instead, I am sad to see Bio-Formats versions<br>
> that have little semantic meaning [4] with respect to Bio-Formats itself,<br>
> all because of the simultaneous top-down release schedule driven by the<br>
> OMERO project.<br>
><br>
> Instead, my proposal emphasizes the individual projects as useful to the<br>
> community and the world in their own right. For example, MDB Tools Java is<br>
> not available anywhere else to my knowledge [5], since we rescued it from<br>
> an obscure forum post. Wouldn't it be a wonderful service to the community<br>
> to post it to its own Git repository on GitHub, so that it is easy to find,<br>
> and the greater community (beyond just those interested in OME) might<br>
> contribute back?<br>
><br>
> I understand I am proposing a substantial change. It is certainly about<br>
> more than just improving the code generation process. That's why I changed<br>
> the subject. And I understand the reluctance to change anything that might<br>
> disrupt current development processes. But in this case, I think the change<br>
> would be well worth it to make the OME projects into even better members of<br>
> the global FOSS community.<br>
><br>
> Regards,<br>
> Curtis<br>
><br>
> [1] <a href="http://trac.openmicroscopy.org.uk/ome/ticket/10435#comment:8" target="_blank">http://trac.openmicroscopy.org.uk/ome/ticket/10435#comment:8</a><br>
><br>
> [2] One quick example of how the gratuitous releases cause problems:<br>
> whenever I update Bio-Formats using the ImageJ updater, it notices that<br>
> several of the JARs actually have no functional changes, and does not alter<br>
> the version number of the JAR files. Yes, we could potentially fix this in<br>
> the updater, but wouldn't it make more sense to simply not make vacuous<br>
> releases in the first place?<br>
><br>
> [3] Note that we are not even that consistent, since there are several<br>
> other projects we develop like native-lib-loader which are *not*<br>
> synchronized with the OMERO/Bio-Formats version. This works just fine,<br>
> since we treat those projects as external dependencies. The same thing<br>
> would work fine for the components/forks, components/stubs and ome-xml.<br>
><br>
> [4] ImageJ2 and SCIFIO are now using SemVer, which conveys useful<br>
> information in the version number: <a href="http://semver.org/" target="_blank">http://semver.org/</a><br>
><br>
> [5] Well, potentially this: <a href="http://jackcess.sourceforge.net/" target="_blank">http://jackcess.sourceforge.net/</a><br>
><br>
><br>
> On Tue, Jun 11, 2013 at 12:52 PM, Melissa Linkert <<br>
> <a href="mailto:melissa@glencoesoftware.com">melissa@glencoesoftware.com</a>> wrote:<br>
><br>
> > Hi Curtis,<br>
> ><br>
> > > That said, I think Bio-Formats would greatly benefit from substantial<br>
> > > modularization of components. We are realizing this with SCIFIO, and I<br>
> > > think it applies to the OME-XML component as well.<br>
> > ><br>
> > > Below, I will lay out what I think is a better structure for the build<br>
> > > system, which would result in more advantages and less pain than with the<br>
> > > current structure.<br>
> > ...<br>
> > > MetadataStore, MetadataRetrieve, etc., would move to the ome-xml<br>
> > component,<br>
> > > keeping all generated code together.<br>
> > ><br>
> > > One Git repository for each of:<br>
> > ><br>
> > > - SCIFIO (<a href="https://github.com/scifio/scifio" target="_blank">https://github.com/scifio/scifio</a>)<br>
> > > - OME-XML (<a href="https://github.com/openmicroscopy/ome-xml" target="_blank">https://github.com/openmicroscopy/ome-xml</a>)<br>
> > > - Bio-Formats (<a href="https://github.com/openmicroscopy/bioformats" target="_blank">https://github.com/openmicroscopy/bioformats</a>)<br>
> > > - Fork: Apache POI (<a href="https://github.com/openmicroscopy/ome-poi" target="_blank">https://github.com/openmicroscopy/ome-poi</a>)<br>
> > > -- change package prefix to avoid third party code collisions<br>
> > > - Fork: MDB Tools Java (<a href="https://github.com/openmicroscopy/ome-mdb-tools" target="_blank">https://github.com/openmicroscopy/ome-mdb-tools</a>)<br>
> > > -- change package prefix to avoid third party code collisions<br>
> > > - Fork: JAI Image I/O (<a href="https://github.com/scifio/scifio-jai-image-io" target="_blank">https://github.com/scifio/scifio-jai-image-io</a>)<br>
> > > -- change package prefix to avoid third party code collisions<br>
> > > - Stub: LWF (<a href="https://github.com/scifio/lwf-stubs" target="_blank">https://github.com/scifio/lwf-stubs</a>)<br>
> ><br>
> > What you and Mark do for your work on SCIFIO is up to you, but I would<br>
> > be extremely hesitant to do something like this for Bio-Formats itself.<br>
> > Spreading one codebase across 7 different repositories is at best<br>
> > invasive, and would have a substantial impact upon anyone who routinely<br>
> > works on Bio-Formats.<br>
> ><br>
> > > In other words, OME-XML gets its own Git repository, which includes all<br>
> > the<br>
> > > code generated code. Each fork and stub also has its own repository in<br>
> > the<br>
> > > relevant namespace.<br>
> ><br>
> > Note that we've previously (and relatively recently) put quite a bit of<br>
> > effort into getting the OME-XML code and specification into bioformats.git.<br>
> ><br>
> > > Dependencies between repositories would be done by release version<br>
> > > coupling. For Maven projects (i.e., SCIFIO), simply making releases and<br>
> > > using release dependencies would be sufficient to facilitate repeatable<br>
> > > builds. For Ant-based projects (i.e., stuff in openmicroscopy namespace),<br>
> > > release JARs would continue to be committed to the repository as they are<br>
> > > now, or they could be resolved remotely via Ivy or similar.<br>
> ><br>
> > I don't agree that doing that makes things easier. If we were to follow<br>
> > this proposal, and unless I misunderstand, then making one simple change in<br>
> > the OME-XML schema (e.g. changing a single field type) would require:<br>
> ><br>
> > - a pull request into whichever repository houses the specification<br>
> > (currently bioformats.git)<br>
> > - creation of "release" artifacts from whichever repository houses the<br>
> > specification<br>
> > - a pull request into ome-xml.git (to update the autogenerated code)<br>
> > - creation of release JARs from ome-xml.git<br>
> > - a pull request into scifio.git (to update SCIFIO readers)<br>
> > - creation of release JARs from scifio.git<br>
> > - a pull request into bioformats.git (to update Bio-Formats readers)<br>
> ><br>
> > ...instead of what we have now, which is a single pull request into<br>
> > bioformats.git.<br>
> ><br>
> > > This would making building Bio-Formats much simpler and faster. As Roger<br>
> > > pointed out, we do not really need to code generate the OME-XML stuff on<br>
> > > every build, but rather only when the schema changes. Of course, the<br>
> > > OME-XML component contains other code which would be subject to change<br>
> > > between schema releases, but that's fine.<br>
> ><br>
> > I understand the desire for smarter autogeneration, but I think that<br>
> > would be much better accomplished within our existing build systems,<br>
> > rather than fragmenting the codebase.<br>
> ><br>
> > > This more modular structure would also facilitate these components being<br>
> > > developed on separate release cycles. The forks and stubs rarely change<br>
> > and<br>
> > > do not need to be released with every OME release. And the OME-XML<br>
> > project<br>
> > > could be released along side schema changes (i.e., twice a year) rather<br>
> > > than with every OME release.<br>
> ><br>
> > Our policy so far has been to release everything at once; I think it<br>
> > would make more sense to agree first whether that should be changed, and<br>
> > then consider solutions. I personally do favor having everything be<br>
> > released at once, as it makes keeping track of version numbers (mentally<br>
> > and when supporting users) much easier.<br>
> ><br>
> > Again, what you do with respect to <a href="http://github.com/scifio/scifio" target="_blank">http://github.com/scifio/scifio</a> is up<br>
> > to you. Doing this for Bio-Formats itself would have a non-trivial impact<br>
> > on every single OME team member and a large portion of the developer<br>
> > community,<br>
> > and as such I think it would be better to consider other options for<br>
> > making autogeneration easier.<br>
> ><br>
> > Regards,<br>
> > -Melissa<br>
> ><br>
> > On Mon, Jun 10, 2013 at 10:50:25AM -0500, Curtis Rueden wrote:<br>
> > > Hi Roger & everyone,<br>
> > ><br>
> > > Sorry for the delay in reply. After spending the last couple of weeks on<br>
> > > ImageJ build issues related to native code components (specifically, the<br>
> > > ImageJ launcher in C), I have some new perspective on the new code<br>
> > > generation of the Bio-Formats build system.<br>
> > ><br>
> > > First of all, I want to say thanks to Roger for solving the build for<br>
> > both<br>
> > > Ant and Maven. I know maintaining the dual build systems can be<br>
> > substantial<br>
> > > extra work. But I think the Maven system has many advantages, so I am<br>
> > happy<br>
> > > it is being maintained.<br>
> > ><br>
> > > That said, I think Bio-Formats would greatly benefit from substantial<br>
> > > modularization of components. We are realizing this with SCIFIO, and I<br>
> > > think it applies to the OME-XML component as well.<br>
> > ><br>
> > > Below, I will lay out what I think is a better structure for the build<br>
> > > system, which would result in more advantages and less pain than with the<br>
> > > current structure.<br>
> > ><br>
> > > > One thing which might be an issue is that while xsd-fu generates the<br>
> > > > ome-xml model code, which could potentially be downloaded, it also<br>
> > > > generates all the MetadataStore, MetadateRetrieve and all the other<br>
> > > > Metadata-related classes in scifio, including OMEXMLMetadataImpl.<br>
> > > > Given that these are paired with the generated model code, generating<br>
> > > > one and downloading the other may result in breakage on model changes,<br>
> > > > or changes in xsd-fu or the templates which change the generated code.<br>
> > ><br>
> > > MetadataStore, MetadataRetrieve, etc., would move to the ome-xml<br>
> > component,<br>
> > > keeping all generated code together.<br>
> > ><br>
> > > One Git repository for each of:<br>
> > ><br>
> > > - SCIFIO (<a href="https://github.com/scifio/scifio" target="_blank">https://github.com/scifio/scifio</a>)<br>
> > > - OME-XML (<a href="https://github.com/openmicroscopy/ome-xml" target="_blank">https://github.com/openmicroscopy/ome-xml</a>)<br>
> > > - Bio-Formats (<a href="https://github.com/openmicroscopy/bioformats" target="_blank">https://github.com/openmicroscopy/bioformats</a>)<br>
> > > - Fork: Apache POI (<a href="https://github.com/openmicroscopy/ome-poi" target="_blank">https://github.com/openmicroscopy/ome-poi</a>)<br>
> > > -- change package prefix to avoid third party code collisions<br>
> > > - Fork: MDB Tools Java (<a href="https://github.com/openmicroscopy/ome-mdb-tools" target="_blank">https://github.com/openmicroscopy/ome-mdb-tools</a>)<br>
> > > -- change package prefix to avoid third party code collisions<br>
> > > - Fork: JAI Image I/O (<a href="https://github.com/scifio/scifio-jai-image-io" target="_blank">https://github.com/scifio/scifio-jai-image-io</a>)<br>
> > > -- change package prefix to avoid third party code collisions<br>
> > > - Stub: LWF (<a href="https://github.com/scifio/lwf-stubs" target="_blank">https://github.com/scifio/lwf-stubs</a>)<br>
> > ><br>
> > > In other words, OME-XML gets its own Git repository, which includes all<br>
> > the<br>
> > > code generated code. Each fork and stub also has its own repository in<br>
> > the<br>
> > > relevant namespace.<br>
> > ><br>
> > > Dependencies between repositories would be done by release version<br>
> > > coupling. For Maven projects (i.e., SCIFIO), simply making releases and<br>
> > > using release dependencies would be sufficient to facilitate repeatable<br>
> > > builds. For Ant-based projects (i.e., stuff in openmicroscopy namespace),<br>
> > > release JARs would continue to be committed to the repository as they are<br>
> > > now, or they could be resolved remotely via Ivy or similar.<br>
> > ><br>
> > > This would making building Bio-Formats much simpler and faster. As Roger<br>
> > > pointed out, we do not really need to code generate the OME-XML stuff on<br>
> > > every build, but rather only when the schema changes. Of course, the<br>
> > > OME-XML component contains other code which would be subject to change<br>
> > > between schema releases, but that's fine.<br>
> > ><br>
> > > This more modular structure would also facilitate these components being<br>
> > > developed on separate release cycles. The forks and stubs rarely change<br>
> > and<br>
> > > do not need to be released with every OME release. And the OME-XML<br>
> > project<br>
> > > could be released along side schema changes (i.e., twice a year) rather<br>
> > > than with every OME release.<br>
> > ><br>
> > > Comments welcome.<br>
> > ><br>
> > > Regards,<br>
> > > Curtis<br>
> > ><br>
> > ><br>
> > > On Thu, May 2, 2013 at 12:25 PM, Roger Leigh <<a href="mailto:r.leigh@dundee.ac.uk">r.leigh@dundee.ac.uk</a>><br>
> > wrote:<br>
> > ><br>
> > > > On 02/05/2013 16:52, Curtis Rueden wrote:<br>
> > > ><br>
> > > > > If so, the build is completely identical--the sources which get<br>
> > > >> > generated on the fly from the templates by xsd-fu are identical<br>
> > bar a<br>
> > > >> > few lines comments in the top boilerplate.<br>
> > > >><br>
> > > >> OK, good to know.<br>
> > > >><br>
> > > >> One more question/concern: presumably, the Bio-Formats build no longer<br>
> > > >> functions on Windows, due to the Python + Genshi dependency. With the<br>
> > > >> Ant build, this might be non-trivial to solve. But solving the issue<br>
> > > >> with Maven is very straightforward: include the "ome-xml" module in<br>
> > the<br>
> > > >> reactor only within a profile. Then, when that profile is not enabled,<br>
> > > >> Maven will resolve the ome-xml dependency from the remote repository<br>
> > > >> rather than regenerating and rebuilding the code. This would eliminate<br>
> > > >> the need to install Genshi, and make it easier to build on Windows<br>
> > > >> again. What do you think?<br>
> > > >><br>
> > > ><br>
> > > > I'm afraid I'm no authority on Maven, so I'm not sure. Maybe Melissa<br>
> > or<br>
> > > > Josh have a better take on this than me. I assume that this will work<br>
> > > > correctly on Windows if python is installed?<br>
> > > ><br>
> > > > One thing which might be an issue is that while xsd-fu generates the<br>
> > > > ome-xml model code, which could potentially be downloaded, it also<br>
> > > > generates all the MetadataStore, MetadateRetrieve and all the other<br>
> > > > Metadata-related classes in scifio, including OMEXMLMetadataImpl.<br>
> > Given<br>
> > > > that these are paired with the generated model code, generating one and<br>
> > > > downloading the other may result in breakage on model changes,<br>
> > > > or changes in xsd-fu or the templates which change the generated code.<br>
> > > ><br>
> > > > While it's not all enabled yet, I'd like to have the model selectable<br>
> > as<br>
> > > > an ant properly (it's xsdfu.schemaver), so that it's possible to change<br>
> > > > to a different model when building. There's currently some hardcoded<br>
> > > > "2012-06" versions which need to be switched to change to use the<br>
> > > > property value.<br>
> > > ><br>
> > > ><br>
> > > > Regards,<br>
> > > > Roger<br>
> > > ><br>
> > > > --<br>
> > > > Dr Roger Leigh -- Open Microscopy Environment<br>
> > > > Wellcome Trust Centre for Gene Regulation and Expression,<br>
> > > > College of Life Sciences, University of Dundee, Dow Street,<br>
> > > > Dundee DD1 5EH Scotland UK Tel: (01382) 386364<br>
> > > ><br>
> > > ><br>
> > > > The University of Dundee is a registered Scottish Charity, No: SC015096<br>
> > > ><br>
> > > > ______________________________**_________________<br>
> > > > ome-devel mailing list<br>
> > > > ome-devel@lists.**<a href="http://openmicroscopy.org.uk" target="_blank">openmicroscopy.org.uk</a><<br>
> > <a href="mailto:ome-devel@lists.openmicroscopy.org.uk">ome-devel@lists.openmicroscopy.org.uk</a>><br>
> > > > <a href="http://lists.openmicroscopy." target="_blank">http://lists.openmicroscopy.</a>**<a href="http://org.uk/mailman/listinfo/ome-**devel" target="_blank">org.uk/mailman/listinfo/ome-**devel</a><<br>
> > <a href="http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel" target="_blank">http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel</a>><br>
> > > ><br>
> ><br>
> > > _______________________________________________<br>
> > > ome-devel mailing list<br>
> > > <a href="mailto:ome-devel@lists.openmicroscopy.org.uk">ome-devel@lists.openmicroscopy.org.uk</a><br>
> > > <a href="http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel" target="_blank">http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel</a><br>
> ><br>
> ><br>
</div></div></blockquote></div><br></div>