[ome-devel] Modularising the bio-formats build using git submodules

[Josh Moore]

On Aug 5, 2013, at 2:15 AM, Curtis Rueden wrote:

> Hi Roger,
> 
> Nice, thanks for your efforts. Before we proceed too far down the
> submodules road though, I am curious how it compares to the use of git
> subtree, a relatively newer approach to this problem. Josh Moore and I
> briefly discussed it when I last visited; Josh, do you remember the name of
> the project you cited that is structured that way?

The project I mentioned to you, Curtis, was solarized. It used subtree to keep both a unified repo and broken out subtrees in sync. I don't know if the bridge was two-way or just one-way (especially in terms of a PR workflow).

See:
 * https://github.com/altercation/solarized/blob/master/utils/subtrees.mkd
 * http://endot.org/2012/02/12/git-subtree-tracking-made-easy/

~J.

> Regards,
> Curtis
> On Aug 4, 2013 5:31 PM, "Roger Leigh" <rleigh at dundee.ac.uk> wrote:
> 
>> I've been thinking a little bit about how we might be able to better
>> modularise things so that individual components could be built
>> separately, independently released and used by others without having to
>> build all of bio-formats, but also still allowing the existing setup to
>> be used.  I've recently been looking at Qt, and their repo at
>> http://qt.gitorious.org/ (git://gitorious.org/qt/qt5.**git<http://gitorious.org/qt/qt5.git>)
>> contains 34
>> submodules referenced by this tiny top-level repo; run the
>> "init-repository" script to set it all up.  Qt is huge, with each
>> component being an entire project in its own right.  You can clone and
>> build each separately, or as a whole by cloning the toplevel container
>> repo.
>> 
>> I've made a test repository using this one component per submodule
>> scheme, which is here: https://github.com/rleigh-**dundee/bioformats5<https://github.com/rleigh-dundee/bioformats5>
>> (https://github.com/rleigh-**dundee/bioformats5.git<https://github.com/rleigh-dundee/bioformats5.git>);
>> you can see all the
>> dependent submodules here:
>> https://github.com/rleigh-**dundee?tab=repositories<https://github.com/rleigh-dundee?tab=repositories>(they have a
>> "bioformats5-" prefix).
>> 
>> After cloning, you just need to do a "git submodule init; git submodule
>> update" to set it all up.  There are 23 components in total.  We could
>> also split out the junk like the binary jars into a submodule as well.
>> 
>> This was done in a very simplistic way, one submodule per component with
>> a pom.xml.  It may not be the ideal arrangement, it's just a proof of
>> concept.  Each submodule was generated from current develop with
>> "git filter-branch --tag-name-filter cat --prune-empty
>> --subdirectory-filter components/test-suite HEAD".  The top-level is
>> just the current develop with the components deleted and replaced by the
>> corresponding submodule.  But the top-level could be replaced with a new
>> repo which would drop all the history duplicated in the submodules.  Or
>> we could have multiple toplevels with different selections of
>> submodules, which would allow old, unused components to be dropped.
>> 
>> This scheme could also allow openmicroscopy.git to include the set of
>> specific bioformats components it needs, rather than the bioformats
>> submodule used currently.  And it could itself be split up into
>> submodules per component in the same manner, though this would require
>> some preparatory refactoring.
>> 
>> 
>> Regards,
>> Roger