HIHiJon89·Dec 6, 2017
[Copying my reply from reddit]
I can give you some background on the way we structure things in case it's helpful. We use GitHub for our open-source projects, GitHub:Enterprise for our internal code, and our engineering team has about 300 engineers. Each project or feature usually lives in its own repository, and is split into a number of Maven modules. Using the maven-snapshot-accelerator ( https://github.com/HubSpot/maven-snapshot-accelerator ) as an example, it's roughly 1,000 lines of code split into 6 Maven modules (representing a small percentage of what my team owns). There's the root pom which just aggregates the other modules, a core module that contains the POJOs, a REST API module that uses the POJOs, a client module which uses the POJOs to hit the REST API, a module for the maven plugin, and a module for the maven extension. I
Read the rest
f this was an internal project, it might also have supporting Hadoop jobs, Kafka workers, Spark jobs, etc. depending on how the system was designed. Each of these would usually live in a separate Maven module. I'm not sure how much thought we gave to splitting into Maven modules along these lines, it just kind of seemed natural to us. It also has a few benefits compared to larger modules:
- The dependency trees are smaller, because you're pulling in a more focused set of libraries rather than the kitchen sink
- Our build infrastructure only builds what has changed when you push a commit, but it can only do this at Maven module granularity. So by splitting things up this way, pushing a change to your kafka workers won't rebuild your Spark or Hadoop jobs.
Using one of our larger open-source projects as an example, Singularity ( https://github.com/HubSpot/Singularity ) is split into 15 Maven modules (and represents maybe a third to a half of what that team is responsible for).
Snapshots vs. versions is a whole different topic and probably deserving of its own blog post. More generally, whether we should bend our workflow to the tools or bend the tools to our workflow is a constant tension and something we're always thinking about, so feedback is always appreciated.