Motivation for layers

by Jeff Vroom

As a thought experiment, you might find it interesting to think about where you find the layer design pattern in software you use. Here's a few I can think of: Java's classPath, "monkey patching", maven overlays, objective C 'mixins'.

Now let's think about the natural world and consider the places where "layered designs" occur. You find layers in the structure of just about anything - organic or inorganic. For a clear example, take a look at the structure of the brain. Layers are a fundamental way complexity is organized in nature. So we see layers in both the computational and natural worlds frequently and yet in modern programming languages, there's no unifying way to use them for organizing software we build. Instead, layers appear here and there as ad-hoc features. Unlike in nature, programmers do not today use them as the essential organizational abstraction. What if we changed all of that and introduced layers as both a feature for organizing programs, as well as a language construct for composing types?

StrataCode attempts to answer that question by using layers to add a hinge point to our design flexibility for building and maintaining larger, and more complex systems more easily. You can use layer oriented programming to build traditional module structures, or use layers to refine or extend a class, or change the configuration of an instance without renaming. Plugins built into the language and the replace and merge operations built into the modular structure.

More flexible than modules

Like modules, they are a packaging mechanism with dependencies. We say that a layer may extend one or more other layers which ensures those other layers are below it in the stack. Where you'd use a module before, you'll use a layer with StrataCode. Unlike modules though, layers can include changes, or apply deltas to what's underneath. The process works like layers in Photoshop but where we merge by name, not pixel position.

Still statically typed

Layers preserve encapsulation and are statically typed, adhering to the SOLID principles of O/O design. They may replace or add to an object's interface or override but cannot remove a previously defined contract. They also cannot depend on a layer that comes after them in the current list of layers. Just like Java, you have traceable code paths, edit time errors, find usages, refactoring and more. They are similar to Delta oriented programming and can also be used for product feature-lines.

Modules and circular references

Today we use modules, not layers. As modules grow in size, they tend to develop circular dependencies. For example, Module A depends naturally on Module B but as the project grows, some minor aspect of B may develop a dependency on some minor aspect of A, possibly due to a poor modularization decision. Refactoring to eliminate the circular dependency would work but breaks compatibility and so may not be an option. The API contract has been "burned in stone" as they say. Allowing the new circular in the long term though creates severe code management problems. Conceptually now A and B must be updated in lockstep. You lose many of the benefits of modularization in the first place. Changes in B may depend on changes in A and vice versa. It becomes impossible to test a new version of A with an old version of B and vice versa. That reduces your ability to detect when the interfaces of your modules change in an incompatible way.

The circular reference is not a pattern that would form in nature. In nature, a new thing would come along to modify what's been burned in stone, even though it involves fixed type names which are tied to the modular structure. Nature is good at repurposing names with modifications and yet our programming languages don't recognize that at the structural level.

Our code models are mostly based on graphs of objects. But nature's code - DNA is linear. There's no room for circular references and yet lots of room for modification of the same name.

While life has lots of circular references in form, they are all built from layered processes and layered structures. In our programming language, the code itself is like the DNA and so we should have a layered structure at the core and that is in the language itself. It's the evolutionary component which means, it's not fundament to how you build your code, but how it lives long term.

StrataCode uses code processing and a self-aware dynamic runtime with management UIs to fill in the missing gaps in the biology metaphor. A system which is aware of itself, can reproduce itself with a declarative grammar that structures the modifications, even when then human is in the loop of those modifications.

Layers eliminate circular references

With layers, a downstream layer cannot refer to an upstream layer. Code dependencies are enforced at compilation time to ensure dependencies only go one way. This is essential to preserving modularity as systems grow.

Instead, when you would need to add a cyclic dependency, due to the addition of that minor feature, this code is added in a separate layer that extends both of the previous A and B layers. It separates that minor dependency from the bulk of the code. You retain the independence of the rest of the code without breaking APIs or compatibility. If you want existing clients to see this new module, the new layer is named "A" and the old "A" is renamed. If you want to offer this as an opt-in feature, you give the new layer a new name and tell clients to point to that layer to get that feature.

Languages like OCaml do not allow cyclic references between modules. It's a hard-line decision that yields faster start up times, more modular systems so in some ways a good language level design decision. But it comes at a huge cost to "upgradeability". As your system grows more complex, some changes may require refactoring to avoid a conflicting dependency. Without the ability to use a cyclic reference, you need to move code from one type to another to avoid a new upstream dependency. That will inevitably break any code using the code that has to move, causing unnecessary work, headache, and overall friction for the consumers of any upstream systems.

Layers give you the best of both worlds: the option to enforce one-way dependencies in the modular structure for a given group of modules, and the ability to incrementally add cyclic relationships to types while keeping the code modular.

Here's a diagram comparing the way dependencies evolve in your code from one version to the next, comparing a typical modular organization to one using layers:

Dependencies between Modules and Layers

Version 1 of both designs is the same. But in both designs for version 2 we need two new dependencies. Some minor feature added to module C now depend on some minor feature of types in A and similarly a minor feature in module E now depends on D. With modules only, these new dependencies take away many of the benefits of modularity leaving essentially one hairball of code. When using layers, we can make the same changes without creating a hairball by adding these new minor features in a new layer. This layer modifies types C and E creating new versions of these types which have the feature, but only when that layer is present. With layers, only code which requires the new features needs to depend on the new layer. The bulk of the existing code does not have the new dependency. Tests and older code can continue to use the older independent versions retaining the benefits of the existing module structure. APIs do not change as code moves from one layer to the next. To get the same benefit with modules, we'd have to create new classes which breaks the existing APIs.

Dealing with complex types

It's very difficult to keep two complex types in your system from developing a circular dependency between them - e.g. "User" and "ShoppingCart". The most intuitive types are built from natural entities in the system and two complex entities may need to depend on each other to implement certain features. By splitting complex types into a small number of well organized layers, you can support these types of use cases beautifully, and make those large files easier to navigate as well.

Mechanics of layers

Like modules, layers may extend one or more other layers which they depend upon. This exposes all of the types and instances in that layer to the extending layer. Layers also can pass along imports to subsequent layers. Each layer gets a nice sandbox of all of the types imported by the layers it extends. Downstream code does not depend on the exact package name of an imported type, allowing you to easily substitute variants by adding an intermediate layer.

For any specific use case, it's easy to create one layer which pulls in all dependent layers. But it's also convenient to add additional layers to the run configuration, as options, or new ways to run the one application layer. Either way, all dependent layers are automatically included and sorted. Ultimately each collection of layers - a layered system - maintains a single ordered list of layers at any given time which are merged to form the "runtime view" of the application.

An unconstrained layer can modify any type, in any package and sometimes this is useful because the layer may ultimately affect any type in the system. But such 'global layers' are relatively rare. In most cases you specify a package for the layer. When you do, you can only modify types in that package in that layer. Additionally, files in the layer's directory use that package without the extra directories Java typically makes you create to organize code. This is particularly nice when layers are edited by non-programmers, or when you can infer categorization by context. Instead of a "one size fits all" project directory structure, framework developers can create simpler project types, customized for different groups of users: e.g. deployment-configuration, application-configuration, source code, localization, user-interface style, etc.

Certain layers are designed as "build layers". These layers are validated and compiled to produce Java classes or applications which can be run or tested. When you compile a stack of layers, you might have more than one build layer in the stack. In that case, the build is done incrementally - so only the changed types from one stack to the next are re-generated and recompiled. This makes it efficient to define incremental builds - so only the source you are changing gets recompiled from one build to the next.

Why layers?

Why objects, why methods, etc. Consider that object-oriented design may have missed a key "hinge point" for code customization. Layers add the the ability to refine types and refactor code to keep dependencies separate from complex types. Some languages add "mixin features" in an ad-hoc way that undermines code traceability and design integrity. Perhaps the lack of layers has helped fuel the growth for dynamically typed languages because O/O frameworks are awkward when applied to certain problems. The addition of dynamic layers allows you to work with dynamically modified types, but still retain the concept of types.

At the end of the day, it's difficult to describe the need for layers, because clarity of this design principle appears at higher design complexity than most have the need to consider. You'll realize the benefits when maintaining large, highly customized code bases, often distributed over many developers, like those commonly found in enterprise applications and frameworks. In those situations, layers provide major benefits by helping to improve workflows for enterprise systems, separating design, administration, business rules, workflow, and code.

When combined with StrataCode's declarative features, layers provide a hinge point for customizing any property of any object. You can replace any element of any HTML file. Append to or replace any component. Override any formula. Though layers add a new concept, the IDE and management UI frameworks make it easy to build powerful, customizable applications.

Pure domain models

Among software architects, the term "domain model" usually refers to the core business code. The goal is to keep this code pure - so it does not depend on the the user-interface, database, etc for maximum reusability. The domain model specifies the types of objects involved in the business, the properties of those objects, configuration of the objects, operations performed on them, and rules that should be applied or enforced. Ideally this model is mostly declarative and independent from the database, user-inteface and other framework specific aspects of the system. The faster you can evolve your domain model, the faster your business evolves towards greater efficiency.

The design challenge begins when you try to express your domain model in a programming language. You need a good type system, efficient compiler, etc, and at the runtime level you need flexibility to create "many integrations" with highly varying needs for the timeliness, rate, and integrity requirements for accessing and updating data. Your code rapidly collects dependencies on both the runtime, and on external systems. Each dependency limits your ability to reuse that code in other contexts. You satisfy the conflicting dependencies through code copying, and building remote interfaces - treating this logic as a service coupled loosely to other systems and code bases. These remote systems frequently need to copy data as well, data de-normalization, for operational needs. Whatever approach you take, the logic behind the domain model leaks into other systems with more difficult to manage dependencies. This makes it harder to adapt to changing business needs.

Layering helps you preserve an independent version of your domain model which is expressed in all of the contexts in which it's required. Dependencies exist in framework specific layers that annotate or modify the domain model in a structured way. Those dependencies are translated into the generated code. By keeping them out of the domain model source, you make that source reusable in any future context. Using annotations, base layers, base classes, etc. you can write powerful framework features to manipulate the code of your domain model so it's properly serialized, stored in a database, de-normalized, and decoupled for operational reasons but without touching the domain model using powerful code-generation, and framework features that operate on the code as data. For situations where you need to customize domain model code for a framework, you override features in a traceable way that makes it easier and faster to maintain. When you need to break apart services for operational needs, share the overlapping parts of the domain model between services so you have statically typed system that is easily refactored.

Let's look at a simple domain model object:

class Quiz {
   String name;
   List<question> questions; 
}
You can add JPA persistence with a new layer that contains a simple Quiz.sc file that modifies the Quiz object:
@Entity
@Table(name="quiz")
Quiz {
   // Primary key is the quiz name
   override @Id name;

   // Define a one to many relationship between a quiz and its
   // questions.  The questions will automatically be persisted when the
   // quiz is persisted.
   override @OneToMany(cascade=CascadeType.ALL, fetch=FetchType.EAGER) questions;
}
You can add annotations to existing properties and methods from a layer that modifies the domain model layer. When you modify a property in a layer, you add that property to the list of properties in that layer. This additional way to create multiple overlapping sets of properties has a variety of uses in improving design of systems. For one, management UIs can organize different views customized for different audiences. Or you can set additional layer-level defaults which apply to all properties in a layer - making them all by default persistent, traceable, serializable, etc.

To use a quiz, you might also create an instance of a Quiz in one layer:

object ScienceQuiz extends Quiz {
   name = "Science";
}
and initialize it in a separate layer:
ScienceQuiz {

   questions {
      object question1 extends Question {
         question = "The galaxy we live in is called the Milky Way.  It is shaped\n" +
                    "approximately like:";
         answerChoices = { "A round ball", "A doughnut", "A pretzel", "A flat spiral" };
         answerIndex = 3;
         answerDetail = "The Milky Way has four spiral arms radiating out from a\n" +
                        "central cluster of stars (nucleus).  Our solar system is\n" +
                        "located on one of the spiral arms, quite far from the\n" +
                        "center.";
      }
   }
}

This is a nice example of a configuration layer you could hand-off to someone else.

To make that easy, each layer is stored in a separate directory with parallel file names and path structure which can be swapped in and out of the application for different purposes. You can map where source files in a layer end up in the build configuration by extending base layers which define the context.

So layers help partition assets among people: business analysts, designers etc. One person defines the data model, another manages persistence, a third manages business data and rules. Layers help separate assets along role boundaries for better workflows.

Just as layers separate code by their dependencies, so do they separate code by the individuals who manage them. These two are commonly related for essential reasons, so why not use an environment which supports that separation.

Styles and design elements

Style sheets separate configuration from code in a clean way and are very powerful but sometimes overkill and overly complicated. How do you find and isolate the effects of a code change? Style sheets do not have the equivalent of a "find usages" at development time. Fortunately good debuggers help us trace that down at runtime or we'd be lost.

Sometimes it's better to constrain things and not use the more flexible selector types in CSS. If you use a simpler model for style management with standard types, layers and multiple inheritance you retain the ability to find all usages and make changes using static typing for more reliability and control.

Layers by themselves provide separation between code and styles. The code exposes the customizable properties - on both classes and instances through the same name-space. Designers deal with a single tree of types and properties. All of this is toolable because of strong typing, and Java's visibility rules.

UnitConverter {
   foreground = Color.WHITE;
   background = Color.BLACK;
   errorLabel {
     foreground = Color.RED;
   }
}

StrataCode's multiple inheritance feature lets you apply properties across the class hierarchy as well. This gives you a strongly typed version of stylesheet's "class" selector. Object names are analagos to the "id" selector.

The design phase pratically requires immediate updates so you can experiment with different looks quickly and dynamic layers let you do that without compromising on runtime performance.

Testing/Monitoring

It's common to want to instrument your code in various ways to improve it's testability and monitorability. In some cases, there are hard tradeoffs to make for performance or readability. Aspect oriented programming for a while was viewed as a viable approach to this problem for inserting code involved in "cross-cutting" concerns: logging, assertions, timing, etc. While it did reduce the quantity of code you had to write, it did not use traceable or debuggable patterns and so never gained widespread traction.

Layers let you inject code from separate files so that you can add this code in a way that preserves these code paths. For example, you could modify the core 'process()' method with a method that does timing around that method, then called super.process(). When you include this layer, you'd have a monitored version of the application and when it was excluded you'd avoid that overhead and any dependencies it created to the monitoring package. The layer itself would be a directory tree that showed all monitoring hinge points. If you renamed the "process" method, this layer would be updated automatically... if you deleted it, you'd get an error reminding you that you need to update the monitoring hook.

Here's a simple example of inserting monitoring code before and after some servlet:

MainProductionServlet {
  void service(ServletRequest request, ServletResponse response) {
    SystemMonitor.monitorStart("ProductionServlet");
    boolean success = false;
    try {
       super.service(reuqest, response);
       success = true;
    }
    finally {
       SystemMonitor.monitorEnd("ProductionServlet", success);
    }
  }
}

Behind the scenes, StrataCode will create a new MainProductionServlet class when this testing layer is included. It will use that class instead of the default one. You can create layers which add much more testing and monitoring logic because it is isolated both from the core code and core runtime.

Localization

Localization is one of the most common and most important forms of application customization. Layers give you some new tools and benefits as with other forms of customization.

For programmers, the easiest way to develop code is to hardcode the strings so it's easiest to change them and understand the code. Localization typically involves putting a resource identifier into the code and moving the string into a separate file. Some systems let you use one language in the code and then provide a separate file that maps strings in that language to another based on string-name.

Both of these schemes have problems. Most of the time, you are debugging the program in one locale so the resource identifier schema adds overhead to something you do a lot - go from error message to source code. But that's just an annoyance. You also lose static typing for compile time errors. Instead you find out about missing resources at runtime... which means you have to run everything in each locale. Given the number of possible locales and the breadth of your test suite, that inevitably leads to error messages you fail to localize.

programmers define strings with static final/const variables and initialize them as they might normally. You simply override those strings in sub-layers, one for each locale. At compile time, a new class is generated which replaces the original strings with the new strings. You get a new efficient localized executable for a smaller download size. Or use a dynamic layer and apply it at runtime for a resource bundle approach.

MainAppPanel {
   welcomeMessage="Welcome {0}!";
}
Localization does not stop at changing strings. You might also need to adjust UI spacing for a given language, or re-arrange menu items to preserve case-sensitive order. With StrataCode, one layer can manage all of these aspects by providing one modularization structure that can modify code and configuration making it easy to manage and build localized systems.
MainAppPanel {
   welcomeMessage = "Willkommen {0}!";
   leftSideWidth := windowSize * 0.25;
}