Leveraging your code

by Jeff Vroom

Are you a programmer looking to leverage your code? In other words, do you want to maximize the value you deliver per hour of your time over the long term? If this challenge interests you, you may quickly encounter a choice. Should you choose the most powerful programming language - one that lets you express yourself with the most concise and flexible operators and libraries, perhaps using a functional language, or will you get more leverage by coding in a way that improves the readability, customizability, and longevity of your code. Instead of coding for yourself, should you be coding for everyone else so you can write more code and let others maintain it?

For applications which need to be customized and maintained by large teams, to me the choice is clear. Clients and users of these applications are happier and more productive when they can make the changes the need without programmers. Programmers can accomplish more in the long term when they code in a way that improves the readability and flexibility of their code, so it continues to evolve even when they are building something new. To get more leverage from our code, at a high level, we need cleaner separation of framework code from application code. Let the framework code do the heavy lifting of mapping the application code to meet the platform requirements, keeping the application code so it solves the domain problem in a single block of code. We would like our domain models to be normalized, so there's a single source of truth. They should be readable and declarative, except when imperative is required to implement business rules that are most easily modelled that way. When it's necessary to customize logic for the platform, keep that logic separate and attached to the domain model so we can trace these dependencies as we change the code.

With these high-level goals and principles, I've had a long career building systems that try to allow:

This search started more than 20 years ago when I worked on a first-generation visual data-flow application geared for scientists called (AVS), then designed and built a reactive, declarative object-oriented system called AVS/Express. The applications built on these systems were at the time more concise, more declarative, and easier to manage for scientists and both platforms enjoyed success. But I learned how quickly these simple, declarative, object graphs became too large to navigate visually. Though there were large reusable features shared between applications, it was difficult to create larger resuable parts from the source code and actually share them. You could customize applications easily but it was hard to find the logic you wanted to change and making a new copy required forking the entire application creating a maintenance challenge. I found that object-oriented patterns for navigating and reusing code alone were not enough to permit independent customizations in a way that scaled as customization requirements grew.

That experience inspired my contribution to the design of the layered component configuration for the ATG Commerce Platform (aka Dynamo). By merging layers of component configuration, Dynamo's nucleus breaks complex applications into manageable slices of medium sized features, making it easier to configure families of application environments and maintain complex software systems.

The lessons learned from watching hundreds of enterprise applications built on ATG inspired the design of StrataCode layers. The more I thought about organizing systems in layers, the more potential I saw. It felt like a basic missing organizational abstraction for information systems.

In nature you can't help but see the way layers and layered processes are embedded in the organization of all things - from the rings of a redwood tree to organization of the brain. We use layered structures all of the time in computing as well, and even as a way of organizing executables with the system path, and classes in your class path. What if layers were the fundamental way to organize code and assemble programs? When Photoshop added layers as a organizational abstraction for image editing, it changed everything. Can layers in code do the same thing for programming?

Introducing StrataCode

StrataCode let's you organize your Java code using layers, an extension of the traditional patterns used by object oriented programmers to organize code as modules. Each new layer can modify or override types, methods, and properties defined in previous layers. By adding and removing layers from the stack, you create new versions of your application incrementally. You still benefit from static typing and compile any given stack of layers into one Java program so the runtime view is no different. There are only a few small syntax extensions so if you know Java, you can read a given StrataCode file, but to understand how it runs, you need to learn how that layers combine different files to create a single ordinary java file. You may start with one file per class, all in one layer for your designs. Down the road, as requirements and dependencies grow, the value of layers grows - whether it's more people making changes, more code, more products, more customizations, more versions you need to support, or a code base is becoming monolithic and a bottleneck from external and internal dependencies.

StrataCode is more than just layers. The underlying code-processing tools make it easy for framework developers to process code using an incremental read-modify-write code processor called Parselets. This means generated code is the same as the source, except where a transformation is required (e.g. a field converted to get and set methods, or a new layer is merged in). Comments are preserved and inserted so you can debug it in either source form or the generated form.

Components, data-bindings, layers, and other language features are all implemented using parselets.

When you use code-generation in this way, application source code can stay independent of framework code, remaining easy to read. Framework dependencies are injected into the source during code-generation in some cases, or merged in via layers when you really need framework specific logic at the application level. For example, if you need to override the default persistance annotations or add a custom query for performance. In these cases, using Java's static typing system, you can most often trace the effects of a domain model change using find-and-replace type operations. Framework code is maintained by framework developers using easy-to-learn and use code-processing tools, giving programmers and architects more leverage over how application code gets deployed.

Learning to Code with Layers

Learning layers can be a lot like learning object-oriented programming. Not all programmers take to that metaphor for designing their code, but everyone can code against a well designed object-oriented API. The same should hold true for layers. You can use them like modules and start out using StrataCode as a build tool, and use APIs and applications built with layers just like Java.

The StrataCode IDE helps by providing wizards for the common operations, edit-time editing and a Java-like editing experience for StrataCode.

Refactoring

As the needs for your system become more complicated, instead of refactoring objects, you can cut and paste code into layers with less work and without changing published APIs. With layers, you have flexibility to split classes or component instance configuration based on the dependencies in those slices of functionality. When programmers organize code based on dependencies, that code can be reused in more contexts, and can be easier to navigate.

Dependencies between Modules and Layers

For example, let's say you have a class called UserAccount which both defines the properties and persistence for that class. The first time you need to use that class without a database, or use it with a different database, you will need to refactor it into two classes, then change all references to use the right class. With layers, you just move the code that depends on the database into a new layer which extends the base layer. That's frequently a quick copy/paste task and gives you two versions of your code - one which depends on the database, and one which does not. You also just made the code more modular by localizing all of the aspects of persistence into one file, and the persistence code of that module in one directory. Annotations set on the persistence layer can change the default behavior of the fields and methods which means less for you to specify. Similar code ends up in the same layer, making it easier to read and maintain as well.

Additionally, with layers you can eliminate recursive references that might develop between modules, even without changing the APIs. Dependencies with Modules Code that adds the recursive dependency is put in a new layer that modifies types: Dependencies with Layers

Layer Basics

To build and run any StrataCode application just specify a layer, or the set of layers you want to run. One command first performs code-generation, then compilation, then runs the code if desired. The resulting build directories represent typical projects for the runtimes you are using. Any processes that need to be started are started automatically by the framework layers you include.

Each layer has a definition file that acts like your build script. No XML, Javascript or Groovy - instead use statically-typed StrataCode, using the same dynamic runtime behind dynamic layers. Why not use the full power of an IDE and static typing to write and manage your build files?

Here are some details on how the StrataCode build process works. First all layer files are parsed the extended layers are added. The layer stack is split into separate, ordered stacks, one for each runtime or process. This let's StrataCode manage more than one process from the same stack - for example, client javascript and server. All layers in each stack are all initialized, then all started. During the init/start phases, code or components in the layer definition file can add source path and class path directories.

Layers tend to fall into one or more categories: framework, application, domain model, UI, option, deployment. Framework layers add package dependencies, register for all supported file-types, specify path-name mapping, and runtime/process constraints that help StrataCode determine

The framework layers an application layer extends determines the file types it can have and how they are processed. Framework layers can add new languages which are parsed, file types which are processed, annotation processors, set the build-paths, and more. Application layers specify normal source code. Configuration layers are declarative layers exposed as needed. They typically customize the classes and instances defined in application layers using selected types, properties and methods specified by the developer.

Once all layers are started, a validation phase is run on all layers to check for possible errors or problems. Then the build phase begins.

The last layer in the stack is considered the build layer for the application. Intermediate layers in the stack can set buildLayer=true so that they are built separately, so the buildLayer only needs to recompile the deltas.

When you perform an incremental build, each build layer only needs to update changed files. (Note: This feature is not yet fully reliable except for small changes).

Layers do not change the nature of Java types at runtime. The generated code is easy to read and debug to understand what is happening.

StrataCode IDE

The IntelliJ IDE for StrataCode supports main features of Java IDEs: navigation, refactoring, debugging, and edit-time errors that prove the integrity of your code, so you are debugging fewer runtime errors. You can debug either the StrataCode source, or the generated code, or quickly switch back and forth.

In the IDE's "Run Configuration" editor, you choose a set of compiled layers, or dynamic layers and options.

In the Layers View, you can see all of the layers accessible in your project.
You can filter layers by runtime, process, type. You can view all of the available or 'inactive' layers. Or after you've started a program, you can use layers view for the active layers.

Code-Processing and Management UIs

One great way to empower users of your system, and allow it to scale, is to build a great management UI - the user interface that customizes the system itself. With StrataCode, developers can easily create layers for specific slices of configuration, rules, queries, or even simple methods. They can delegate the management of these assets to others in the organization. New versions of the application can be created from the hinge points defined for these application slices. Rather than requiring the developer to build a new plugin API or explicit management UI, they get both for free by using layers directly by using a framework and exposing slices of their domain model. A layer template can be defined by setting annotations or creating a simple annotation-only layer that lists properties, methods, or types.

The same dynamic code framework which powers the IDE, also is available in the dynamic runtime for creating these management UIs: code completion, syntax highlighting, edit-time errors for all formats. This includes the ability to patch code on the fly, detect when a restart is required, and help ensure a smooth restart.

These management UIs can support immediate viewing of changes for rapid prototyping in many cases - whether you are changing styles in a stylesheet, segmentation rules for your customers, or adding a new category of products. When your components are reactive, they can respond to property changes without restarts. If they are dynamic you can change methods and add fields in many cases as well.

Scalability of Management UIs

The realty is that most SaaS platforms have their management UIs directly edit the live site, so updates are immediate. If you want a test/staging environment you can set one up and now deal with maintaining two complete copies of your system. It takes discipline, or good tools to avoid problems.

For large systems, with multiple overlapping development time-lines, and content and code contributors far and wide, most SaaS platforms won't support this well at all and will stop scaling for that solution area.

Layers provide a solution to help SaaS platforms improve management of customization intent, which helps address this problem. With a layer, you can isolate or group interrelated changes to code, data, files by combining them into the same layer, or one layer which includes another stack of layers. It's easy to include them in a test environment for testing, or even use them in a multi-variate testing environment to compare performance. When it comes time to deploy them, you can merge or if problems arise, roll them back reliably.

In a way, layers are a different form of separable 'pull request', a set of deltas to code, but where it assembles the stack for a specific version at system build/run time.

Usability of Management UIs

Management UIs are best when they can target specific types of users - e.g. a designer, administrator, merchandiser. For each type of user of your system, you can have different "layer templates" - each of which can change overlapping sets of types and properties. When you create a new template, it can copy some default slice of code, or it may start out empty. The base layers it extends will define the set of types and properties they can change. As changes are made to this layer, source files for the affected types are incrementally updated. At any given time, the management UI can show the "diff" of changes made in the source files, as well as the changes made from the previous layer.

To make this flexibility manageable, everything is traceable with tools. You can quickly find all layers for a given type, property, or method. You can find all usages to an identifier and change names just like with Java. Read, modify, and write any source file in any layer without being a language expert using the provide APIs.

Layers and Source Control

Layers complement the strengths of source control systems and provide an alternative when you encounter source control problems. The strengths of source-control are too numerous to mention, but they break down when you try to incorporate files edited by tools which do not support incremental, localized changes that are easily represented by diff'ing the old and new version. Or when you have have two sets of files in the same source repository which have completely different workflow constraints - i.e. being edited by different groups on different time scales and review processes. Or whenever a single repository is too large. They are not great at migrating files and maintaining history when you reorganize, although hopefully that limitation will be fixed.

StrataCode helps complement source control in several ways:

  1. Configuration or code that's modified by a management UI is changed incrementally so the diffs are still valuable. You can develop more tools which edit assets under source control put key assets which are managed by tools under source control more easily.

  2. Different layers can be managed by different source control repositories and then merged during the build. This makes it much easier to move assets back and forth between teams, or break out assets for customization during deployment, testing, or for longer-term development projects. Even code that's edited by your management UI is readable and manageable by developers. The nature of layers means that moving some code into a new repository, managed with a separate lifecycle is easy. Two developers could even work on different overlapping parts of the same type, in two separate layers.

To keep layers from piling up over time, one layer can be merged into another and discarded, either by hand or programmatically. This lets you use them temporarily, to isolate some changes from the rest of your code.

If you've used monkey patching, overlays, or written a patch-script you've worked your way around the lack of this essential metaphor when you need it. Instead you can do it in a more manageable way by adding a layer.

Frameworks Built on Layers

StrataCode has evolved into a rich, well designed code-processing framework with a relatively small, clean, easy-to-read-and-debug code base. The power of code-generation combined with layers opens up opportunities for framework development and I have had lots of experience building frameworks. It's evolved over years, and integrations with several frameworks have improved the design to be "good enough" for most applications.

StrataCode includes a layered component-oriented, data-binding and synchronization set of framework layers. I built them using lessons learned from my experience with these platforms. You can use StrataCode with other frameworks easily - in fact, it's been integrated with several standard Java frameworks already and adding more is easy as well.

If you look at the complete platform, you'll see how it can simplify your Java code, separate framework code from application code to remove platform dependencies, make applications more declarative and easier to customize.

If you want to use StrataCode with your existing projects, just point a layer at your maven or git repo and see if it can replace your existing build/run toolset. How fast can you create new customized versions of your system? You can try out any other features easily by plugging them into a new layer. Replace a file, add new fields, override methods, etc. Use it for testing, development, deployment configurations as you like.

For brief introductions to the frameworks read these articles or for details, consult the documentation.