Sync Framework Motivation

by Jeff Vroom

One of the larger complicating factors in software design, performance, and maintenance begins when two programs need to communicate. The developer needs to design or choose a protocol, perhaps adds versioning so it can be easily upraded or changed. They need to make it secure, reliable and possibly deal with retries in case of failure or handle the additional connection related errors that might occur.

In my experience, quite a bit of code goes into solving this problem. It's one of the large contributors to "code that's not application logic".

Increasingly frameworks are code-generating serialization logic rather than requiring the developer to write that explicitly to reduce the volume of this code with a declarative approach. Perhaps the next step in this evolution is to recognize that most client/server applications at their core are solving problems more simply and robustly solved using a synchronization metaphor. Client processes start out by replicating various slices and properties of data models. As they make changes, those changes are queued up and applied back to the server. Certain properties are fetched on-demand, or explicitly using an API, which makes it easy to fetch a batch of data at a time. When a new object is created on the server in response to a client's request, it can automatically be sync'd to the client in the reply, or lazily fetched when it's first referenced by an object being sync'd to the client. In many applications, you can reduce or eliminate the wrapper code, and many of the RPC methods you have to code by hand using declarative annotations. When RPC is needed, it works naturally with the sync framework.

Synchronization also benefits greatly from the layers feature in StrataCode. With layers, we can split code into the 2 or 3 required pieces: client, server, and shared. We can detect automatically when a method call is in a remote process.

Synchronization leverages StrataCode's code-generation features as well. Metadata required for synchronization is added as readable code to the generated files making it easy to navigate and debug. To enable synchronization, we convert fields to get/set methods where the set method fires a data binding event. Finally, a method call can be converted transparently into a remote method call during the code-generation phase. If the runtime does not support synchronous remote methods, this is only allowed if the method call is in a data binding expression. Otherwise, an error is generated at compile time.

Sync Features

One of the advantages of doing synchronization declaratively is that you can turn on some advanced features through configuration, and by leveraging the synchronization design pattern.

With data binding and data sync in StrataCode, there's no need for asynchronous programming or function closures. A domain model object can be expressed with a method call in a data binding expression which can later be deployed in a remote process from the call site without changing the code. This uses a model where property change events typically fire asynchronous methods and set properties again when the results are received.

Synchronization is built on the "scope" mechanism in StrataCode. Objects can be configured to have an appropriate lifecycle and visibility for their purpose. An object can be global to the system, global for each top-level application object (usually the web page the client is using), one browser session, one appSession (i.e. one application for one session), one window (i.e. browser window), or one request, or created and destroyed as part of a single operation.

Synchronization supports real time connections. When the client is using a real time connection, the server can push changes to the client.

Synchronization propagates change events from one scope to the next and performs necessary locking to prevent two conflicting threads from running at the same time. So if you have a shared object between two scopes, changes made by one are queued and delivered to the other. For example, you can use the StrataCode command line to execute methods in the browser when using a real-time connection. This makes it easy to script client/server applications.

Because it uses a stateful protocol, and because object graphs are preserved after deserialization, you can reduce redundant data transmission and eliminate bugs introduced when two copies of the same object instance end up on the client.

Using synchronization to manage state has a nice benefit for scaling web applications. Your server keeps a session which caches information held by that client, so it can avoid redundant requests to the database. If the client's copy goes away, they can choose to refresh from the server's cache so they do not lose their state. Similarly if that server session goes away, because it expired or the server process was restarted, the client's copy can be used to refresh the server. So again, you do not lose any context. This approach is designed to allow developers to support a balance of reliability and efficiency all from a declarative model. I believe ultimately it will be an easy way to produce scalable web applications with the lowest latency and infrastructure cost.

Easier Multi-Process Development

There are many advantages to decoupling components that can be efficiently managed independently. We've all read about Microservices and the success of businesses like Amazon which made decoupling of systems a major priority. But the downside of a decoupled system is the complexity of standing up a complete environment. Developers are at their most efficient when you have a quick and reliable - change code, build, run, test cycle.

With the StrataCode system, you can build all processes from the same stack of layers. The layer definition files are also the perfect place for the code that updates and restarts processes as necessary. So for any project, just running 'scc list-of-layers' should be the way developers work, and operations deploys no matter how many processes or environments are required to let them run their code. Layers can handle all of the details - including all of the configuration specific to this user's environment. Because StrataCode itself is built on code-generation, supports maven, multiple processes, customizing and running scripts a system like 'chef' is not really needed. Dev-ops can manage their own set of production layers which keep their environments separate from development.

See these examples: UnitConverter, TodoList, and Program Editor.

Client/Server Synchronization Using Layers

Today client/server programmers commonly use remote procedure calls (RPC) with a serialization protocol like JSON to communicate with the server. Some frameworks require that you write code to convert to and from JSON, others handle that transparently, or declaratively.

You can also use the sync framework with simple RPC directed by annotations. These cause a SyncManager.addSyncType(.) call to be made in the generated code which registers the set of remote properties and their metadata.

The biggest hassle with direct RPC code is that you have to handle the response asynchronously (at least when you are using an asynchronous runtime like the browser). Stratacode gives you an option to avoid response listeners when you use remote methods in data binding expressions. You can bind one or more read-write properties, to the results of a data binding expression, triggered by other client properties. A stream of change events is sent to the server and the stream of results are returned. A set of properties are updated and the UI is updated in response to those changes.

When you use this model, you have declarative/functional access to data you retrieve asynchronously, and can cleanly invoke remote methods in response to events even when they are async. The same data binding expressions will work in either a client/server environment, or local which makes it easier to reuse domain model code, even when the domain model has remote methods. This is particularly valuable when a business user is editing data-binding expressions that control some combination of client and server functionality.

By it's nature, RPC involves copying state back and forth explicitly. Programmers have to write that code explicitly and spend a lot of time with tricky code that amounts to reimplementing the same patterns over and over again. What you are really doing here is synchronizing objects, or parts of those objects, between the client and server. StrataCode offers a model for doing just that - synchronize objects automatically using declarative patterns.

You annotate classes, objects and properties with the @Sync annotation, or put them in a shared layer and annotate the layer itself with @Sync. If you set @Sync on a layer, StrataCode detects which layers overlap between different processes and generates code to synchronize just the overlapping classes, objects, and properties back and forth.

When a type is marked with @Sync, calls are inserted into the code to track instance creation, and property changes. Those events are recorded and sent to the client or server on the next sync (i.e. at the end of the request, or at the end of some user-interface interaction).

The synchronization framework lets you build rich, interactive apps, with complex graph-based data models with little to no remote procedure calls, no data transfer objects, and with a statically typed protocol - for efficient errors and easily versioned protocols. The declarative patterns implemented by the sync framework support the most common uses cases - create, update, delete, lazily fetching of references and collections, and more. When your application needs more control, you can set the @Sync annotation on properties or types to fine-tune the behavior. If you still need more control, you can fall back to RPC on the same objects. Or start out with an RPC call, and use synchronization to apply or receive changes to the results. When you call a remote method using already-synchronized objects as parameters, those args are passed by reference making it much easier to build an API which works both locally or remotely.

The protocol between client and server is designed to be readable and organized - so when you change "firstName" and "lastName" of a User object, it comes across as two property changes of the User type. Serializers are implemented in JSON and SCN (a StrataCode layer which is converted to JS before being shipped to the client)

Read more in the documentation.