Building a transaction service for managing large scale editing experiences

A common challenge developers have to tackle when building applications consuming large data sets, is how to create a maintainable and scalable user experience for editing. They may face hundreds of thousands and even millions of records on which they execute CRUD operations that need to be sent to a server and saved to a database.

They want to make sure no redundant calls to the server are executed.

At first sight it may look like we can take advantage of the ngrx/store capabilities in order to handle such scenarios. The store is usually an application wide singleton service, useful to cover complex component interaction scenarios.

In this case we were looking at enhancing one component which won’t make the most of the benefits of NgRx Store and we are also looking for some specific functionality beyond the scope of Redux — keeping an accessible chronological history of all actions and being able to manipulate that history and thus the resulting state. We saw fit for that purpose to rely on an encompassing service to handle state. How the service specifically reduces the actions to a state would depend on the implementation and could even make use of a store internally if you choose so.

In this article, we describe how our team has approached the challenge to minimize the number of HTTP requests sent to the server when updating records. We will also look at how bundling operations can allow for a more flexible client app with additional features.

What was the issue?

In our Angular client app we present the data set to the user in a table UI, where each table row maps to a record from the data. With CRUD operations, while the user can add or delete, edits in particular are cell-by-cell. So without any additional work, your component would need to react to each edit and send an HTTP request, containing only the information for this single cell to the server.

Imagine you have a table with 20 columns and the user performs edits on all cells in a single row — this would total twenty requests to the server for a single record update. Assuming your data layer is record-based, those API calls will result in 20 data record updates as well. That’s 19 too many and you can multiply that by the number of users of your app and draw conclusions for the extra load on your backend.

What is worse — if the user makes 5 edits, then change their mind and revert the change to the original values — this would still mean 10 HTTP requests are sent to the server for no change at all! Another drawback is that it may be tedious to handle each individual operation (edit, add, delete) and separate them into requests.

This was not the experience we would like to provide in our Ignite UI for Angular library.

Taking steps: Transactions

To tackle those issues, we implemented a service that would bundle the updates and that would allow us to reduce the number of HTTP requests to the server. The Transaction Service is an injectable middleware (through Angular’s DI) that a component may use to accumulate changes without immediately affecting the underlying data, much like you’d do when implementing the Unit of Work pattern.

You can add a transaction and commit or clear all changes. As a bonus — as it keeps a detailed log, it can also undo and redo operations. It also has a pending session (more on that later) that records all added changes and once the session ends, adds them to the log as a single unit. We have a couple of implementations, below we will cover all the possible features.

Every time you execute an operation (transaction), it is added to the transaction log and undo stack. All the changes in the transaction log are then accumulated per record. From that point, the service maintains an aggregated state that consists only of add/update/delete operations for unique records.

An example interface for the service may look similar to:

A more detailed interface from our source can be found here and the Transaction interface looks like:

Implementations of the service are decorated with @Injectable so that they can be provided to the component, pipes, etc. As the components inject the services by their type and token, you may provide different implementation of the Transaction service, depending on your needs. Note that the type serves as a token if not specified explicitly with @Inject.

Once you’ve injected the service in your component, to perform an “add” operation, you would create a new transaction and add it like so:

For existing records, the add method accepts an optional parameter — a reference to the original record.

The id of the transaction matches the record to update and the type of update (add, update or delete). For the newValue of each transaction we went with change snapshots using partial objects. This makes it very clear what the change is — for example { name: “John NotDoe”} would signify that just the name property of the record has been modified. This also allows the service to easily accumulate state using Object.assign and apply that onto the original when needed.

Calling undo, moves a change to the redo stack and back to the undo stack when you call redo. Aggregated state is updated accordingly and as with all operations, the service emits onStateUpdate to let consuming components know they should update their visual state.

Speaking of visual updates, for these we went with a custom @Pipe — this way changes applied through transactions are seamless and further transformations (such as filtering, sorting and so on, depending on your component’s features) can still be chained. In our implementation we use a pure pipe, meaning it is executed only when Angular detects a change to an input value. In our case, since transactions don’t manipulate the data, we have defined an additional number parameter pipeTrigger, which the component may increment to trigger the pipe, when there is a change in the Transaction state.

Here is how a simplified example for such a pipe may look like:

How transactions enable new editing features

The transactions provided us with the opportunity to implement two separate editing features in our grid component. We will go through their specifications as you may find them relevant to your use cases and you may implement similar behavior in your component or reuse them.

Our grid module provides a very basic implementation of the service with just pending session functionality, yet that’s still plenty for Row Editing! By using startPending() and endPending() Row editing can combine multiple per-cell operations into a single change. This means editing multiple cells of a single record creates a single transaction and you can handle just the row edit event. This is also more in line with the record-based API your ORM (data layer) would have.

With the accumulated state being a partial object, we can also use the service to check which cell has been edited and build UI around that. As you can see on the image below, we have updated two of the cells (italic) and we are currently editing the “OrderDate” cell for this record.

Cancel ends the pending session, while “Done” would create a transaction, if the currently provided service implements that.

Speaking of implementation, the Ignite UI for Angular package also ships with a full-fledged transaction service implementation (IgxTransactionService) with full transaction, undo and redo support. All you need to do is provide it as a drop-in replacement for the basic built-in and you have already enabled Batch Editing!

We have again used the per-row state in the service to provide UI cues to the user when a particular row has unsaved changes. Below the last two rows are newly-added (all italics), one is deleted, and one has an edited cell. All rows with unsaved changes have a modified indicator on the left edge of the Grid.

Both cell and row editing modes can work with transactions. The main difference is that cell edits are added to the transaction log when the cell exits edit mode. When we have row editing, a transaction is added only after the whole row exits edit mode. In both cases the aggregated state of the grid is uniform — consisting of all the updated, added and deleted rows.

This automatically solves the issue of dealing with per-cell edits as those are now accumulated under one record. As a bonus, users may update the very same cell multiple times, undo and redo as much as needed without sending redundant requests to the backend. When the actual sync to the server happens is to your app’s use case and we’ll discuss options next.

Now let us get to the practical part of our article.

How is the Transaction Service consumed by components?

To implement Batch Editing using your component, you need to provide a Transaction Service implementation. As mentioned above, Ignite UI for Angular package ships with one (IgxTransactionService), but you can also substitute your own.

In this example, we would demonstrate the use of our transaction service in a custom table control. So, the first step would be to import it to your component like this:

Then we would define our table component template:

The injected service instance is exposed as transactions property on the table and we can use its API to bind our actions. In order to achieve this, we need to bind to an event emitted by our component each time a record is updated and instead of saving the update to our data, we should add it as a transaction as described above.

The undo and redo buttons are enabled based on canUndo and canRedo that return if the respective undo/redo stack has entries. We may also commit the changes via commit method or discard all changes with the clear method.

You may take a closer look at an example similar to the one we are using in this section in our Batch Editing docs. While exploring how the Ignite UI for Angular grid is consuming the Transaction Service, note that the IgxGrid uses a specific injection token for the service (IgxGridTransaction) provided in its module, and we need to override it by providing the Transaction Service on a parent component level. You can find the complete sample on StackBlitz.

These examples are informative about what transactions look like and how are they managed but all the changes we make are stored locally. How would our solution look like when we have a server, we send our transactions to?

Syncing the transactions to the backend

In order to process and send the transactions in our Angular application, we would create a service to encapsulate the data access in order to separate it from the presentation layer. The service in the example will make good use of Angular’s HttpClient semantic methods to match our REST endpoints. The IgxGrid in this example is bound to data set with cities and the service will have a commitCities method that accepts an array of transactions to send.

We can reuse the sample from the previous chapter and only expand what the commit method does:

The getAggregatedChanges method of the Transaction service returns the current aggregated state as transactions, while the optional parameter merges the change snapshot with the original record for the transaction value (most ORM updates require the full record). We call commitCities method, providing it with the transactions.

Once the service completes, we commit the transactions in our Grid’s service — this updates the client data and clears out the logs, causing the Grid to remove the unsaved markers. While app-specific, usually the record IDs are auto-generated by the data layer and endpoints for ADD operation return the updated entities. We have included a snippet saving the return result of the ADD call and an update to the data after the commit. This completes the sync of the client data with the server. The last bit recreates the data array reference to trigger the OnPush change detection strategy of the Data Grid.

Now let us see how those transactions are processed by the service. At that point, we have three different approaches to send them to the backend.

The first option is to send all the transactions in a single request and process them on the server. The second option is to iterate through all the transactions in the service and execute separate HTTP request for each of them, matching a standard per-record RESTful API.

The third option, which we would implement here, assumes your REST endpoints support multiple records. This allows us to group all the transactions by type and send an individual request to the backend for each transaction type, for a maximum of 3 requests.

Matching the change type with the respective HttpClient verb method — delete, additions with post (depending on the HTTP spec your backend supports, add can also be done through a PUT verb), updates with put and we extract the required data format (IDs or raw values) based on the endpoint requirements:

We check if we have any transaction for each type of request and if this is the case — we push the corresponding request to the requests array. This guarantees that no redundant empty requests are sent to the backend.

We merge all possible requests into a single Observable so we can subscribe to it in our component and execute logic after all transactions are processed.

Conclusion

As we compare our batch updating implementation to the solutions that do not use the Transactions Service, we see the Transaction Service helps us reduce the number of requests sent to the backend a lot. It also allows enhancements to the editing user experience by providing visual indications for the state of the changes and opens up possibilities for additional features such as undo/redo.

The Transaction Service approach can level up the editing experience of your next app using not only the Ignite UI for Angular Grid but also when working with your custom components.