NoSQL databases have a solid niche in the world of operational databases, especially for new greenfield projects. One of the good examples of such a database, strongly promoted by Microsoft, is a Cosmos DB, in particular in its native flavor of SQL API. Cosmos DB gets more and more traction among the customers, Microsoft actively invests in its development, making it the recommended option for many scenarios where both hot and warm storage are required.

When comes to the methodology which could be used to build the most efficient data model, you hardly can find one. There is no clear and formal methodology that describes how to approach the data modeling, only a set of suggestions. In this article, I'd like to close this gap and define such a methodology in a clear and formal way. We'll start with the hardest thing, the theory, and in the second part of the article I'll explain how it works in pratice.

Methodology in a nutshell

To build a robust data model for Cosmos DB, that would serve the application needs in the most efficient way, you should use the approach described on the diagram later. The diagram contains the three main steps of the modeling process marked in gray. Each step requires a list of inputs and generates a target output. All inputs and outputs are represented by white rectangles.

/uploads/stories/how-to-apply-cosmos-db-modeling-methodology-using-ddd-aggregates-dot-part-1/cosmos-data-modeling_1.jpg?1656340121007

The main modeling activities consists of the following three steps:

Define a list of Aggregates using domain-driven design practices based on Conceptual data model and Application workflow
Prepare a logical data model considering two additional inputs—data realities and statistics of the commands and queries usage. This part of the data modeling activities is the core of this article
Verify the model by a set of integration and performance tests. Apply a set of optimizations to come up with the final physical data model

Let’s review each step of this modeling flow in more detail, what kind of activities they include, and what are the inputs and outputs of each step.

Modeling activity inputs

The application workflow and conceptual data model are first-class citizens in the data modeling process, as well as in Chebotko’s approach. They are the main inputs for further data model building.

Application workflow

Once we understand the main actors and building blocks of the business domain, we need to understand the application workflow along with all the data access patterns described by queries and commands that build this workflow.

It is important to understand behavior of objects and the story they are going to tell us. We need to understand what the data lifecycle looks like and how are the objects created, updated, and deleted; how do the objects communicate; what are the main data retrieval patterns; what parameters are used to search for objects; how are the results represented; which objects are married and returned together.

Conceptual data model

Any modeling activity starts with exploring the business domain. We need to have a clear understanding of what we are modeling, what are the main actors and entities, the main building blocks, and how they are related to each other. It is important to cover structural associations: consist-off, is-a, has, and similar types of relationships. The outcome of the conceptual data modeling could be represented in different ways. I propose to use one of these two options:

Classical conceptual data model represented by a set of entities that are connected between each other by a set of relationships. A multiplicity factor is specified for each relationship
Domain data model represented by the UML class data diagram

Both models describe the data domain from the structure perspective and serve as the first step in the data modeling flow.

Queries and commands statistics

When queries and commands are defined, it is important to understand the expected throughput for different kinds of requests and understand the statistics of their running.

When you build something from scratch, it is important to predict the number of such requests at least at the order of magnitude level, so that for each of the requests we could set the proper priority. If you are dealing with a migration project and the queries and commands are already implemented and running on production, it is essential to gather the real production statistics for each request.

Data realities (Data profiling)

It is really hard to prepare any data model without knowing the data. Data volumes, cardinality, and the relationship nature are very important aspects driving many decisions while modeling Partition key definition, ID field selection, data embedding vs referencing, distributing data among containers, all these decisions are highly dependent on the data realities.

DDD Design: Aggregate definition

Theory first: Building blocks of domain-driven design and definition of aggregate

Let’s review some aspects of the domain-driven design concepts related to model-driven design, such as entities, value objects, and aggregates. Let’s consider extracts from Eric Evans’s book:

Entity (reference object) — an object defined primarily by its identity. An entity is an object distinguished by its identity, rather than its attributes.

Value Object — an object that represents a descriptive aspect of the domain with no conceptual identity. We care only about for what they are, not who or which they are.

Aggregates — an aggregate is a cluster of associated objects that we treat as a unit for the purpose of data changes. Each aggregate has a root and a boundary. The boundary defines what is inside the aggregate.

Root of Aggregate — the root is a single specific entity contained in the aggregate that outside objects are allowed to hold references to. Entities other than the root have a local identity but that identity needs to be distinguishable only within the aggregate. Root entities have a global identity, entities inside the boundary have a local identity, unique only within the aggregate.

The aggregate definition process

The methodology of aggregate definition is not new and is described in detail in a set of resources, in particular in the mentioned above Eric Evans’s book “Domain-Driven Design: Tackling Complexity in the Heart of Software”. Here are the main steps of the aggregate definition process:

Use the conceptual data model as a starting point to shape the data domain and define all the general entities and relationships involved in the application workflow
Split all the entities from the conceptual data model into the entities and value objects according to the DDD definitions
Use the application workflow as an input to identify the anchor entities from the whole list of entities
Having the anchor entities, define the boundary for each aggregate
Iterate through the application workflow analyzing the queries and commands and re-evaluating the aggregate definitions

Mapping the DDD aggregates to the logical data model

We came to the most crucial part of the whole process—how to map previously developed DDD aggregates to the logical data model. For this process, we should also use the two additional inputs: data realities (data profiling information) and queries and commands usage statistics.

The mapping process assumes you have basic knowledge of the main Cosmos DB building blocks. Please refer to the Appendix part for a quick recap of the main concepts.

Let’s review the mapping guidelines as a sequence of mapping rules:

MR1 (aggregates). Aggregates map to containers. All the entities and value objects defined in a scope or boundary of one aggregate should be stored in the same container.\

MR2 (anchor entity). The anchor entity becomes a first-class citizen inside a proper container and maps to the proper Cosmos document type. The primary key identifier of the anchor entity defines the partition key of the container. The ID field could be set to be equal to the primary key as well.

MR3 (non-anchor entities). The non-anchor entities of the aggregate map to the separate Cosmos document type inside the same container. Since the non-anchor entities are always bound to the anchor entities, they could derive the anchor entity primary key, so they have the partition key defined properly. The primary key of the non-anchor entities maps to the ID field of such a document type.

NOTE

Based on the data realities and query statistics there could be an exception: if the write operations on the non-anchor entities are happening rarely and the relationships are one-to-few, the non-anchor entity could be embedded into the anchor entity.

MR4 (value objects). There are two main options for mapping value objects to Cosmos DB items:

Value objects are embedded in the document types they are related to as embedded JSON objects. This option should be chosen under the following conditions:

There are one-to-few relationships between the parent entity and child value objects
The value objects bound to the parent entity do not change frequently
The value objects bound to the parent entity will not grow without bound

Value objects similarly to the non-anchor entities map to the separate document types in the same container. We select this option in other cases: when value object data does changes frequently or when a number of items could grow in an unbounded manner

MR5 (Dictionary Container). The set of aggregates that consists of just one orphan anchor entity and logically corresponds to the dictionary-like data should be grouped and stored in one specially dedicated container with a name similar to Dictionary.

One possible way to structure data inside this container is to set the partition key to be equal to the dictionary name and map the ID field to the primary key of a specific dictionary.

MR6 (Inter aggregate relationships). By definition, the relationships between entities located in different aggregates could be made only in case the parent entity of the relationships plays the role of the anchor entity inside its aggregate. In such a case the child entity contains just a reference (primary key value) of the parent entity. The referencing techniques should be used.

Physical optimization

Physical optimization is the final step in the modeling flow. While the logical data model shapes the data model and establishes the foundation, a series of reviews and adjustments still should take place.

The topics to review include reviewing the size of the partitions and data distribution among them to make sure the data is distributed evenly and there are no hot or too large partitions.

Index strategy is another important area to review. In many cases, the default full coverage of the index could slow down the write performance as well as increase the cost of all the write operations. Here the most important inputs would be queries and commands usage statistics as well as data realities. We need to balance between optimizing the reads and slowing down the writes.

To close the topics on indexes, sometimes for extremely big containers with lots of physical partitions, cross partition queries that use indexes could become costly and inefficient. In such a case there is a need to duplicate the dataset into another container with a different partition key, so that the cross-partition query becomes the partition-driven one.

Some queries which require the aggregations could be optimized by introducing the counters—special additional fields that store pre-calculated numbers.

Using this methodology, you can build data models with any level of complexity for both greenfield projects and large legacy migrations. We're done with the theory — wait for the next part where I explain how to use this methodology in practice.

Methodology in a nutshell

Modeling activity inputs

Application workflow

Conceptual data model

Queries and commands statistics

Data realities (Data profiling)

DDD Design: Aggregate definition

Theory first: Building blocks of domain-driven design and definition of aggregate

The aggregate definition process

Mapping the DDD aggregates to the logical data model

Physical optimization