Previously we gave an overview about the system architecture. You learned two things: It is reactive by building upon the Vert.x framework and it features a statically typed Entity Data Model whose data resides within a graph database. Specifically, we claimed that we could query this data directly from the front-end via a type-safe query language in a reactive manner.
Thus, today you will join a database query on its voyage through our stack from the front-end to the back-end and back in a full-circle to the front-end with the requested data. This post will introduce the individual stages required to make this work, such as formulating the query, executing the query and processing the result.
Our journey begins with Jane, who explores a Workroom within our Marketing Project Management solution in her browser. As she wants to view all tasks within the Workroom, our stack must provide this data.
First, look at the whole JanusGraph database to get an overview:
Our graph contains several entities, connected via relations. Each element has a type (written between angle brackets) and a generated ID. Moreover, elements may also have attributes.
Now, the front-end controller specifies a query that navigates the graph in a natural way:
let query = Query.fromEntitiesOfType(WorkroomType.instance()) .restrict(R.id(42)) .followRelationsOfType(TaskForWorkroomType.instance()).build();
In plain English, this reads:
The following diagram visualizes our walk through the graph. Visited elements are colored in green:
If you notice a similarity of our query language to established graph traversal languages, you hit the mark: Our query language builds upon the Gremlin language from Apache Tinkerpop. However, tightly coupled with our Entity Data Model, it adds features to better integrate with our technology stack. For example, we want to ensure that our queries are syntactically correct so we enforce this with our query language by using a Builder pattern. We raise compile-time errors if the used types do not match. Furthermore, it is aware of type inheritance, has a built-in security mechanism, and can fetch whole graph structures in one go.
The data our query needs to fetch resides in our Java-based back-end land. Thus, we need someone who can translate our Typescript-based query into a lingua franca – a common language that the back-end also understands. More specifically, the query needs to become a serializable Statement that consists of Steps. These data structures can transform into a JSON representation that can be interpreted on both the front-end and the back-end.
Luckily, our typed front-end query object can translate itself into an untyped but serializable Statement. Its steps look something like this:
If we wrote the same query in Java, it would look syntactically different, but result in a Statement of the same structure. Thus, turning a query into a Statement is a one-way transformation that takes place entirely on the client side.
As our query has now become a Statement, it can set sail towards the data repository as a JSON representation via the Vert.x event bus. Meanwhile, our controller will stay available for other demands, since the event bus will notify it asynchronously once the query yields results.
Our back-end endpoints strictly live by the old saying “Never trust travelers from front-end land”.
After all, our query language gives the front-end much flexibility for retrieving data. Hence, even if Jane became evil and tampered with the query on the front-end, she still only must be able to retrieve the data she is entitled to see.
Therefore, the endpoints will add some trustworthy information to the Statement:
Next, the back-end endpoint will forward the Statement to the persistence component for processing. It uses Apache Tinkerpop with the Gremlin Graph Traversal Machine to operate on data within a JanusGraph database. Check out the tutorial on tinkerpop.apache.org to see how a “Gremlin” traverses a graph.
Unfortunately, ordinary Gremlins do not understand our Steps nor know about concepts like type-safety and security. Thus, the persistence component optimizes the query, translates it into the Gremlin language and embeds the advanced concepts of our query language before its minions do their work.
Our persistence component’s translator automatically transpiles our Statement into a Gremlin statement like this:
g.traversal().V().has(“ID”,42).has(“$type_workroom”,”workroom”) .where(__.out(“personContributesToWorkroom”).has(“ID”,23)) .out(“taskForWorkroom”) .path();
Even without knowing Gremlin, you will recognize the graph navigation pattern that we defined in our initial query. However, the translator applied some changes:
Firstly, the translator realized that it could utilize an index on the ID field to narrow down the set of starting elements to a single Workroom. Unlike our initial statement, it puts this restriction prior to the type check in line 1. This saves Gremlin from scanning the whole graph for starting points, which is essential to ensure scalability. Your key takeaway: “Always be kind to your Gremlins and give them a shortcut to their starting points”.
Secondly, the where clause in line 2 wraps another traversal that ensures that only elements visible to the user can be visited. It checks if Jane (ID 23) is actually a member of the given Workroom. Depending on the security configuration and optimization possibilities, our translator may embed multiple traversals such as this.
Summed up, depending on the complexity and nature of the statement, the translator will automatically add Gremlin code that respects our stack’s features. Especially for modifying statements, it has to consider cascading deletions and constraint validation.
Every time a Gremlin has found a path through the graph, it immediately pushes the individual entities and relations found onto the event bus.
Meanwhile, the front-end is ready to receive entities and relations from the event bus. Thanks to Angular data binding, as soon as the first Task entity arrives, Jane will immediately see it on the UI.
As the front-end receives whole paths, it assembles a navigable Object Graph of connected entities:
While this may be overkill for our simple example, imagine the real-world scenario of a Workroom’s task list: For each task, we want to display data that is stored in individual entities like the task’s assignees, the comments as well as the files that are involved in completing the task. Our graph language allows formulating this as a single complex query that the back-end might split into multiple queries to ensure reactivity. However, the front-end only has to consider a single query and displays data as it becomes available.
This experience is what we call “Full-stack reactive”: Users can already interact with partial results while the front-end still processes their request.
For CELUM’s reactive stack, the journey has just begun and many new features are still to come. In contrast, as Jane retrieved her tasks, our query’s quest ends.
Congratulations! You are now familiar with the basic principles of reactive architecture, and have learned about what is necessary to provide data to the front-end via graph queries. Future articles will cover the specific stages in more detail.
As you now know what we work on at CELUM, listen to Will about how we work at CELUM.