Open Manta Designer
Data lineage segment is a rapidly developing field that enables large corporations with extensive and complex data architectures to create impact analyses, improve data quality and identify data dependencies.
One of the leading companies in this technology segment is Manta Tools offering a world-class data lineage platform that automatically scans your data environment to build a powerful map of all data flows and deliver it through a native UI and other channels to both technical and non-technical users.
In August 2022, we started cooperation with Manta Tools on developing the so-called Open Manta Designer, which is a graphical software tool that allows Manta clients to enrich data lineage metadata that is automatically created by Manta scanners and thus increase the flexibility of the entire Manta platform. Coding Bear was selected as the company that should deliver the product end-to-end, from initial analyses to a thoroughly tested ready to use product.
Nine months later, we delivered the product and it became a part of the official Manta R42 release.
“Coding Bear has been a great partner to Manta as they assist with a very important project we call the OpenManta Designer. This project entails the creation of a graphical user interface that allows less technical users to use drag and drop techniques to model, design and fulfill their understanding of how data flows through their organizations. Coding Bear works closely with Manta Engineering and Product Management to deliver, on time, effective code that meets the requirements and specifications, including creative approaches to tackle difficult problems. Coding Bear has exceeded all expectations and Manta confidently recommends Coding Bear for critical projects, especially in the data and user interface domain.”
Ernie Ostic
Chief Evangelist at Manta Software and former SVP of Products
Solution Overview
In the fast changing environment of Data Lineage and given the rapid development of the Manta platform itself, it was clear from the beginning that we needed to create a tool that allowed for flexible customisation and extensibility in the future.
That’s why the behavior of the entire visualization and design tool is controlled by easily editable configuration data without the need to change code. When Manta develops a new type of data source scanner, the necessary adjustments in Open Manta Designer are literally a matter of minutes. Similarly, it is not a problem to immediately start supporting new technologies for which Manta currently does not have automated scanners.
Since a significant part of the entire tool is visualizing nodes and creating data flows between them, we did a thorough extensive analysis and PoC of existing visualization libraries that we could use. Of all the possibilities, we finally chose the Konva.js library, which allows for great flexibility, although it means that many features that other high-level libraries offer in the base must be implemented by ourselves.
This turned out to be a suitable choice from a performance point of view as well.
Some of the Open Manta Designer features:
- ability to create custom Node hierarchies depicting data sources (various DBMS, ETL and reporting tools, programming languages etc.)
- flexible creation of data flows among the data sources simply by drag & drop at drawing canvases
- support of multiple drawing canvases with automated synchronization of data, support of undo and redo
- full integration with Manta platform including import and export of custom metadata
- SAAS ready solution
Open Manta Designer provides a graphical user interface for users to build lineages to augment Manta's out-of-the-box scanners. To reduce the barrier to entry for end-to-end lineage, OpenManta Designer allows users to integrate custom assets faster and more effectively.
OpenManta Designer also creates manual data lineage using a drag-and-drop interface and/or a spreadsheet, allowing users to draw connections between related items.
How did we delivered the solution
The project started with a more detailed specification of the required functionalities and the design of the overall architecture of the solution. As the requirements were specified in greater detail including the design of the user interface, this phase lasted approximately 4 months from September to December 2022. The output was a complete specification, including the prioritisation of requirements and detailed estimates of the effort required to implement them.
The agile development of the solution began at the turn of the year, with the first significant milestone in April 2023, when the first MVP solution was delivered to selected Manta clients as part of Manta release 41. These clients from the ranks of larger US corporations provided initial feedback and confirmed the assumption that the entire project team is moving in the desired direction and that a similar tool will make it easier for end users to manage metadata.
For the next 4 months, we have been implementing other advanced functionalities and incorporated feedback on the first MVP version.
We delivered the final product at the end of August 2023 and it became part of the official version of the Manta R42 as originally planned.
Q&A with Ernie Ostic
Interview with Ernie Ostic, Chief Evangelist at Manta Software and former SVP of Products
Who is Manta?
Ernie Ostic: Manta is an independent software vendor and market leader in the data lineage space. We support customers whose needs include (but are not limited to) cross-system impact analysis, information governance, and faster and more effective application migration. Manta’s Unified Lineage Platform analyzes source code and delivers detailed analytics on the data pipelines represented by corporate systems.
What was the main objective of the project?
Ernie: Manta also provides the ability to creatively add your “own” lineage to the Unified Lineage Platform. This means having the ability to define your own processes and their data flows, or add value (new characteristics) to flows that Manta has already analyzed. The methodology for doing this utilizes .csv files at its core, and for certain kinds of users, this can be difficult to implement. The challenge was to build a graphical user interface to allow less technical users to more easily generate their own lineage assets and create their own data pipelines and dataflow specifications.
How do Manta customers benefit from the solution?
Ernie: Coding Bears was asked to design a user interface that would ultimately generate the .csv files needed “under the covers” for Manta to ingest the lineage artifacts constructed by less technical users. Generally these users are architects and data analysts who are somewhat technical, but aren’t formal developers. This meant coming up with a model for the UI design that would adapt to the Manta architecture while also providing flexibility for model creation while conforming to existing Manta APIs. When finished, the project that Coding Bears is working on (the OpenManta Designer) will enable Manta customers to extend their lineage solution to legacy technologies, new in-house applications, and current and future new web based applications.
How was the cooperation between Manta and Coding Bears?
Coding Bears is an independent entity, but work together as though they were directly part of the team. They have great interaction with the Manta Engineering and Product Management teams, participate together in all Sprints, and provide excellent and timely feedback to all requests. This includes current coding, but extends back to the beginning of the project when the teams were going back and forth to understand the specification and come up with the initial design.
Coding Bear has been a great team to work with. Creative, hard-working, dedicated, and fully embracing Manta values and methodology.
What were the technology challenges?
Technology challenges include, but are not limited to:
- designing creative ways to represent existing Manta models in a way that users can customise, including rules for specifying the behaviour of assets in the model and how they interact with each other.
- selection of appropriate UI controls, and also the selection of the right library for the OpenManta Designer as well as future Manta offerings
- experimentation with various graphical display functions critical the project, such as dragging and dropping curved lines between objects
