
eMondrian: Relational OLAP Server for Real-Time with ADOMD.NET and Excel Support
In response to the client's request for the optimization of the weekly reporting process, the "Input Forms" project was initiated. The goal was to increase the convenience and aesthetics of reporting while maintaining the familiar Excel workflow. As a result, a service was developed that allows for data upload and processing, transforming it into visual analytics for management.
Initially, the "Input Forms" service enabled employees to continue reporting in Excel by uploading and downloading files to a special data processing service that converted them into analytics for the management dashboard. The project became dynamic and customizable, with automatic data calculations, except for the report configuration, which was built on standard JSON. The service included type definitions, formatting, and constraints, allowing any employee to generate reports, mastering the system in just a couple of days.
Currently, the service is fully implemented; its main structure is frozen, but we continue to optimize and enhance it. Due to the variety of report types, there was a need for a management system, leading to the development of an admin panel for all microservices. The admin panel is built on RBAC architecture and covers all necessary needs; plans are in place to transition to an attribute-based system.
The "Input Forms" service has become a starting point for creating a series of dashboards that provide clear data visualization. The Echarts library, known for its reliability, was used for displaying charts. At this point, we have decided to use Clickhouse as the database for our data warehouse.
At the beginning of development, we did not anticipate a large data flow and used AirFlow as the orchestrator. It transferred data from the input forms database to a storage system that at that time consisted of PostgreSQL. However, with the increasing volume of data from the HR system 1C, we decided to apply a different technology, which led to research on DBT technology. This resulted in choosing Dagster—a readily integrable open-source solution that paired excellently with DBT for our needs.
The reporting service is completely secure: data resides within a closed environment, and passwords are encrypted in a separate service. An effective alerting system is set up to notify about failures, with tasks automatically created in Jira. All authentication occurs through Keycloak.
Visually, the service appears simple due to the complex technologies used "under the hood." After testing various technologies, we concluded that there was a need to use our own LakeHouse based on S3. We began using Trino along with Apache Iceberg, which allowed us to efficiently store raw data and significantly reduce its volume.
Currently, we aim to reduce the use of Python in DWH development as it is not fast enough for data calculations. We are gradually moving towards mastering the GO programming language for writing servics (according to client requirements).
As a result, the reporting process has significantly improved, providing convenience and accessibility for users. The created service not only simplified data upload and processing but also became the foundation for further development of analytical tools within the company. Transitioning to DBT allowed us to effectively manage growing data volumes, while using modern technologies like Trino and Apache Iceberg ensured reliable storage and processing of information.
The project continues to evolve: we are focused on optimizing existing processes, improving the user interface, and implementing new functionalities. Ultimately, our services will become a powerful platform for managerial decision-making.