Apache Kafka : Real World Case Study in Integration and Data Sharing Across Databases and Apps
Many medium to large companies face the problem of legacy systems and applications that have been patched together over the years to accommodate growth and changes in the company.
A typical scenario might be a company started with a few hundred employees has now grown to thousands of employees over the years.
Managing the user data has been patched together usually by manually entering the same data into different applications several times. As we all know, manual data entry often leads to errors that take time to find and correct, not to mention the hours of labor wasted during the actual data entry and edits.
One of our clients, a well-known and respected international construction company, had exactly this problem. How to manage thousands of users’ data across several applications and databases.
The company was founded in 1937 and started with a handful of employees. Of course, this was long before computers were even invented so all of that user data was stored in paper forms.
Jump ahead to 83 years later and the company has grown to over 3,000 employees scattered across the globe.
The client came to us with the following problems related to their user management situation:
The company had a central database where employee data was stored, a master employee database.
The company was manually entering data in the master employee database, as well as Vista Accounting software, Windows Active Directory, a CRM app, an LRM app (Learning Management), Trimble Project, Trimble e-Builder, Project Site, Primavera P3 app, and Skire Oracle.
The whole system was patched together with various legacy scripts, power shell scripts, manual entry, web forms, paper forms, and APIs.
In short, the system was a nightmare to manage, both from a user side as well as from the IT support side.
AECCloud proposed to use Apache Kafka to connect the various databases and apps so they could share data relevant for each application. This ensured that all employee data was entered only once and shared across all applications and databases as needed in real-time.
The proposed updated system, as illustrated below, would address the issues the client was having with their legacy systems as well as add new features they had been wanting to add but was too time and cost-prohibitive due to the spider web of scripts and APIs connecting the apps and databases.
The company, in its move to integrate Apache Kafka, added the following apps and solutions to their user management system:
Microsoft CRM (Customer Relationship Management)
Employee evaluation application
Docebo LMS Platform (Learning Management System)
Trimble Project
Trimble e-Builder
Oracle Skire (formerly known as Primavera Unifier)
ProjectSight (General Contractor Project Management software)
Vista Accounting
WebDAM digital asset management solution
Primavera P3
Notifications to specific users for specific events such as entry success/fail, errors, new record added, etc.
Apache Kafka was a perfect match for this project. As we learned in the previous Apache Kafka articles, Kafka’s strength lies in its distributed publish-subscribe messaging system with extremely high throughput, built-in partitioning, replication, and fault tolerance.
Data could now be entered into one form. That data is now passed on to the various applications and databases with no further user-side action. Additions, deletions, or modifications are made in one form and updated across the various applications in real-time.
Future updates, removing or adding new applications or data fields are now more modular. Each system stands alone and can be modified as the needs of the client change over time.
An application can be removed or added without affecting the other applications. Specific data can be added or removed that are flowing into a particular database or application as well.
The entire system can be visualized as Lego blocks which can be configured in a variety of different ways with relative ease and no or extraordinarily little downtime.
Needless to say, the client is more than satisfied with the final integration. They now have a modern system that can accommodate a wide variety of applications, databases, and data requirements that can be easily modified to fit the current company needs in real-time.
Apache Kafka is an extremely powerful software solution that can turn nightmare data sharing into an easily managed system.
Find out how AECCloud can help your company with its data-sharing issues. Email or give us a call today.