Revamping Data Pipeline Infrastructure to Increase Owner Satisfaction at Twiddy

In an ever-evolving technological landscape, embracing new methodologies is vital for enhancing efficiency. Our data and analytics interns recently undertook a significant overhaul of one of Twiddy’s data pipeline infrastructures, implementing Airbyte pipelines with Kestra orchestration to replace an existing Java application. Motivated by several challenges with the previous system, most importantly a complete loss of data transmission to their Owner Dashboard, this transformation has led to improved data connection management, greater reliability, and increased accessibility and efficiency.

The Old: Dependency on Java Application

Data Connections

Twiddy’s prior system depended on a Java application that managed connections to various data sources like BigQuery (BQ), PostgreSQL (PG), and Microsoft SQL Server (MS SS). These were responsible for key business areas such as the owner dashboard, guest profiles, and guest reservations.

Challenges with the Java Application

• Complexity and Readability: One of the most daunting tasks was navigating the application’s complex and dense structure. Without adequate documentation, deciphering the functionality of each section was cumbersome. Additionally, this was the only Java code in the entire Twiddy ecosystem, creating an impediment for engineers who worked primarily with other applications.
• Lack of Alignment with Data Processing Needs: Despite being written in Java, the application did not cater to specific data processing requirements. Inconsistencies in parsing SQL code and transmitting pipelines made it apparent that the system was not designed with data professionals in mind.
• Stability Issues: With key components like the owner dashboard malfunctioning for months and a lack of proper documentation, the system’s interdependency led to frequent breakdowns, causing considerable challenges in maintaining the application.

The New: Implementing Airbyte Pipelines with Kestra Orchestration

Decoupling the System

Our approach for Twiddy emphasized a decoupled structure, moving away from the former Java application. If one part breaks, it doesn’t impact the rest of the system, making it easier to identify and resolve issues. Each part of the ELT (Extract, Load, Transform) process has been modularized so that a data analyst can easily create new connections to disparate destinations without any input needed from the development team.

Cloud Orchestration

Previously confined to a local server, our implementation of Airbyte pipelines orchestrated by Kestra has now shifted everything to the cloud. This move has brought unparalleled flexibility, enabling anyone in Twiddy’s organization to run the system as needed, with the added benefit that all connectors are free.

Automation

Kestra has enabled seamless integration and execution of complex tasks without manual intervention. The benefits of this automation extend beyond mere time-saving; it promotes consistency, reduces the possibility of human errors, and allows for scalable and flexible orchestration according to the specific needs of a task or system. By embracing Kestra’s automation features, Twiddy can now achieve a more efficient, reliable, and streamlined process, aligning with modern demands for agility and responsiveness in a data-driven environment.

Efficiency

By eliminating the dependencies on the old Java application, we helped Twiddy drastically reduce the time required to implement the pipelines, thus achieving a more streamlined process.

Conclusion

Twiddy’s transition from a Java-based application to Airbyte pipelines with Kestra orchestration marks an essential step towards modernity and efficiency. By addressing the previous system’s drawbacks, including challenges with readability, alignment with data needs, and stability, we have forged a more resilient and adaptable infrastructure. This improvement not only enhances Twiddy’s internal operations but also re-establishes the industry-leading owner experience they have become known for.