Industry-leading fintech company to migrate from enterprise data catalog to modern data discovery
A modern fintech company built an application as a money management platform. The platform enables users to earn, save, spend, invest, and donate money from an all-in-one mobile app.
Challenge
Legacy enterprise data catalog not working as expected
Initially, the company used a legacy enterprise data catalog to gain visibility in a centralized location for lifecycle management. The initial deployment took approximately three months, requiring support from the infrastructure team and way more manual setup than expected. As a mainly on-premises tool, the legacy enterprise data catalog lacked stability and was not always up-to-date. After a significant upfront investment of resources and capital to get the enterprise data catalog in place, the company soon realized there were some major shortcomings outlined below.
Lacked Visibility Into Pipelines
The company uses dbt for its transformations, causing many temporary tables. The legacy tool created a lot of noise across the different tables. While the previous data catalog provided data lineage for Snowflake, the visibility was limited to only the table level. The lack of accurate column-level lineage made it challenging to track exactly where data was coming from or determine the impact of changes to their data model.
Inability to Connect with Tableau Online
Like many modern organizations, the company was using a cloud-based business intelligence solution, Tableau Online, to enable reporting and analytics for end users. While the previous legacy data catalog supported Tableau Server, it failed to provide stable support for Tableau Online. This disconnect from Tableau created additional lineage flow visibility issues because users could not understand how the data models impacted the Tableau workbooks and dashboards. This limited the end-user adoption rate since the tool failed to meet the business intelligence team’s use case.
Without a reliable connection to their Tableau Online instance, the efficacy of the previous data catalog was severely reduced. Since most of their data consumption was happening on the BI layer, the data catalog could not provide visibility into who was consuming which data, what the most popular datasets were, or help end users find the right workbooks and dashboards.
Limited End-User Adoption
Although the company wanted to democratize data ownership, it had to limit the number of super user licenses due to costs, limiting how many people could own documentation. While people could still search for documentation and send it to others, few people could create/modify descriptions or be designated as owners of particular data assets. The inability to crowdsource information and share the burden of documentation as well as upkeep meant maintaining the data catalog became a burden for the few data team members with the right permissions.
The limited functionality for most of their users, on top of spotty lineage and unreliable visibility into data consumption, prevented the data catalog from becoming a key part of daily workflows for anyone beyond the data team. And even then, the data team struggled to get all the information they needed from the tool. With a high price point in the hundreds of thousands of dollars for the data catalog, the company needed to see high adoption across their different stakeholders to get the right ROI from the tool - but it soon became clear that wasn’t going to be the case.
“While we’d rate our previous data catalog as a 6/10, Select Star is a 9/10, giving us the features we need and an expanded set of use cases.”
Data Architect
Solution
Automated, cloud-native solution built for the Modern Data Stack
In response to these challenges, the data team sought a modern data discovery solution that was purpose-built for working with modern data stack. The new solution provided many benefits including…
Reduced Operational and Maintenance Costs
The company focused on Select Star’s cloud-native service to eliminate time spent managing infrastructure or waiting for new releases. With Select Star, the company could sync its data to Select Star within a few hours, on their own. Further, Select Star’s highly automated platform reduced the amount of manual work required, thanks to features like propagating tags and auto-documentation based on lineage.
Scalable Pricing Model
To democratize data effectively across the organization, the company wanted a solution with scalability, moving away from paying for each individual data manager. Select Star’s pricing model doesn’t distinguish between admins, data managers, or users with view-only privileges, providing the flexibility and scalability needed as the company plans to expand its engineering team to over 200 users. This model also allows more users to participate in data management by crowdsourcing descriptions and spreading data ownership across more stakeholders, reducing the burden on the data team.
Automated Visibility into Lineage and Popularity
Full visibility across all data and connectors was a key requirement. Select Star’s native integrations with Tableau Online, Snowflake and dbt cloud provided a reliable connection to their key data platforms. The company was able to gain visibility into data lineage, at a column level, from their raw data in Snowflake down to specific views and dashboards in their Tableau workbooks. Select Star provided other automated insights, like popularity and usage for all of their data assets, which allowed the data team to understand how data was being consumed across the organization.
The company plans to use Select Star’s lineage to automate tagging for all personally identifiable information (PII) data downstream from their source tables. This capability will enable them to track sensitive information across their data model and mask values in queries where PII may be present.
“People are more involved. Beyond the data engineering team, we’re seeing many other teams start to log in to find and use data."
Enabled Change Management
Select Star’s visibility into lineage has enabled more robust change management process. The company leverages Select Star’s capabilities for downstream impact analysis for business intelligence and batch and dependency tagging. The company can build in alerts as part of its code review flow. When someone changes a data structure, the analytics engineer gets a notification, then can review the potential impact by looking at the lineage in Select Star.
Result
High adoption and immediate impact for the data team and beyond.
Data Models They Could Trust
Select Star’s intuitive interface enables more people across the company to leverage data and analytics, providing collaboration features within the system. For example, the popularity dashboard makes self-service data discovery easier for business users and ensures they can trust the data models. The company plans to add more descriptions at a more granular level to supplement what Tableau allows so that people have a better understanding of the data and how it’s used.
With more users, the organization plans to decentralize ownership so that more teams can own assets. Within the first few months, the organization has plans to leverage Select Star’s tagging functionality to develop curated data sets across different services/domains and assign the relevant data owners to ensure metadata is accurate. The company has more people involved in the data process, and thanks to Select Star, less time is wasted trying to find or understand the data at hand.
Informed Decision-Making That Amplified ROI
Select Star enables clear visibility into column lineage between Tableau and Snowflake. This improved their confidence when making changes and understanding the impact they would have on their data model. Combined with the simple and easy-to-use interface, this visibility elevated the business user’s data experience and enabled better data-driven decision-making.
Select Star also enhanced visibility for business users into what data exists already and the exact context behind it. This enables them to ask for changes or for new views of data that help them leverage the analytics more effectively.
Combining high adoption across a broad set of users, improved visibility into their data platforms and usage patterns, and a more economical pricing structure, the company has realized an immediate return on its investment in Select Star. By delivering reduced storage costs, saving time spent on data-related questions, and increasing trust in their data, Select Star has helped the Company realize the advantages of a modern data catalog.
