State Government, Education

Unlocking Data Insights for a State Education Agency

Data is an integral part of a state education agency’s ability to enable academic success and evaluate how various programs support that initiative. This reporting environment requires an intricate network of software and processes to centralize, organize, and protect the information. However, historical data can often live in a cumbersome patchwork of sources, requiring manual entry and reporting. Further, the data becomes siloed without a centralized system, complicating data storage and retrieval. So, when a state education department received a federal grant to improve its data-reporting capabilities to unlock key insights, they turned to UDig for the solutions they needed. After carefully analyzing the data they held, we helped them develop a data repository that streamlines how they operate — and modernizes how they work. 



A segmented data environment made it difficult for a state education agency to respond to new requests for data insights on academic success created by changing legislative requirements.


Centralize and organize hundreds of varied data sources to enable our client to provide on-demand and predictive analytics for evolving data needs.


A data-enabled agency that can leverage predictive analytics to provide its school districts, students, and parents with data-driven information that fosters informed decisions.

Challenge: Segmented Data Environment Hindering Legislative Reporting Requirements

Our client had 13 years of data they were managing across 3,000+ data sources. Despite having this robust amount of data, they had difficulty responding to legislative reporting needs, causing delays and inaccuracies. Their legacy data was siloed, creating a cumbersome process for storing and retrieving data. Further, the data sources were all varied in quality — raw, aggregated, clean — making it nearly impossible to know which data version employees were processing.

This existing data environment meant they: 

  • Lacked a cohesive repository for storing and managing data 
  • Relied on laborious manual reporting processes across various teams and departments 
  • Were unable to quickly respond to governmental reporting demands 
  • Were unable to conduct effective data profiling  
  • Lacked transparency into their data 

Strategy: Modernize and Streamline Data Reporting Capabilities

Our initial priority was helping our client meet state-legislative reporting requirements. The agency needed a data environment that enabled them to seamlessly provide on-demand and predictive analytics. Also, at any point, their reporting requirements could change, meaning their solution had to enable their teams to swiftly respond to any new criteria. To support this dynamic capability, the state education agency needed a data mart that could become the best source for any descriptive reporting on academic success. To do so, we worked in two phases: 1. Discovery and 2. Implementation.

Phase 1: Discovery 
To meet our client’s goals of upgrading and centralizing their data environment, we first had to do a data deep dive. We worked with their team to profile their data and organize it, so we understood what they had, what was valuable, and what they no longer needed. Further, we needed to identify how the data must work for them, so we could define the best build approach to meet their needs. 

Phase 2: Implementation
Once we conducted this discovery phase, we began building their data platform. We had to ensure our solutions met these goals: 

  • Create two data structures to consolidate their thousands of legacy data sources. They needed a data lake to organize historical and ongoing data sources, as well as future operational reports. They also required a relational model that contained non Ed-Fi data and Ed-Fi–based data store and allowed reporting across these two systems.
  • Design a data-ingestion process to load each of the legacy data sources into the new data structures. This process had to be seamless while enabling them to meet future reporting demands.  

As we worked with our client, we became a steadfast partner to our stakeholders and helped represent the business interests of how and why to best implement each solution.  

Outcome: A Data-Enabled Agency Empowering the Public with Actionable Information

Today, our client is a data-enabled agency that can swiftly and accurately report education-centric data at any moment. Their cloud-based data repository centralizes and stores all data, providing a cost-effective solution for predictive-reporting capabilities around academic success. With more data visibility, they can turn insights into actionable strategies. Further, thorough data profiling capabilities enable their team to better understand their data sources and how to best organize the information. They also have purpose-driven data warehouses focused on specific subjects, which they can further build out, as needed. This environment has optimized their robust analytics and data science capabilities and set them up for future demands. By building this foundation, all stakeholders — from the parents to legislative bodies — have more visibility into academic success, driving more informed decisions and actions.

How We Did It

Advanced Analytics & Visualization
Assessment & Roadmap
Data Engineering
Data Profiling
Data Quality
Solution Architecture

Tech Stack

  • Azure Data Lake
  • Azure Machine Learning
  • Azure SQL
  • Azure Synapse Analytics
  • Python
  • SQL