Designing & developing a public health solution on a modern cloud platform requires deep expertise in the healthcare domain and technical cloud infrastructure. Every country has always prioritized public health, but the COVID-19 pandemic has reinforced this. "The urgency has never been greater, especially after COVID-19 highlighted the need for real-time data insights in public health decision-making," says Milind, a seasoned principal data/solution architect. "In public health, the sooner officials make a decision, the more lives they can save on the ground." " So for me and my team, the challenge was to ensure our solution should be highly scalable, durable and reliable."
Ensuring the cleansing and transformation of raw data received from hospitals for analysis posed a significant challenge. Healthcare domain knowledge and the ability to discern which data points require filtering and which should proceed to the next stage of data analysis are essential for this raw stage. "Data quality remains a significant issue in public health, prompting me and my team to dedicate significant time to designing and constructing a rules-based dynamic data quality layer that is easily reconfigurable when necessary."
Milind shares an impactful public health project wherein he acted as the lead Principal Solution Architect/Data Engineer. He stands as a key player in designing, architecting and implementing the state-of-the-art solution that would potentially act as the central reporting tool for all public health needs. His core technical expertise proved to be pivotal in building a modern solution that is capable of processing petabytes of data on a daily basis.
Healthcare providers typically maintain legacy systems in the field. These systems store data in various formats, including flat files, JSON, XML, and specific healthcare formats like HL7v2. To design a process that includes the processing of such heterogeneous datasets at volume requires a thorough knowledge of cloud services and a deep understanding of data engineering. "Initially, we started with two different datasets. One set of data involved processing terabytes of daily flat files," Milind explains. "I designed an event-driven proof-of-concept on the cloud to process this data end-to-end. Another involved processing over 3.5 million complex JSON transactions daily - a true test of our data engineering skills."
Milind's deep knowledge of modern cloud-based platforms was helpful in deciding on the appropriate platform to analyze these vast amounts of data sets. "With so many modern solutions available on the market, not every platform has the potential to cater to our needs. I evaluated different platforms available based on pros and cons and eventually chose a platform for all our current and future needs," as Milind pointed out. This project was a defining moment in Milind's career because the impact of this solution went way beyond what he ever imagined. "This project gave me purpose because the decisions I made while building this complex ecosystem will effectively serve public health needs on the ground. There is no better satisfaction than knowing your work will save and improve people's lives."
Milind, an expert in his cloud analytics field, is determined to keep applying his skills to improve the public health infrastructure and vows to continue contributing to the open source healthcare community in general. He is a member of HIMSS (Healthcare Information and Management Systems Society). "Modern cloud platforms hold immense potential to revolutionize public health," Milind concludes. "By leveraging these technologies and fostering collaboration, we can build a more resilient and data-driven public health system for everyone."
