Senior Data Engineer (AI and ML frameworks)

Job Description

Data Standardization and Transformation:

• Convert diverse data structures from various EHR systems into a unified format based on FHIR standards

• Map and normalize incoming data to the FHIR data model, ensuring consistency and completeness

Kafka Integration:

• Consume and process events from the Kafka stream produced by the Data Writer Module

• Deserialize and validate incoming data to ensure adherence to required standards

Data Segmentation:

• Separate data streams for warehousing and AI model training, applying specific preprocessing steps for each purpose

• Prepare and validate data for storage and machine learning model training

Error Handling and Logging:

• Implement robust error handling mechanisms to track and resolve data mapping issues

• Maintain detailed logs for auditing and troubleshooting purposes

Data Ingestion and Processing:

• Use LLMs to extract structured data from EHRs, research articles, and clinical notes

• Ensure semantic consistency and interoperability during data ingestion

Knowledge Graph Construction:

• Integrate extracted data into a knowledge graph, representing entities and relationships for semantic data integration

• Implement contextual understanding and querying of complex relationships within the knowledge graph (KG)

Advanced Predictive Modeling:

• Leverage KGs and LLMs to enhance data interoperability and predictive analytics

• Develop frameworks for contextualized insights and personalized medicine recommendations

Feedback Loop:

• Continuously update the knowledge graph with new data using LLMs, ensuring up-to-date and relevant insights.

Work Closely with Cross-Functional Teams

• Collaborate with data scientists, AI specialists, and software engineers to design and implement data processing solutions

• Communicate effectively with stakeholders to align on goals and deliverables

Contribute to Engineering Culture:

• Foster a culture of innovation, collaboration, and continuous improvement within the engineering team

Qualifications

• Deep understanding of patterns and software development practices for event-driven architectures
• Hands-on experience with stateful stream data processing solutions (Kafka or similar streaming platforms)
• Strong knowledge of data serialization/deserialization using various data formats (at minimum JSON and Avro), and integration with schema registries
• Proven Python software development expertise, with experience in data processing and integration (most of the software is written in Python)
• Practical experience building end-to-end solutions with Apache Flink or a similar platform
• Experience with containerization and orchestration using Kubernetes (K8s) and Helm, especially on Google Kubernetes Engine (GKE)
• Familiarity with Google Cloud Platform (GCP) or a similar cloud platform
• Hands-on experience implementing data quality solutions for schema-on-read or schema-less data
• Hands-on experience integrating with Apache Kafka, particularly the Confluent Platform
• Familiarity with AI and ML frameworks
• Proficiency in SQL and experience with both relational and NoSQL databases
• Experience with graph databases like Neo4j or RDF-based systems
• Experience in the healthcare domain and familiarity with healthcare standards such as FHIR and HL7 for data interoperability

WOULD BE A PLUS:

• Experience with web data scraping

Additional Information

PERSONAL PROFILE

• Strong problem-solving skills, with the ability to design innovative solutions for complex data integration and processing challenges

• Excellent communication skills, with the ability to articulate complex technical concepts and work effectively with various stakeholders

• Commitment to improving healthcare through data-driven solutions and technology

• Stay abreast of the latest technologies and industry trends while continually improving your skills and knowledge

• Ability to work in a collaborative environment, valuing diverse perspectives and contributing to a positive team culture