Senior Python Kafka Engineer DRS Platform Modernization || Boston MA/Remote

Role :: Senior Python Kafka Engineer DRS Platform Modernization

Location :: Boston MA/Remote

Duration :: long terms

Responsibilities

We are seeking a hands-on Senior Software Engineer to support the enhancement of an event driven digital repository platform. The role will focus on improving observability event lineage workflow reconciliation and Kafka delivery reliability across a distributed microservices ecosystem built primarily on Python and Kafka The engineer will work closely with the clients internal engineering team to implement solutions in a live production environment

Primary Skills Mandatory

Python Development

Strong experience developing and maintaining production grade Python applications

Understanding of concurrency multithreading and asynchronous processing

Experience with API integrations and distributed systems

Familiarity with Python testing frameworks PyTest preferred

Apache Kafka

Handson experience with Kafka in production environments

Strong understanding of

Consumer groups

Offset management

Partitioning strategies

Retry mechanisms

Deadletter queues

Message delivery guarantees

Consumer rebalancing and troubleshooting

PostgreSQL

Strong SQL skills

Experience with PostgreSQL performance tuning

Experience using JSONJSONB capabilities

Data modelling and query optimization

Observability Logging

Design and implementation of structured JSON logging

Log standardization across distributed services

Monitoring ing and troubleshooting best practices

EventDriven Architecture

Understanding of distributed systems and event driven design patterns

Experience with idempotency transactional processing and messaging reliability

Secondary Skills Preferred

OpenTelemetry

Distributed tracing implementation

Instrumentation of microservices

End-to-end transaction visibility

Open Lineage Data Lineage Solutions

Event lineage tracking

Workflow traceability

Data movement and process visibility

Kubernetes

Working knowledge of containerized workloads

Deployments ConfigMaps Secrets scaling and troubleshooting

GitOps Deployment Tools

ArgoCD

Kustomize

CICD deployment practices

Monitoring Platforms

Grafana

Loki

Metrics and dashboard creation

Key Responsibilities

Implement structured logging standards across Python based services

Enhance Kafka consumer and producer reliability

Improve retry recovery and dead letter handling mechanisms

Design and implement end-to-end event lineage and tracing capabilities

Develop reconciliation and completion tracking mechanisms to identify incomplete workflows

Analyse and resolve production issues related to event processing and workflow execution

Collaborate with platform engineers to improve observability and operational excellence

Contribute to architecture discussions and implementation decisions related to distributed systems

Experience

8 years of software engineering experience

3 years working with Kafka based distributed systems

Strong hands-on Python development experience

Experience working in Kubernetes based environments

Experience with observability and monitoring implementations

Skills

Mandatory Skills : Python