Principal Backend Engineer

Role : Principal Backend Engineer

Location : India (remote)

Type : Full time

ROLE OVERVIEW

We are seeking a Principal Backend Engineer to own the scalability, reliability, and architectural evolution of an AI platform as it transitions from a prototype to a production-grade, cloud-native platform on Google Cloud Platform (GCP). This is a high-impact, high-ownership role at the intersection of distributed systems engineering, Kubernetes-based platform engineering, and applied AI.

The engineer will be expected to design and operate backend services, platform capabilities, and infrastructure primitives that enable secure, scalable, and reliable AI application development across internal and customer-facing environments.

REQUIRED SKILLS & EXPERIENCE

• 10+ years of backend engineering experience with strong focus on distributed systems, production-grade platform design, and cloud-native architecture.
• Expert-level Python proficiency; deep hands-on experience with FastAPI, SQLAlchemy, Pydantic v2, and async/concurrent programming using asyncio, ThreadPoolExecutor, or equivalent concurrency patterns.
• Proven experience designing and operating microservices at scale, including service decomposition, inter-service communication, failure isolation, API contracts, and service ownership models.
• Strong PostgreSQL expertise, including schema design, indexing, query optimization, migration management, and multi-tenant isolation using row-level security, schema-per-tenant, or equivalent tenancy patterns.
• Hands-on GCP production experience, including Compute Engine, Google Kubernetes Engine, Cloud SQL for PostgreSQL, Cloud Storage, Cloud Load Balancing, Cloud IAM, Secret Manager, Cloud Logging, Cloud Monitoring, and ideally Vertex AI.
• Strong Kubernetes experience, preferably with GKE, including workload deployment, services, ingress, autoscaling, config/secrets management, health probes, resource limits, rolling deployments, and production troubleshooting.
• Experience with container-based deployments using Docker, Docker Compose, Kubernetes manifests, Helm charts, and cloud-native service orchestration patterns.
• Deep understanding of event-driven and async architectures, including Pub/Sub-style messaging, background workers, polling workers, idempotency guarantees, retry strategies, dead-letter queues, and backpressure handling.
• Experience integrating LLM APIs in production, including OpenAI, Anthropic Claude, Google Gemini, Mistral, or Vertex AI-hosted models, with focus on rate limiting, structured output enforcement, prompt/version governance, observability, and multi-provider failover.
• Strong API lifecycle management skills, including OpenAPI 3.1, API versioning, backward compatibility, deprecation policies, schema governance, and platform-level API standards.
• Practical knowledge of distributed system patterns, including sagas, circuit breakers, idempotency keys, at-least-once delivery, retries with exponential backoff, graceful degradation, and eventual consistency trade-offs.
• Security engineering fundamentals, including OAuth2, JWT, RBAC, CORS, tenant-aware authorization, secrets management, workload identity, service accounts, and cloud IAM role design.
• Experience building platform capabilities such as developer portals, internal service templates, deployment workflows, environment provisioning, CI/CD automation, observability dashboards, and self-service tooling for engineering teams.
• Prior experience developing AI/ML or developer platforms that expose reusable backend primitives such as model gateways, evaluation workflows, sandbox environments, dataset management, feature flags, tenant configuration, and audit/logging services.
• Prior experience building developer platforms, internal tooling, or AI/ML serving platforms at large-scale organizations, with emphasis on reliability, governance, multi-tenancy, and operational excellence.