Data Engineer

Overview:

• We are seeking a Senior Software Engineer – Data Marketplace to help build a modern, enterprise-scale data platform that makes data easy to discover, understand, and consume. This role focuses on developing scalable services, search and catalog capabilities, and AI-powered data experiences.
• This is a hands-on individual contributor role working across cloud infrastructure, data platforms, and advanced search/ML systems. You’ll collaborate with cross-functional teams to deliver high-impact solutions that improve how users find and use data.

Qualifications

• 5+ years of professional software development experience
• Strong proficiency in Python (including testing frameworks like pytest/unittest)
• Experience building and deploying applications on AWS
• Hands-on experience with Terraform for Infrastructure as Code
• Experience with Amazon OpenSearch or Elasticsearch, including search/query capabilities
• Familiarity with AI/ML concepts such as embeddings, semantic search, or model integration
• Experience working with serverless and cloud-native architectures (e.g., Lambda, API Gateway)
• Strong understanding of data pipelines and distributed systems
• Experience with version control (e.g., GitHub) and documentation tools

Nice to Haves

• Experience with vector search, RAG (Retrieval-Augmented Generation), or LLM-based systems
• Exposure to data catalog or governance platforms (e.g., Collibra)
• Experience with event-driven architectures (e.g., Kafka)
• Familiarity with additional AWS services such as Lake Formation
• Experience working in large-scale data platform or marketplace environments

Responsibilities

• Design, develop, and maintain Python-based services, APIs, and data pipelines
• Build and manage AWS infrastructure using Terraform
• Develop and enhance search and discovery capabilities using OpenSearch
• Implement AI/ML-powered features, including semantic search and data retrieval
• Collaborate with engineering, data, and security teams to deliver end-to-end solutions
• Work across AWS services such as Lambda, API Gateway, Glue, S3, DynamoDB, IAM, and VPC
• Improve system reliability through testing, CI/CD, and observability best practices
• Contribute to architectural decisions and technical standards
• Mentor junior engineers and share best practices
• Solve complex problems with ambiguous requirements and propose scalable solutions