Mid-Level Data Scientist - AI (Remote)

Verified Job On Employer Career Site

Job Summary:

PFF is a leading sports analytics company that transforms complex football data into powerful insights. They are seeking an experienced AI Data Scientist with deep expertise in Large Language Models (LLMs) to enhance the performance and reliability of generative AI systems across their consumer and B2B products.

Responsibilities:

• Design robust evaluation frameworks for LLMs (e.g., hallucination detection, factual accuracy, prompt robustness, calibration).

• Use model tracing tools (e.g., OpenAI trace logs, LangSmith, Weights & Biases) to analyze model behavior and failure modes.

• Lead or assist with LLM distillation projects, compressing large foundation models into smaller performant versions fine-tuned on football-specific domains.

• Create and test structured prompting strategies, build RAG pipelines, and implement safety/guardrails.

• Curate and synthesize football-specific data and knowledge graphs for fine-tuning, evaluation, and few-shot performance.

• Deploy dashboards and alerts to monitor drift, toxicity, bias, or degradation in real-world usage.

• Collaborate with engineers, designers, and product teams to bring LLM-powered features to life in production.

Qualifications:

Required:

• 3+ years experience in applied AI/ML roles with at least 1–2 years focused on LLMs or foundation models.

• Deep understanding of transformer-based models, their limitations, and evaluation strategies.

• Proficiency with Python and key libraries: Hugging Face Transformers, LangChain, OpenAI API, PyTorch/TensorFlow.

• Experience with tracing/debugging LLM outputs using tools like LangSmith, W&B Traces, or custom logs.

• Experience with model distillation, quantization, or fine-tuning on domain-specific tasks.

• Strong communication skills with the ability to translate complex model behavior into actionable insights.

Preferred:

• Prior experience building or maintaining RAG pipelines (e.g., vector databases, retrieval logic, hybrid search).

• Knowledge of sports analytics or passion for football.

• Familiarity with prompt evaluation benchmarks (e.g., TruthfulQA, MMLU, ARC) or custom evaluation harnesses.

• Experience with safety/guardrail frameworks (e.g., OpenAI Moderation, Guardrails.ai, Rebuff).

Company:

PFF is transforming the sports performance analytics experience end-to-end, building products that help NFL & NCAA and soccer front offices, coaching staffs, media partners and digital subscribers evaluate player performance at an unmatched level. Founded in 2015, the company is headquartered in Cincinnati, Ohio, USA, with a team of 201-500 employees. The company is currently Growth Stage.