Role Overview As an AI Engineer - RAG Platform Optimization within banks central AI capability, you take ownership of continuously improving the accuracy, retrieval quality, and response effectiveness of the bank's enterprise Retrieval-Augmented Generation (RAG) platform. Operating at the convergence of Information Retrieval, Data Science, and Software Engineering, your core mission is failure analysis, hypothesis-driven experimentation, and evaluation framework optimization. Rather than managing people or building standard web UI applications, you focus strictly on measurable model quality-analyzing production telemetry in Databricks, running chunking/embedding experiments, building continuous feedback loops, and ensuring every release raises the performance benchmark for thousands of internal users.
Key Responsibilities:
Continuous RAG Model Optimization:
Perform failure analysis on low-quality responses, identify underlying retrieval or grounding flaws, and run large-scale experiments on chunking, embedding models, prompt optimization, and reranking strategies.
GenAI Evaluation & Measurement Frameworks:
Build and enhance automated evaluation pipelines to measure retrieval precision/recall, response correctness, hallucination rates, and user satisfaction metrics.
Data Analysis & Telemetry Insights:
Analyze user interaction data, production telemetry, and evaluation datasets within Databricks to detect performance regressions and uncover quality improvement opportunities.
Production Pipeline Implementation:
Translate data-driven experimentation results into scalable, maintainable Python production code and automated CI/CD data pipelines on Azure.
Cross-Functional Technical Alignment:
Partner with Software Engineers, Data Scientists, Platform Engineers, and Product Managers to embed evaluation metrics into automated release pipelines.
Your Deliverables:
Automated GenAI evaluation suites and synthetic benchmark datasets tracking retrieval and response quality.
Optimized chunking, embedding, reranking, and prompt strategies deployed directly into production pipelines.
Production telemetry dashboards in Databricks tracking model performance metrics, error categories, and user trust indicators.
CI/CD-integrated testing scripts that prevent quality regressions prior to production deployments.
Your Profile:
Education (Hard Requirement):
Master's degree (WO) or equivalent academic level in Data Science, Artificial Intelligence, Computer Science, Mathematics, Statistics, or a related quantitative field.
Experience & Track Record (Hard Requirements):
Minimum of
3+ years of professional experience
in machine learning, applied AI, or data science.
Demonstrable track record of improving
production-grade RAG systems
or advanced retrieval-based AI solutions.
Strong hands‑on proficiency in
Python, Databricks, LLM architecture, and Azure cloud environments .
Proven experience designing evaluation methodologies, quality metrics, and continuous feedback loops for GenAI systems.
Proficiency in written and spoken
English
(corporate business language).
Nice‑to‑Have Technical Experience:
Experience with
fine‑tuning language models
and reranking algorithms.
Hands‑on experience with
AI observability tools
(e.g., MLflow, TruLens, Ragas, LangSmith).
Knowledge of synthetic evaluation dataset creation and benchmark curation.
#J-18808-Ljbffr
AI Engineer in Utrecht werkgever: Fitura
Als Senior CX & Change Communications Specialist bij ons toonaangevende bedrijf in Eindhoven, krijg je de kans om een significante impact te maken op klantgerichte transformaties. We bieden een stimulerende werkomgeving waar samenwerking en innovatie centraal staan, met volop mogelijkheden voor persoonlijke en professionele groei. Geniet van een competitief salaris, flexibele werkuren en een cultuur die diversiteit en inclusie waardeert, waardoor je je volledig kunt ontplooien in je rol.
