Job Description

RESPONSIBILITIES:ML Model Deployment & Platform Management• Lead the design, implementation, and ongoing maintenance of scalable ML infrastructure on Databricks, including ML flow for experiment tracking, model registry, and model serving endpoints.• Oversee the development of the ML Ops platform and automated pipelines for deploying, monitoring, and maintaining models within production environments.• Implement robust solutions for model versioning, systematic retraining, and comprehensive artifact management using Databricks Unity Catalog for ML governance.• Design and manage Databricks Feature Store for consistent feature engineering across training and inference pipelines.Generative AI & LLM Operations• Architect and implement Retrieval-Augmented Generation (RAG) systems for document Q&A, enabling business teams to query fund documents, investor letters, and market research.• Design, deploy, and manage vector database solutions (Databricks Vector Search, Pinecone, or similar) for semantic search and retrieval across enterprise documents.• Lead LLM fine-tuning and customization initiatives, training models like Claude or open-source alternatives with CIM proprietary data while ensuring data privacy and compliance.• Develop and optimize document processing pipelines including PDF parsing, chunking strategies, and embedding generation for RAG applications.• Implement prompt engineering best practices and LLM evaluation frameworks to ensure output quality, relevance, and factual accuracy.• Build guardrails and safety measures for GenAI applications, including hallucination detection, output validation, and source attribution.Automation & CI/CD Pipelines• Design and implement extensive automation across the ML workflow, covering model training, testing, validation, and deployment using Databricks Workflows and Asset Bundles.• Set up robust CI/CD pipelines for both traditional ML models and GenAI applications, leveraging GitHub Actions, Azure DevOps, or similar tools.• Automate complex data and model workflows utilizing orchestration tools such as Airflow, Prefect, or Databricks Workflows.

Job Tags

Similar Jobs