AI/LLM DevOps Quick Reference Guide

A concise reference guide for implementing DevOps and CI/CD practices for AI/LLM applications.

🚀 Quick Start Checklist

Day 1: Foundation

• Set up version control (Git) for code, models, and data
• Configure basic CI pipeline (unit tests, linting)
• Containerize model serving application
• Set up model registry (MLflow or similar)
• Implement basic monitoring (logs, metrics)

Week 1: Core Infrastructure

• Deploy Kubernetes cluster or managed service
• Set up artifact storage (S3, GCS, or similar)
• Configure Prometheus + Grafana monitoring
• Implement model versioning strategy
• Create staging environment

Month 1: Production Readiness

• Implement canary deployment strategy
• Set up A/B testing framework
• Configure comprehensive observability
• Establish rollback procedures
• Implement security scanning

📊 Tool Comparison Matrix

Category	Open Source	Cloud-Managed	Best For
Experiment Tracking	MLflow, DVC	SageMaker, Vertex AI	MLflow for flexibility
Model Registry	MLflow, BentoML	SageMaker, Azure ML	MLflow for multi-cloud
Workflow Orchestration	Kubeflow, Airflow	SageMaker Pipelines	Kubeflow for K8s
Feature Store	Feast	SageMaker FS, Vertex FS	Feast for portability
Monitoring	Prometheus, Grafana	Datadog, New Relic	Prometheus for cost
LLM Observability	Langfuse, Phoenix	Datadog LLM, Arize	Arize for enterprise

🔧 Essential Commands

MLflow Commands

# Start MLflow UI
mlflow ui --host 0.0.0.0 --port 5000
 
# Register model
mlflow models register -m runs:/<run-id>/model -n <model-name>
 
# Serve model
mlflow models serve -m models:/<model-name>/<version> -p 5001
 
# Promote to production
mlflow models transition-stage -n <model-name> -v <version> -s Production

DVC Commands

# Initialize DVC
dvc init
 
# Track data file
dvc add data/training_data.csv
 
# Create pipeline
dvc run -n train -d src/train.py -d data/prepared \
        -o models/model.pkl python src/train.py
 
# Reproduce pipeline
dvc repro
 
# Push to remote storage
dvc push

Kubernetes Deployments

# Deploy model server
kubectl apply -f model-deployment.yaml
 
# Scale deployment
kubectl scale deployment model-server --replicas=10
 
# Update image (basic rolling update)
kubectl set image deployment/model-server model-server=new-image:tag
 
# Rollback deployment
kubectl rollout undo deployment/model-server
 
# Check rollout status
kubectl rollout status deployment/model-server

Docker Commands for ML

# Build with GPU support
docker build -t model-server:latest \
  --build-arg CUDA_VERSION=11.8 \
  -f Dockerfile.gpu .
 
# Run with GPU
docker run --gpus all -p 8080:8080 model-server:latest
 
# Multi-stage build for smaller images
docker build -t model-server:slim \
  --target production \
  -f Dockerfile.multistage .

📈 Key Metrics to Monitor

Model Performance

• Accuracy/F1/AUC: Track per model version
• Prediction latency: P50, P95, P99
• Throughput: Requests per second
• Data drift: Feature distribution changes
• Model drift: Prediction distribution changes

System Performance

• GPU utilization: Memory and compute
• CPU/Memory usage: Container resource consumption
• Queue depth: Pending prediction requests
• Error rates: 4xx, 5xx responses
• Cost per prediction: Cloud resource costs

LLM-Specific Metrics

• Token usage: Input/output tokens per request
• Cost per request: Based on token usage
• Hallucination rate: Custom scoring metrics
• Safety violations: Prompt injection attempts
• Response quality: User feedback scores

🚦 Deployment Strategies

1. Blue-Green Deployment

# Quick setup
active: blue (v1.0) - 100% traffic
standby: green (v2.0) - 0% traffic
rollback: instant switch to blue

2. Canary Deployment

# Progressive rollout
stage 1: 1% traffic (10 mins)
stage 2: 5% traffic (30 mins)
stage 3: 25% traffic (1 hour)
stage 4: 50% traffic (2 hours)
stage 5: 100% traffic

3. Shadow Deployment

# Risk-free testing
production: v1.0 (serves all requests)
shadow: v2.0 (processes same requests, no user impact)
comparison: automated diff analysis

🛡️ Security Best Practices

Container Security

# Use specific base image versions
FROM python:3.11-slim-bookworm
 
# Run as non-root user
RUN useradd -m -u 1000 mluser
USER mluser
 
# Copy only necessary files
COPY --chown=mluser:mluser requirements.txt .
COPY --chown=mluser:mluser src/ ./src/
 
# Scan for vulnerabilities
# trivy image model-server:latest

API Security

# Rate limiting
from flask_limiter import Limiter
limiter = Limiter(
    app,
    key_func=lambda: get_user_id(),
    default_limits=["100 per minute"]
)
 
# Input validation
from pydantic import BaseModel, validator
class PredictionRequest(BaseModel):
    input_text: str
    
    @validator('input_text')
    def validate_length(cls, v):
        if len(v) > 10000:
            raise ValueError('Input too long')
        return v

🔄 CI/CD Pipeline Templates

GitHub Actions Quick Setup

name: ML Pipeline
on: [push]
 
jobs:
  test-train-deploy:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3
    - uses: actions/setup-python@v4
    - run: pip install -r requirements.txt
    - run: pytest tests/
    - run: python train.py
    - run: python evaluate.py
    - run: |
        if [ "${{ github.ref }}" == "refs/heads/main" ]; then
          python deploy.py
        fi

GitLab CI Quick Setup

stages:
  - test
  - train
  - deploy
 
test:
  stage: test
  script:
    - pip install -r requirements.txt
    - pytest tests/
 
train:
  stage: train
  script:
    - python train.py
  artifacts:
    paths:
      - models/
 
deploy:
  stage: deploy
  script:
    - python deploy.py
  only:
    - main

📱 Alerting Rules

Prometheus Alert Examples

groups:
  - name: ml_alerts
    rules:
    - alert: ModelAccuracyDegraded
      expr: model_accuracy < 0.90
      for: 5m
      annotations:
        summary: "Model accuracy below threshold"
        
    - alert: HighPredictionLatency
      expr: histogram_quantile(0.99, prediction_latency_bucket) > 0.5
      for: 5m
      annotations:
        summary: "P99 latency above 500ms"
        
    - alert: DataDriftDetected
      expr: feature_drift_score > 0.3
      for: 10m
      annotations:
        summary: "Significant data drift detected"

🌟 Best Practices Summary

Version Control

• ✅ Version models with semantic versioning
• ✅ Track data lineage with DVC or similar
• ✅ Store model metadata alongside artifacts
• ✅ Use immutable model registries

Testing

• ✅ Unit test preprocessing functions
• ✅ Integration test model endpoints
• ✅ Performance test under load
• ✅ Test rollback procedures
• ✅ Validate against holdout datasets

Monitoring

• ✅ Monitor both technical and business metrics
• ✅ Set up alerting for anomalies
• ✅ Track cost per prediction
• ✅ Implement distributed tracing
• ✅ Log all predictions for audit

Deployment

• ✅ Always use staged rollouts
• ✅ Implement automated rollback
• ✅ Test in shadow mode first
• ✅ Use feature flags for control
• ✅ Maintain deployment runbooks

🚨 Common Pitfalls to Avoid

1. Not versioning training data - Always track data changes
2. Ignoring model drift - Monitor performance continuously
3. Manual deployments - Automate everything
4. No rollback plan - Always have an escape route
5. Insufficient monitoring - You can’t fix what you can’t see
6. Ignoring costs - LLMs can be expensive at scale
7. Security as afterthought - Build security in from start

📚 Quick Links

Documentation

• Comprehensive MLOps Guide
• Implementation Patterns
• LLM Observability Guide

Tools

• MLflow
• Kubeflow
• DVC
• Prometheus
• Argo Rollouts

Cloud Platforms

• AWS SageMaker
• Google Vertex AI
• Azure Machine Learning

🎯 Next Steps

1. Start Small: Implement basic CI/CD with model versioning
2. Add Monitoring: Set up Prometheus + Grafana
3. Implement Staging: Create a staging environment
4. Add Canary Deployments: Start with 5% traffic tests
5. Scale Up: Add more sophisticated monitoring and automation

Remember: The goal is to make ML deployments as reliable and automated as traditional software deployments, while accounting for the unique challenges of ML systems.