An autonomous, self-healing MLOps pipeline and regression service that continuously models foreign exchange (FX) risk. The system forecasts expected continuous volatility using an LSTM neural network, actively monitors inference health via Prometheus and Grafana, and triggers automated retraining loops upon statistical data drift or error degradation.
Note: This is a continuous regression model, not a classification system. It outputs a normalized expected volatility score ([0, 1]), which is then mapped to operational risk categories for alerting.
+-----------------------------------------------------------------------------+
| INCOMING FX MARKET DATA |
+-----------------------------------------------------------------------------+
|
v
+-----------------------------------+
| FastAPI High-Throughput Engine |
+-----------------------------------+
|
+----------------+----------------+
| |
v v
+--------------------------+ +--------------------------+
| LSTM Model Engine | | Observability Exporter |
| (Risk Score Inference) | | (Prometheus /metrics) |
+--------------------------+ +--------------------------+
| |
v v
+--------------------------+ +--------------------------+
| Drift & Error Monitor | | Grafana Dashboards |
| - KS-Test (alpha=0.05) | | - Latency & Status |
| - RMSE Drift (>20%) | | - Real-time RMSE & MAPE |
+--------------------------+ +--------------------------+
|
[Degradation Detected]
|
v
+--------------------------+
| Self-Healing Pipeline |
| - Auto-retrain Loop |
| - Model Artifact Bump |
| - Hot-swap Deployment |
+--------------------------+
1. Continuous Risk Scoring
Target Output: Predicts normalized volatility scores from 0.0 (Minimal Risk) to 1.0 (Critical Volatility).
Operational Thresholds: Maps continuous float values into operational tiers: LOW, MEDIUM, HIGH, and CRITICAL.
Currency Support: Production pipelines for EUR/USD and INR/USD.
2. Autonomic Self-Healing (MLOps)
Real-Time Statistical Drift Detection: Performs automated two-sample Kolmogorov-Smirnov (KS) tests (alpha = 0.05) across input feature windows.
Automated Threshold Triggers: Detects performance degradation if live prediction error (RMSE) expands >20% against baseline.
Closed-Loop Retraining: Automatically executes downstream data ingestion, model retraining, validation, and zero-downtime weight updates.
3. Production Monitoring & Telemetry
Prometheus Instrumentation: Exposes continuous inference latency, HTTP request rates, real-time RMSE, MAE, R², and drift status at /metrics.
Pre-configured Grafana Dashboards: Ready-to-run dashboard tracking pipeline health, data distribution shifts, and active model versions.
Tech Stack
Layer Technology Purpose
Language Python 3.12 Core runtime environment
Deep Learning PyTorch 2.x 2-layer stacked LSTM regression network
API Serving FastAPI + Uvicorn Asynchronous prediction microservice
Data Sourcing Yahoo Finance (yfinance) Automated market sequence ingestion
Drift Detection SciPy (KS-Test) Non-parametric statistical distribution validation
Telemetry Prometheus Metric aggregation and time-series monitoring
Observability Grafana System health and prediction visualizer
Containers Docker & Docker Compose Containerized observability infrastructure
Repository Structure
Plaintext
.
├── main.py # Unified CLI pipeline orchestrator
├── requirements.txt # Python dependencies
├── docker-compose.yml # Multi-container telemetry stack
├── api/
│ └── app.py # FastAPI server and Prometheus exporter
├── src/
│ ├── data_loader.py # Automated FX data extraction
│ ├── feature_engineering.py# Technical indicators (RSI, Log-returns, Volatility)
│ ├── model.py # PyTorch LSTM network definition
│ ├── trainer.py # Early-stopping training loop
│ ├── predictor.py # Production batch/single-record inference
│ ├── monitor.py # Real-time RMSE tracking & drift detection
│ ├── decision_engine.py # Retrain heuristics & threshold verification
│ └── retrain_pipeline.py # Closed-loop self-healing pipeline
├── config/
│ └── settings.py # Global paths, hyperparameters, and thresholds
├── monitoring/
│ ├── prometheus/ # Prometheus scraper configs
│ └── grafana/ # Pre-provisioned dashboards and datasources
├── models/ # Serialized PyTorch model artifacts (.pt)
├── data/ # Raw and engineered time-series caches
└── tests/ # Unit and integration test suites
Quickstart Guide
1. Environment Setup
Clone the repository and install required dependencies:
Bash
git clone [https://github.com/sun-9545sunoj/self_heal_forex_prediction_model.git](https://github.com/sun-9545sunoj/self_heal_forex_prediction_model.git)
cd self_heal_forex_prediction_model
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
pip install -r requirements.txt
2. Ingest Data & Train Baseline Model
Download historical currency sequences and train the initial LSTM:
Bash
# Ingest historical forex records
python main.py download
# Train initial model weights
python main.py train
3. Launch Observability Stack
Start Prometheus and Grafana using Docker Compose:
Bash
docker-compose up -d
Grafana: http://localhost:3000 (admin / admin)
Prometheus: http://localhost:9090
4. Serve the API
Run the FastAPI inference service:
Bash
python main.py serve
The API will be available at http://localhost:8000 (Interactive Swagger Docs: /docs).
API Reference
Method Endpoint Description
GET /health Service status, current model version, and uptime
POST /predict?pair=EURUSD Executes inference; returns continuous score & category
GET /monitor Latest RMSE, MAE, and KS-drift flags
GET /history Historical performance log across inference batches
GET /pairs List of supported currency pairs
GET /metrics Prometheus-formatted telemetry scrape target
Sample Prediction Response (POST /predict?pair=EURUSD)
JSON
{
"risk_score": 0.78,
"risk_level": "HIGH",
"model_version": 77,
"model_status": "STABLE",
"drift_detected": false,
"rmse": 0.0023,
"mae": 0.0018,
"r2_score": 0.94,
"mape": 12.5,
"within_threshold_pct": 85.0,
"timestamp": "2026-01-28T12:00:00Z"
}
Model & Training Specification
Architecture: 2-layer Stacked LSTM (64 hidden units, dropout = 0.2).
Input Representation: 30-day temporal sliding window incorporating:
Simple returns & Logarithmic returns
Rolling annualized volatility
Relative Strength Index (RSI, 14-period window)
Output: Continuous bounded scalar via Sigmoid layer representing expected volatility.
Optimization: Mean Squared Error (MSE) loss, Adam optimizer (lr = 1e-3), gradient clipping, and patience-based early stopping.
Regression Evaluation Metrics
The system monitors continuous accuracy via statistical and regression error metrics:
Metric Full Form Interpretation
RMSE Root Mean Square Error Penalizes large outliers; primary self-healing trigger.
MAE Mean Absolute Error Median scale of typical absolute deviation.
R² Score Coefficient of Determination Proportion of volatility variance explained by the model.
MAPE Mean Absolute Percentage Error Scale-independent percentage tracking error.
Within Threshold Accuracy within Tolerance Percentage of predictions falling within ±15% of actuals.
CLI Command Reference
The main.py orchestrator provides an interface for routine MLOps workflows:
Bash
python main.py download # Pull latest FX market series
python main.py train # Trigger training on local data cache
python main.py predict # Run batch inference run
python main.py monitor # Output terminal summary of drift & error
python main.py retrain # Force an immediate manual retraining cycle
python main.py serve # Run Uvicorn-hosted API server
python main.py plots # Output static diagnostic plots to disk
python main.py dashboard # Launch monitoring services
Testing
Execute unit, integration, and drift validation tests:
Bash
pytest tests/ -v
License
This project is licensed under the MIT License — see the LICENSE file for details.