NEPSE Market Prediction SystemNEPSE Market Prediction System

Overview

A solo-built, end-to-end machine-learning system for the Nepal Stock Exchange (NEPSE). It collects daily and intraday market data from several public sources, builds a clean 30-year dataset, trains a walk-forward XGBoost model, scores every listed stock after each session, and serves the results through a FastAPI backend and a React dashboard.

What sets it apart is an unusually strict focus on measuring honestly whether the model actually works. Since 21 September 2026 it has been running a pre-registered, 120-session live forward test.

Role: solo, covering design, data engineering, ML, backend, frontend, DevOps, and research. Timeline: April 2026 to present, 227+ commits.

Problem

NEPSE is a small frontier market (~370 listed symbols) that is hard to model well:

• There is no official free historical API. Data is scattered across several websites, some protected by browser-only, WASM-generated auth tokens.
• Many stocks trade thinly, prices hit ±10% daily circuit limits, and unadjusted bonus and right shares look like crashes in raw prices.
• Trading costs are high (~0.94% round trip plus 7.5% capital-gains tax, T+2 settlement), and retail short selling isn't allowed.

The goal was a reliable daily pipeline, and a rigorous answer to whether any predictive signal survives these conditions.

Architecture

Data flows sources → ingestion → canonical Parquet → features → XGBoost → scoring → FastAPI → React dashboard, as traced by the live diagram above. A calendar-aware day runner, integrity checks, and notifications wrap the whole pipeline.

• Contract first: one canonical schema; raw source column names never leave the ingestion layer
• Feature registry: no feature exists unless it is registered first, with enforced name prefixes
• Late fusion: price, macro, and news-sentiment features are built separately, joined on (ts, symbol), and fed to one tabular model. No deep learning by design; the cheapest correct model wins
• Near-zero cost: runs on a MacBook M2 with free data sources only, packaged as Docker Compose services (API, web, pipeline, local-LLM agent) sharing one 2.8 GB image

The day runner automates everything: it waits for the 15:10 NPT close, runs the pipeline, retries if a source hasn't published yet, checks integrity, and shuts down what it started.

Data engineering

• Dataset: 511,274 daily rows across 369 symbols and 5,191 sessions (1995 → 2026), plus ~7.4M floorsheet trade rows and live intraday snapshots
• Session-exact labels: targets use the next real trading session from a calendar that knows holidays, unannounced closures, and NEPSE's April 2026 move from a Sun–Thu to a Mon–Fri week
• Corporate-action masking: flags moves that violate circuit limits and removed 728 fake "crash" labels caused by bonus and right issues
• Destructive-write guards: writers union old and new data and fail closed if a stored session would vanish. A replay guard caught a source republishing yesterday's prices as today's
• Idempotent forecast log: each forecast is written before the next session opens, keyed on (ts, symbol, model_version), first write wins
• Reproducible export: a versioned dataset package with a dataset card and SHA-256 checksums

Model

• XGBoost direction classifier on next-session return: 38 features, depth-2 trees, 150 estimators
• Walk-forward validation with rolling 1,000-session train and 250-session test windows, 5 folds, and a 5-session embargo against leakage
• An XGBoost ranker runs as a shadow challenger, logged daily and never shown as advice
• Governance: artifact hash manifest, promotion gate against saved backups, and retraining blocked during the forward test

Results: the honest version

Early dashboards showed "~60% accuracy, Sharpe 1.4". A series of self-run audits showed those numbers were gross of costs, measured against the wrong baseline, and partly inflated by a date leak. The system was rebuilt to measure correctly, and the findings were kept.

• Ranking signal is real: cross-sectional rank IC of 0.093 (positive on 77% of sessions, t = 23.4) across 1,250 out-of-fold sessions and 275k predictions. The ranker challenger reaches 0.118–0.147
• Bullish calls beat the baseline: +11.8 percentage points of precision over "always up"
• But it isn't tradable: a long-only top-5 strategy with realistic fills, net of costs and tax, returns −1.04% per hold
• Best risk overlay: net Sharpe 0.18 vs buy-and-hold 0.19, with max drawdown cut from −43% to −29%

The conclusion: the model reliably orders stocks, but after NEPSE's costs and without short selling it does not produce a profitable daily long strategy. That conclusion is published inside the product itself, on its integrity page and in a trust line on every forecast.

Honesty is enforced through a pre-registered forward test with pass/fail rules written before going live, tracking of 566 research variants to discount selection bias, correct baselines (random, always-up, always-down, buy-and-hold), and a live accuracy audit that caught and fixed a leak in how forecasts were timestamped.

Engineering challenges solved

• Data behind WASM auth: rebuilt the floorsheet scraper to drive the official site headlessly and intercept its own JSON (~155 s for the full market), with a circuit breaker
• Silent data loss: two writers overwrote each other's sessions; replaced them with a union-merge that treats a shrinking session set as an error
• Resource leak in production: failed headless-browser calls leaked 145 Chromium processes (~7 GB) until the API stopped responding; fixed with guaranteed page cleanup
• Container footprint: consolidated four images into one shared image (4.5 GB → 2.8 GB), freeing ~43 GB
• Security hardening: removed secrets baked into images, mounted them read-only, and bound ports to localhost

The product

A dark, beginner-friendly React dashboard covering market overview, stock deep-dives, confirmed signals, a paper-trading sandbox with NPR 100,000 of virtual cash, a predicted-vs-actual browser, alerts, integrity checks, daily-run history, and reports from a local-LLM agent (Llama 3 8B via Ollama, no paid APIs).

Roadmap

• Finish the 120-session forward test, and promote the ranker only if its pre-registered checks pass
• Move all scraping into an isolated collector service, then onto free cloud hosting
• Publish the cleaned dataset on Hugging Face or Kaggle

Note

The repository is private, so this case study describes the system rather than linking to code. It is a research and practice project, and its outputs are not financial advice.