Weekly wholesale maize price prediction for 5 Kenya counties, 2 weeks ahead. The pipeline combines two data sources (agriBORA weekly prices and KAMIS daily market prices), rich temporal feature engineering, and an ensemble of ML and classical time series models — with walk-forward cross-validation for honest evaluation.
Predict the weekly average wholesale maize price (in KES) for 5 Kenyan counties: Kiambu, Kirinyaga, Mombasa, Nairobi, and Uasin-Gishu. The forecast horizon is 2 weeks ahead, submitted as a rolling prediction. The scoring metric is the average of MAE and RMSE, penalizing both systematic bias and large errors.
Two data sources were available: agriBORA weekly transaction prices (the primary target series) and KAMIS daily market prices across multiple markets. The challenge was that the two sources used different temporal granularities, different market references, and partially overlapping time windows — requiring careful calibration before any modeling.
Libraries, paths, constants, helper functions. Seed set to 1618.
Load KAMIS daily prices and agriBORA weekly transaction prices. Map to ISO week calendar.
Snap KAMIS dates to nearest Monday. Impute wholesale NAs via regression on market and retail price. Volume-weight aggregate daily KAMIS to weekly. Calibrate KAMIS to agriBORA scale via linear regression on overlapping weeks. Fill agriBORA gaps with LOCF.
Rich temporal feature set: lags, rolling statistics, momentum signals, harvest season flags, and cyclical encodings (see below).
Multicollinearity removal via VIF. Chronological train/test split. Walk-forward CV with 5 windows, minimum 52 weeks training data.
LightGBM, XGBoost, Auto-ARIMA, ETS (damped exponential), Prophet (multiplicative, yearly seasonality). Final predictions: equal-weight ensemble of all available forecasts.
Walk-forward CV holdout evaluation + holdout on last 8 weeks for final model selection. Rolling 2-week forecast generation to submission CSV.
KAMIS daily prices come from multiple markets. Rather than simple averaging, the weekly price is computed as sum(Wholesale × SupplyVolume) / sum(SupplyVolume). Markets with higher supply volume have proportionally more influence on the weekly price — which mirrors how market-level prices actually aggregate to county-level prices in practice.
The two data sources measure different things in different ways. A direct merge would introduce a systematic level shift. Instead, a linear regression was fit on the overlapping weeks to calibrate KAMIS prices to the agriBORA scale. After calibration, KAMIS becomes an external feature only — not a competing target.
The feature set covers multiple temporal horizons and signals: lags (1–52 weeks), rolling moving averages (2–52 weeks), rolling volatility (4–26 weeks), momentum (MA4−MA8, MA8−MA26, MA12−MA52), harvest season flags for May/June/October, lean season flags for January/February/July/August, and cyclical sin/cos encodings for month, week-of-year, and quarter.
# Feature engineering: lags lag_windows <- c(1, 2, 3, 4, 5, 6, 8, 12, 16, 26, 52) # Rolling moving averages ma_windows <- c(2, 3, 4, 6, 8, 12, 26, 52) # Rolling volatility vol_windows <- c(4, 8, 12, 26) # Momentum: fast MA minus slow MA momentum <- list(c(4, 8), c(8, 26), c(12, 52)) # Seasonal flags is_harvest_season <- month(date) %in% c(5, 6, 10) is_lean_season <- month(date) %in% c(1, 2, 7, 8) # Cyclical encoding sin_month <- sin(2 * pi * month(date) / 12) cos_month <- cos(2 * pi * month(date) / 12) sin_week <- sin(2 * pi * isoweek(date) / 52) cos_week <- cos(2 * pi * isoweek(date) / 52)
Rather than selecting the best model from walk-forward CV, the final submission uses an equal-weight mean of all available forecasts (LightGBM, XGBoost, ARIMA, ETS, Prophet). This reduces variance across the 5 county series, where different models may perform better in different counties and different weeks of the year.
LightGBM num_leaves=31, lr=0.05, 500 rounds XGBoost max_depth=6, eta=0.05, 500 rounds Auto-ARIMA seasonal=TRUE, frequency=52 ETS automatic damped exponential smoothing Prophet yearly.seasonality=TRUE, multiplicative mode Ensemble equal-weight mean of all model outputs
The highest-leverage steps were data calibration (aligning two sources with different scales before modeling) and feature engineering depth. The ML models benefit significantly from having multiple temporal scales of the same signal — short-lag recency, medium-lag trend, long-lag seasonality — rather than a small flat feature set.
Design insight: In commodity price forecasting, the ensemble of classical time series models and gradient boosting consistently outperforms either family alone. Classical models handle seasonality and trend structure explicitly. ML models capture non-linear interactions between external features. Using both together covers different sources of predictability.