Robust portfolio optimisation: covariance estimation, machine learning forecasting, and clustering-based allocation

Portfolio optimisation is challenging under conditions of covariance noise, unreliable return estimates, and allocation methods that do not account for the natural grouping structure of the asset universe. This thesis addresses these three sources of instability in turn. It refines the covariance matrix, forecasts expected returns through machine learning, and allocates capital through a clustering-based structure. These components are then combined into a single, robust portfolio-construction approach. First, under risk-based objectives, several covariance estimators are evaluated, including shrinkage methods, dynamic conditional correlation models, and a quantile-based extension of the covariance denoising framework proposed in this thesis. Dynamic estimators combined with nonlinear shrinkage, and the denoising extension, offer superior performance. Second, the original Nested Clustered Optimisation framework is extended by incorporating machine-learning forecasts of expected returns together with these refined covariance estimators, and tested under alternative clustering algorithms and distance metrics. The strongest portfolios combine Random Forest forecasts with hierarchical clustering, a rank-based distance metric, and dynamic covariance estimators. These portfolios achieve the highest Sharpe ratios recorded across all strategies tested and outperform the benchmark. The proposed framework delivers a higher level of diversification and lower weight concentration than classical mean–variance optimisation using the same inputs. Third, the framework is extended to hedge fund strategy indices to examine whether its benefits generalise beyond equity portfolios and remain effective across distinct market regimes. Results show that the clustering framework delivers its greatest advantage during periods of market stress, when correlations across strategies converge. The empirical results show that the proposed framework produces stronger out-of-sample performance across equities and hedge funds than classical mean–variance optimisation.