Using Machine Learning to Optimize Portfolio Weights
by Dr. Simiao Zhou, PhD, CFA, Director of Research
Asset allocation is the process of determining how much money you put into each asset in your portfolio. To build a diversified portfolio, investors are often advised to select assets with exposures to different geographical regions or global sectors. Once the opportunity set is identified, you have to decide how much to allocate to each asset, i.e., the portfolio weights. A classic approach is to find an optimal asset mix by trading off the means and variances of the asset returns, based on the seminal work of Harry Markowitz in 1952. However, the portfolio weights derived from Markowitz’s mean-variance optimization are highly sensitive to small deviations in mean returns and are often concentrated in a subset of assets, resulting in an unstable and under-diversified portfolio.
Hierarchical Risk Parity (HRP) is a recently developed algorithm that addresses these drawbacks and improves upon the original concept of risk parity, where each asset contributes a similar amount of risk to the overall portfolio. Since asset returns are difficult to estimate with any accuracy, the HRP algorithm utilizes only the variance/covariance matrix to find the portfolio weights, in contrast to the Markowitz model. It also goes beyond traditional risk parity by applying clustering techniques in machine learning. Clustering takes into consideration similarity among assets, so that highly correlated assets are grouped together when receiving portfolio weights. This avoids concentration in assets that are in fact similar. The Wealthscope tool further adjusts the process by allowing users to specify a desired allocation to fixed income (if they input a fixed income fund(s)) to avoid over-allocation to this asset class, a common complaint about the risk-parity procedure. The implementation of the HRP algorithm is summarized in the appendix.
To illustrate how the HRP algorithm addresses the weaknesses of Markowitz's mean-variance optimizer, consider an asset allocation problem using the following 9 ETFs:
IVV iShares Core S&P 500 ETF
VO Vanguard Mid-Cap ETF
VB Vanguard Small-Cap ETF
EFA iShares MSCI EAFE ETF
EEM iShares MSCI Emerging Markets ETF
XIC iShares Core S&P/TSX Capped Composite Index ETF
XRE iShares S&P/TSX Capped REIT Index ETF
XCB iShares Core Canadian Corporate Bond Index ETF
XGB iShares Core Canadian Government Bond Index ETF
The list of ETFs covers a wide range of assets across different classes, regions, and market capitalization. Using monthly returns from April 2007 to February 2022, we can find the minimum-variance portfolio using mean-variance optimization, with the long-only (non-negativity) constraint. This portfolio has the following weights:
Although the annualized standard deviation of this portfolio, at 4.05%, is the lowest for the sample period in question, the portfolio is concentrated in only 3 assets. In contrast, the HRP algorithm produces a portfolio with the following portfolio weights:
The assets in the table are presented in the same order as they appear from left to right in the cluster tree produced by the hierarchical clustering of the correlation matrix. During the process, the clustering analysis correctly classified the three U.S. equity ETFs in the same cluster as they are most similar to one another. It also classified the two fixed income ETFs in one cluster, and then grouped all the ETFs into one big cluster in the end. The resultant weights from an inverse variance allocation in the algorithm are less concentrated than the minimum-variance portfolio in Table 1, rendering better diversification. The annualized standard deviation of this portfolio for the same sample period is 4.98%. Although the Markowitz minimum-variance portfolio has a lower in-sample risk, its concentrated holdings are susceptible to idiosyncratic risk and are thus likely to yield a higher out-of-sample variance.
One outcome of the HRP algorithm - as is the case for traditional risk parity strategies - is that lower risk assets such as fixed income will receive more weight than you may want in order to balance the risk from each asset in the portfolio. For example, in the HRP portfolio above, the weights of the two fixed income ETFs add up to 74.66%. Professional money managers like hedge funds would use leverage (borrowed money) to increase the total portfolio risk to an acceptable level in return for a higher average return, while maintaining the benefits of risk diversification. However, managing a leveraged portfolio is not practical for most investors.
If your risk tolerance calls for a higher allocation to equity than in the HRP portfolio, we modify the HRP procedure and determine the weights separately for the equity and the fixed income portions of the portfolio. In Table 3, the HRP algorithm was carried out in two stages to generate a traditional 60/40 portfolio.
This portfolio has an annualized standard deviation of 7.80%, which is higher than the unconstrained HRP portfolio but more suitable to the investor’s risk profile.
In conclusion, the HRP algorithm addresses some of the weaknesses of the classic mean-variance optimization and generates a diversified, low-risk portfolio suitable for investors to build their core portfolios. The HRP procedure can also be used to accommodate a predetermined allocation to fixed income by allocating portfolio weights in each broad asset class individually.
To implement the HRP algorithm, there are three basic steps:
First, it converts the correlation matrix of the asset returns to a “distance” metric and creates hierarchically-related asset clusters based on the distances between each asset pair. The clustering analysis begins with each asset being considered as a separate group. The closest two groups are consecutively combined until all assets belong to the same group, resulting in a hierarchy of clusters.
Second, a top-down traversal of the hierarchy sorts the assets by their similarities. The rows and columns of the covariance matrix are then rearranged in the sorted order of the assets so that similar assets are grouped together and dissimilar assets are placed apart. As a result, larger covariances lie along the diagonal and smaller ones are placed around the diagonal.
Finally, the sorted assets are continuously bisected to allocate portfolio weights until all subsets contain only one asset. In each subset from a bisection, asset weights are allocated in proportion to the inverse of their variances, and the weights are next used to obtain the aggregated variance of the subset. The subset then receives a weighting factor proportional to its inverse aggregated variance. The inverse-variance allocation is optimal for a diagonal covariance matrix, yielding the minimum-variance portfolio. As the HRP algorithm reorganizes the covariance matrix into a quasi-diagonal form, the resulting asset allocation is close to the minimum-variance portfolio. By construction, the HRP portfolio has a higher in-sample risk than the minimum-variance portfolio; however, it is shown to produce a lower risk out of sample in simulation studies (See De Prado, 2016).
De Prado, M. L. (2016). Building diversified portfolios that outperform out of sample. The Journal of Portfolio Management, 42(4), 59-69.