Using Machine Learning to Optimize Portfolio Weights
by Dr. Simiao Zhou, PhD, CFA, Director of Research, Wealthscope
Asset allocation is the process of determining how much money you put into each asset in your portfolio. To build a diversified portfolio, investors are generally advised to select assets with exposures to different geographical regions or global sectors. Once the opportunity set is identified, you have to decide how much to allocate to each asset, or the portfolio weights. A classic approach is to find an optimal asset mix by trading off the means and variances of the asset returns, based on the seminal work of Harry Markowitz. However, the portfolio weights derived from Markowitz’s mean-variance optimization are highly sensitive to small deviations in expected returns and are often concentrated in a subset of assets, resulting in unstable and under-diversified portfolios.
Hierarchical Risk Parity (HRP) is a recently developed algorithm that addresses these drawbacks and improves upon the original concept of risk parity, where each asset contributes a similar amount of risk to the overall portfolio. Since asset returns are difficult to estimate with any accuracy, the HRP algorithm utilizes only the variance/covariance matrix to find the portfolio weights, in contrast to the Markowitz model. It also goes beyond traditional risk parity by applying clustering techniques in machine learning. Clustering takes into consideration similarity among assets, so that similar assets are grouped together when receiving portfolio weights. The implementation of the HRP algorithm is summarized in the appendix.
To see how the HRP algorithm addresses the weaknesses of the mean-variance optimizer, consider an asset allocation problem where you intend to use all of the following eight ETFs:
The list of ETFs covers a wide range of assets across different classes, regions, and market capitalization. Using monthly returns from April 2007 to June 2022, we can find the minimum-variance portfolio using mean-variance optimization, with the long-only (non-negativity) constraint. This portfolio has the following weights:
Although the annualized standard deviation of this portfolio, at 5.66%, is the lowest for the sample period in question, the portfolio is concentrated in only 3 assets. In contrast, the HRP algorithm produces a portfolio with the following portfolio weights:
The assets in the table are presented in the same order as they appear from left to right in the cluster tree produced by the hierarchical clustering of the correlation matrix. During the process, the clustering analysis correctly classified the three U.S. equity ETFs in the same cluster as they are most similar to one another. It also classified the two fixed income ETFs in one cluster, and then grouped all the ETFs into one big cluster in the end. The resultant weights from an inverse variance allocation in the algorithm are less concentrated than the minimum-variance portfolio in Table 1, rendering better diversification. The annualized standard deviation of this portfolio for the same sample period is 6.86%. Although the Markowitz minimum-variance portfolio has a lower in-sample risk, its concentrated holdings are susceptible to idiosyncratic risk and are thus likely to yield a higher out-of-sample variance.
One outcome of the HRP algorithm - as is the case for traditional risk parity strategies - is that lower risk assets such as fixed income will receive more weight than you may want in order to balance the risk from each asset in the portfolio. For example, in the HRP portfolio above, the weights of the two fixed income ETFs add up to 80.4%. Professional money managers like hedge funds would use leverage (borrowed money) to increase the total portfolio risk to an acceptable level in return for a higher average return, while maintaining the benefits of risk diversification. However, managing a leveraged portfolio is not practical for most investors.
If your risk tolerance calls for a higher allocation to equity than in the HRP portfolio, we suggest a modification to the HRP procedure where the weights are determined separately for the equity and the fixed income portions of the portfolio. In Table 3, the HRP algorithm was carried out in two stages to generate a traditional 60/40 portfolio:
This portfolio has an annualized standard deviation of 11.5%, which is higher than the unconstrained HRP portfolio but more suitable to the investor’s risk profile - at 40% fixed income instead of 80%. Additionally, it uses all eight ETFs as intended, compared to the Markowitz model.
In conclusion, the HRP algorithm addresses some of the weaknesses of the classic mean-variance optimization and generates a diversified, low-risk portfolio suitable for investors to build their core portfolios. The HRP procedure can also be used to accommodate a predetermined allocation to fixed income by allocating portfolio weights in each broad asset class individually.
Appendix
To implement the HRP algorithm, there are three basic steps:
First, it converts the correlation matrix of the asset returns to a “distance” metric and creates hierarchically-related asset clusters based on the distances between each asset pair. The clustering analysis begins with each asset being considered as a separate group. The closest two groups are consecutively combined until all assets belong to the same group, resulting in a hierarchy of clusters.
Second, a top-down traversal of the hierarchy sorts the assets by their similarities. The rows and columns of the covariance matrix are then rearranged in the sorted order of the assets so that similar assets are grouped together and dissimilar assets are placed apart. As a result, larger covariances lie along the diagonal and smaller ones are placed around the diagonal.
Finally, the sorted assets are continuously bisected to allocate portfolio weights until all subsets contain only one asset. In each subset from a bisection, asset weights are allocated in proportion to the inverse of their variances, and the weights are next used to obtain the aggregated variance of the subset. The subset then receives a weighting factor proportional to its inverse aggregated variance. The inverse-variance allocation is optimal for a diagonal covariance matrix, yielding the minimum-variance portfolio. As the HRP algorithm reorganizes the covariance matrix into a quasi-diagonal form, the resulting asset allocation is close to the minimum-variance portfolio. By construction, the HRP portfolio has a higher in-sample risk than the minimum-variance portfolio; however, it is shown to produce a lower risk out of sample in simulation studies (See De Prado, 2016).
Main Reference
De Prado, M. L. (2016). Building diversified portfolios that outperform out of sample. The Journal of Portfolio Management, 42(4), 59-69.