Trust me, I am a Robo-advisor

Abstract

This paper offers cross-sectional and data-intensive insights into Robo-advisory portfolio structures. For this purpose, we scrape portfolio recommendations for 16 German Robo-advisors. Our sample accounts for about 78% of assets in the German Robo-advisory market. We analyze about 243.000 pairs of recommended portfolios and their corresponding client characteristics. Our results show that current Robo-advice offers limited individualization. Variables that matter in modern portfolio choice like the amount and nature (beta) of human capital or shadow assets are largely ignored. Instead, portfolio recommendations are designed to meet investor preconceptions or the regulator’s understanding of portfolio choice. While ensuring consumer trust and regulatory approval makes business sense, it also limits the economic benefits of Robo-advisors.$^1$

Appendix: Statistical model

This Appendix illustrates our statistical model for a specific Robo-advisor (Scalable) with 7290 input permutations. We run an OLS regression with (ordered, if applicable) factors as independent variables. The dependent variable is the recommended equity allocation. Our results are shown in table 6. The intercept of 53.483% represents the base case allocation to equities. All other regression coefficients describe the marginal effects of answering questions on the robo-advisor homepage across all 7290 choice sets. As we deal with mostly ordered factors, we can not use one hot encoding or Helmert contrasts but rather use orthogonal polynomial contrasts.¹² The extensions .L, .Q, .C denote coefficients from linear, quadratic and cubic regression terms.

For example, the contrast coding for experience is as ordered factor with three levels (0,1,2) where we add the number of “yes” answers with respect to product and financial service experience. Table 5 shows the corresponding contrast. A regression coefficient of 1.699 for the linear contrast on knowledge means that an investor twice ticking the box “none” receives a $-0.7\cdot 1.699\%=-1.18\%$ (percentage points) lower equity recommendation than the base line allocation, while an investor with extensive knowledge will receive a recommendation to add $1.18\%$ to the baseline allocation. For the full effect, we need to add the quadratic contrast or alternatively look at partial dependence plots. Responses to investment goals are easier to interpret as they are modeled as unordered factors using one-hot dummy encoding. A value of -12.349 for wealth preservation means a (ceteris paribus) decrease of 12.349% in the recommenced equity allocation for all investors ticking this box.

Table 5 Contrast matrix. We show the orthogonal polynomial contrast matrix for three levels (0,1,2)

Our regression model explains 97% of the variance of equity allocations. The standard error of the regression (standard deviation of fitted versus actual portfolio recommendations) is 4.96, i.e 2/3 of all predictions are within +/− 4.96% difference to the true value, even though our model did not use any interaction term. However, almost the same performance can be achieved by only including risk aversion and investment goals. The explanatory power fall slightly to 95%. All other variables only account for an additional 2% in explanatory power.

Next, we want to more formally interrogate our regression model to find the most influential variable(s). For this purpose we borrow from the literature on interpretable machine learning and employ the following model agnostic algorithm suggested by Fisher et al. (2018). For each variable we randomly permute the values of that particular feature and recompute the chosen performance metric, in our case $R_{\rm perm}^{2}$. We then record the difference between the baseline metric and the permutated metric $R_{\rm base}^{2}-R_{\rm perm}^{2}$ as our importance score. In order to understand the added value of a given variable we look at model results when the observations for the variable under investigation are reshuffled. Shuffling an important variable will lead to a much larger drop in explanatory power than shuffling an unimportant variable. We repeat this procedure 100 times and estimate the average importance score. The results are shown in figure 1. This confirms our earlier results. Risk aversion and investment goals are the most important variables. Creating noise in these variable leads to the most severe reduction in explanatory power across all variables.

Table 6 Drivers of Robo-advice. OLS regression results of 7290 equity allocation recommendations against input choices with respect to all variable in the Robo-advisor’s questionnaire. The adjusted $R^{2}$ of the regression is $95.62\%$. The standard error of the regression (standard deviation of fitted versus actual portfolio recommendations) is 4.962, i.e 2/3 of all weight predictions are within ± 4.962 difference to the true value

Fig. 1

Variable importance plot. Importance plot of each decision variable in our OLS regression with ordered factors (given in Table 6) defined as change in $R^{2}$ after perturbation. We re-estimate the model several times (as many times as we have explanatory variables), each time with one variable randomized. For each regression we calculate the difference between the $R^{2}$ of the original data and the pertubated data. The larger the difference, the more important the variable. This yields an importance score for each variable. Repeating this exercise 100 times results in the plot below for the five variables with the highest importance score

Finally, we check the direction of influence for each question in the Robo-advisor’s questionnaire. In order to account for nonlinear contrasts we need to compute the cumulative effect of all polynomial terms. For this purpose, we use partial dependence plot as shown in Fig. 2. The idea of partial dependence plots is to estimate a statistical model using the original data and then use this fitted model to make predictions from a modified data set. The modified data set is a complete copy of the original data set, except for the variable of interest where all realizations are replaced by a particular value. The average across all predictions is then used as best estimate for the partial variation of interest. After repeating this process for all level of the variable of interest we can plot this variable against the average responses in a scatterplot. This plot is called partial dependence plot and shows both direction and magnitude of influence.

Fig. 2

Partial dependence plot for volatility For a given answer to the volatility question (e.g. 5%), we copy the complete data set of 7290 observations and replace the oiginal volatility data with that particular value. Next we calculate the predicted values (using the original model estimated from the unmodified data set) and calculate the average predicted value (across all 7290 predictions). This process is repeated for each answer to the volatility question and displayed as scatterplot

We can confirm our results by employing a regression tree in Fig. 3. . Regression trees use explanatory variables to consecutively split the data (using only one variable at each node) into pure clusters with as little intra-cluster variation as possible. Clusters do not need to have the same size (do not need to contain the same number of variations). Instead of making continuous predictions all combinations of explanatory variables that lead into a given terminal node carry the same prediction. In our context (recommended equity allocations from questionnaire inputs, regression trees offer some advantages over linear regressions. The first split selects the most important variable, while the sequence of splits is able to model non-linearities. This allows us to find otherwise hidden nonlinear interactions. While linear regressions can also uncover nonlinear interactions by including all possible cross terms, this requires as many right-hand-side variables as data points and thus results in a loss of all degrees of freedom. Our fitted regression tree identifies the same set of variables as most important in explaining the cross section of equity recommendations. Interestingly its standard error of 1.98 is less than half of a linear regression model, which we take as evidence for nonlinear interactions not covered by a linear regression. High equity recommendations are reserved for investors with low risk aversion, agrressive goals, large levels of wealth and sufficient experience.

Fig. 3

Regression tree The dependent variable is equity allocation in %. All input parameters are used as explanatory variables (features). Each node contains the variable used for a particular data split. Terminal nodes contain box/whisker plots for the realization of the dependent variable in this node as well as the number of occurrences. The prediction on each terminal node is the same for all input combinations that lead to this node. The standard deviation of differences between fitted and actually recommended weights is 1.98

Our fitted regression tree regards time horizon as the first variable to split the data on. Suppose we would be only allowed to split the data once into clusters with as little internal dispersion of equity weights as possible. Our regression tree would then recommend to use the variable time horizon. In this sense time horizon is the most important variable. Short and medium term horizon investors receive recommended allocations between 21.14% (node 4) and 49.73% (node 8) equities while long horizon investors obtain portfolios between 53.6% (node 10) and 73.95% (node 13)%. All percentages are predictions from the regression. tree. The standard error of the regression tree (standard deviation of fitted versus actual portfolio recommendations) is 5.51. Five from six nodes use the variables investment goal and time horizon. This again confirms our previous analysis. Predicted equity recommendations as a function of questionnaire replies rise from left to right. The most aggressive allocations are reserved to long term investors with retirement objectives that react to losses by increasing their equity allocations. Investors with short time horizon looking to fund an upcoming expense receive small equity allocations.¹³