Sharpe Ratio

Mar 03, 2019

A Sharper Sharpe: Its Biased.

In a series of posts, we looked at Damien Challet's 'Sharper estimator' of the Signal-Noise Ratio, which computes the number of drawdowns in the returns series (plus some permutations thereof), then uses a spline function to infer the Signal-Noise Ratio from the drawdown statistic. The spline function is built from the results of Monte Carlo simulations.

In the last post of the series, we looked at the apparent bias of this 'drawdown estimator'. We suggested, somewhat facetiously, that one could achieve similar properties to the drawdown estimator (reduced RMSE, bias, etc.) by taking the traditional moment-based Sharpe ratio and multiplying it by 0.8.

I contacted Challet to present my concerns. I suspected that the spline function was trained with too narrow a range of population Signal-Noise ratios, which would result in this bias, and suggested he expand his simulations as a fix. I see that since that time the sharpeRratio package has gained a proper github page, and the version number was bumped to 1.2. (It has not been released to CRAN, so it is premature to call it "the" 1.2 release.)

In this post, I hope to:

  1. Demonstrate the bias of the drawdown estimator in a way that clearly illustrates why "Sharpe ratio times 0.8" (well, really 0.7) is a valid comparison.
  2. Check whether the bias has been corrected in the 1.2 development version. (Spoiler alert: no.)
  3. Provide further evidence that the issue is the spline function, and not in the estimation of \(\nu\).

In order to compare two versions of the same package in the same R session, I forked the github repo, and made a branch with a renamed package. I have called it sharpeRratioTwo because I do not expect it to be used by anyone, and because naming is still a hard problem in CS. To install the package to play along, one can:

library(devtools)
devtools::install_github('shabbychef/sharpeRratio',ref='astwo')

First, I perform some simulations. I draw 128 days of daily returns from a \(t\) distribution with \(\nu=4\) degrees of freedom. I then compute: the moment-based Sharpe ratio; the moment-based Sharpe ratio, but debiased using higher order moments; the drawdown estimator from the 1.1 version of the package, as installed from CRAN; the drawdown estimator from the 1.2 version of the package; the drawdown estimator from the 1.2 version of the package, but feeding \(\nu\) to the estimator. I do this for 20000 draws of returns. I repeat for 256, and 512 days of data, and for the population Signal-Noise ratio varying from 0.25 to 1.5 in "annualized units" (per square root year), assuming 252 trading days per year. I use doFuture to run the simulations in parallel.

suppressMessages({
  library(dplyr)
  library(tidyr)
  library(tibble)
  library(SharpeR)
  library(sharpeRratio)
  library(sharpeRratioTwo)
  # https://cran.r-project.org/web/packages/doFuture/vignettes/doFuture.html
  library(doFuture)
  registerDoFuture()
  plan(multiprocess)
})
# only works for scalar pzeta:
onesim <- function(nday,pzeta=0.1,nu=4) {
  x <- pzeta + sqrt(1 - (2/nu)) * rt(nday,df=nu)
    srv <- SharpeR::as.sr(x,higher_order=TRUE)
    # mental note: this is much more awkward than it should be,
    # let's make it easier in SharpeR!
    ssr <- srv$sr
    ssr_b <- ssr - SharpeR::sr_bias(snr=ssr,n=nday,cumulants=srv$cumulants)

    ssr <- mean(x) / sd(x)
    sim <- sharpeRratio::estimateSNR(x)
    twm <- sharpeRratioTwo::estimateSNR(x)
    # this cheats and gives the true nu to the estimator
    cht <- sharpeRratioTwo::estimateSNR(x,nu=nu)
    c(ssr,ssr_b,sim$SNR,twm$SNR,cht$SNR)
}
repsim <- function(nrep,nday,pzeta=0.1,nu=4) {
  dummy <- invisible(capture.output(jumble <- replicate(nrep,onesim(nday=nday,pzeta=pzeta,nu=nu)),file='/dev/null'))
  retv <- t(jumble)
  colnames(retv) <- c('sr','sr_unbiased','ddown','ddown_two','ddown_cheat')
    invisible(as.data.frame(retv))
}
manysim <- function(nrep,nday,pzeta,nu=4,nnodes=5) {
  if (nrep > 2*nnodes) {
    # do in parallel.
    nper <- table(1 + ((0:(nrep-1) %% nnodes))) 
    retv <- foreach(i=1:nnodes,.export = c('nday','pzeta','nu')) %dopar% {
      repsim(nrep=nper[i],nday=nday,pzeta=pzeta,nu=nu)
    } %>%
      bind_rows()
  } else {
    retv <- repsim(nrep=nrep,nday=nday,pzeta=pzeta,nu=nu)
  }
  retv 
}
# summarizing function
sim_summary <- function(retv) {
    retv %>%
        tidyr::gather(key=metric,value=value,-pzeta,-nday) %>%
        filter(!is.na(value)) %>%
        group_by(pzeta,nday,metric) %>%
        summarize(meanvalue=mean(value),
                            serr=sd(value) / sqrt(n()),
                            rmse=sqrt(mean((pzeta - value)^2)),
                            nsims=n()) %>%
        ungroup() %>%
        arrange(pzeta,nday,metric)
}

ope <- 252
pzeta <- seq(0.25,1.5,by=0.25) / sqrt(ope)

params <- tidyr::crossing(tibble::tribble(~nday,128,256,512),
                          tibble::tibble(pzeta=pzeta))

nrep <- 20000
# can do 2000 in ~20 minutes using 7 nodes.
set.seed(1234)
system.time({
    results <- params %>%
        group_by(nday,pzeta) %>%
            summarize(sims=list(manysim(nrep=nrep,nnodes=7,pzeta=pzeta,nday=nday))) %>%
        ungroup() %>%
        tidyr::unnest()
})
     user    system   elapsed 
79879.368    20.066 28291.224 

(Don't trust those timings, it should only take 3 and a half hours on 7 cores, but I hibernated my laptop in the middle.)

I compute the mean of each estimator over the 20000 draws, divide that mean estimate by the true Signal-Noise Ratio, then plot versus the annualized SNR. I plot errobars at plus and minus one standard error around the mean. I believe this plot is more informative than previous versions, as it clearly shows the geometric bias of the drawdown estimator. As promised, we see that the drawdown estimator consistently estimates a value around 70% of the true value. This geometric bias appears constant across the range of SNR values we tested. Moreover, it is apparently not affected by sample size: we see about the same bias for 2 years of data as we do for half a year of data. The moment estimator, on the other hand, shows a slight positive bias which is decreasing in sample size, as described by Bao and Miller and Gehr. The higher order moment correction mitigates this bias somewhat, but does not appear to eliminate it entirely.

We also see that the bias of the drawdown estimator is not fixed in the most recent version of the package, and does not appear to due to estimation of the \(\nu\) parameter.
On the contrary, the estimation of \(\nu\) appears to make the bias worse. We conclude that the drawdown estimator is still biased, and we suggest that practicioners do not use this estimator until this issue is resolved.

ph <- results %>% 
    sim_summary() %>%
    mutate(metric=case_when(.$metric=='ddown' ~ 'drawdown estimator v1.1',
                                                    .$metric=='ddown_two' ~ 'drawdown estimator v1.2',
                                                    .$metric=='ddown_cheat' ~ 'drawdown estimator v1.2, nu given',
                                                    .$metric=='sr_unbiased' ~ 'moment estimator, debiased',
                                                    .$metric=='sr' ~ 'moment estimator (SR)',
                                                    TRUE ~ 'error')) %>%
    mutate(bias = meanvalue / pzeta,
                 zeta_pa=sqrt(ope) * pzeta,
                 serr = serr / pzeta) %>%
    ggplot(aes(zeta_pa,bias,color=metric,ymin=bias-serr,ymax=bias+serr)) + 
    geom_line() + geom_point() + geom_errorbar(alpha=0.5) + 
    geom_hline(yintercept=1,linetype=2,alpha=0.5) + 
    facet_wrap(~nday,labeller=label_both) +
    scale_y_log10() + 
    labs(x='Signal-noise ratio (per square root year)',
             y='empirical expected value of estimator, divided by actual value',
             color='estimator',
             title='geometric bias of SR estimators')
print(ph)

plot of chunk plot

atom feed · Copyright © 2018-2019, Steven E. Pav.  
The above references an opinion and is for information purposes only. It is not intended to be investment advice. Seek a duly licensed professional for investment advice.