## A Sharper Sharpe: Its Biased.

In a series of posts, we looked at Damien Challet's 'Sharper estimator' of the Signal-Noise Ratio, which computes the number of drawdowns in the returns series (plus some permutations thereof), then uses a spline function to infer the Signal-Noise Ratio from the drawdown statistic. The spline function is built from the results of Monte Carlo simulations.

In the last post of the series, we looked at the apparent bias of this 'drawdown estimator'. We suggested, somewhat facetiously, that one could achieve similar properties to the drawdown estimator (reduced RMSE, bias, etc.) by taking the traditional moment-based Sharpe ratio and multiplying it by 0.8.

I contacted Challet
to present my concerns. I suspected that the spline function was trained
with too narrow a range of population Signal-Noise ratios, which would result
in this bias, and suggested he expand his simulations as a fix. I see that
since that time the `sharpeRratio`

package has gained a
proper github page, and
the version number was bumped to 1.2.
(It has not been released to CRAN, so it is premature to call it "the" 1.2 release.)

In this post, I hope to:

- Demonstrate the bias of the drawdown estimator in a way that clearly illustrates why "Sharpe ratio times 0.8" (well, really 0.7) is a valid comparison.
- Check whether the bias has been corrected in the 1.2 development version. (Spoiler alert: no.)
- Provide further evidence that the issue is the spline function, and not in the estimation of \(\nu\).

In order to compare two versions of the same package in the same `R`

session,
I forked the github repo, and made a branch with a renamed package. I have
called it `sharpeRratioTwo`

because I do not expect it to be used by anyone,
and because naming is still a hard problem in CS. To install the package to
play along, one can:

```
library(devtools)
devtools::install_github('shabbychef/sharpeRratio',ref='astwo')
```

First, I perform some simulations. I draw 128 days of daily returns from a \(t\)
distribution with \(\nu=4\) degrees of freedom.
I then compute: the moment-based Sharpe ratio;
the moment-based Sharpe ratio, but debiased using higher order moments;
the drawdown estimator from the 1.1 version of the package, as installed from CRAN;
the drawdown estimator from the 1.2 version of the package;
the drawdown estimator from the 1.2 version of the package, but feeding
\(\nu\) to the estimator.
I do this for 20000 draws of returns.
I repeat for 256, and 512 days of data,
and for the population Signal-Noise ratio varying from
0.25 to 1.5 in "annualized units" (per square root year), assuming
252 trading days per year.
I use `doFuture`

to run the simulations in parallel.

```
suppressMessages({
library(dplyr)
library(tidyr)
library(tibble)
library(SharpeR)
library(sharpeRratio)
library(sharpeRratioTwo)
# https://cran.r-project.org/web/packages/doFuture/vignettes/doFuture.html
library(doFuture)
registerDoFuture()
plan(multiprocess)
})
# only works for scalar pzeta:
onesim <- function(nday,pzeta=0.1,nu=4) {
x <- pzeta + sqrt(1 - (2/nu)) * rt(nday,df=nu)
srv <- SharpeR::as.sr(x,higher_order=TRUE)
# mental note: this is much more awkward than it should be,
# let's make it easier in SharpeR!
ssr <- srv$sr
ssr_b <- ssr - SharpeR::sr_bias(snr=ssr,n=nday,cumulants=srv$cumulants)
ssr <- mean(x) / sd(x)
sim <- sharpeRratio::estimateSNR(x)
twm <- sharpeRratioTwo::estimateSNR(x)
# this cheats and gives the true nu to the estimator
cht <- sharpeRratioTwo::estimateSNR(x,nu=nu)
c(ssr,ssr_b,sim$SNR,twm$SNR,cht$SNR)
}
repsim <- function(nrep,nday,pzeta=0.1,nu=4) {
dummy <- invisible(capture.output(jumble <- replicate(nrep,onesim(nday=nday,pzeta=pzeta,nu=nu)),file='/dev/null'))
retv <- t(jumble)
colnames(retv) <- c('sr','sr_unbiased','ddown','ddown_two','ddown_cheat')
invisible(as.data.frame(retv))
}
manysim <- function(nrep,nday,pzeta,nu=4,nnodes=5) {
if (nrep > 2*nnodes) {
# do in parallel.
nper <- table(1 + ((0:(nrep-1) %% nnodes)))
retv <- foreach(i=1:nnodes,.export = c('nday','pzeta','nu')) %dopar% {
repsim(nrep=nper[i],nday=nday,pzeta=pzeta,nu=nu)
} %>%
bind_rows()
} else {
retv <- repsim(nrep=nrep,nday=nday,pzeta=pzeta,nu=nu)
}
retv
}
# summarizing function
sim_summary <- function(retv) {
retv %>%
tidyr::gather(key=metric,value=value,-pzeta,-nday) %>%
filter(!is.na(value)) %>%
group_by(pzeta,nday,metric) %>%
summarize(meanvalue=mean(value),
serr=sd(value) / sqrt(n()),
rmse=sqrt(mean((pzeta - value)^2)),
nsims=n()) %>%
ungroup() %>%
arrange(pzeta,nday,metric)
}
ope <- 252
pzeta <- seq(0.25,1.5,by=0.25) / sqrt(ope)
params <- tidyr::crossing(tibble::tribble(~nday,128,256,512),
tibble::tibble(pzeta=pzeta))
nrep <- 20000
# can do 2000 in ~20 minutes using 7 nodes.
set.seed(1234)
system.time({
results <- params %>%
group_by(nday,pzeta) %>%
summarize(sims=list(manysim(nrep=nrep,nnodes=7,pzeta=pzeta,nday=nday))) %>%
ungroup() %>%
tidyr::unnest()
})
```

```
user system elapsed
79879.368 20.066 28291.224
```

(Don't trust those timings, it should only take 3 and a half hours on 7 cores, but I hibernated my laptop in the middle.)

I compute the mean of each estimator over the 20000 draws,
divide that mean estimate by the true Signal-Noise Ratio,
then plot versus the annualized SNR.
I plot errobars at plus and minus one standard error around the mean.
I believe this plot is more informative than previous
versions, as it clearly shows the geometric bias of the
drawdown estimator.
As promised, we see that the drawdown estimator consistently
estimates a value around 70% of the true value.
This geometric bias appears constant across the range of
SNR values we tested.
Moreover, it is apparently *not* affected by sample size:
we see about the same bias for 2 years of data as we do for
half a year of data.
The moment estimator, on the other hand, shows a slight positive
bias which is decreasing in sample size, as described by Bao and
Miller and Gehr.
The higher order moment correction mitigates this bias somewhat,
but does not appear to eliminate it entirely.

We also see that the bias of the drawdown estimator is not fixed
in the most recent version of the package,
and does not appear to due to estimation of the \(\nu\) parameter.

On the contrary, the estimation of \(\nu\) appears to make the bias *worse*.
We conclude that the drawdown estimator is still biased, and
we suggest that practicioners do *not* use this estimator
until this issue is resolved.

```
ph <- results %>%
sim_summary() %>%
mutate(metric=case_when(.$metric=='ddown' ~ 'drawdown estimator v1.1',
.$metric=='ddown_two' ~ 'drawdown estimator v1.2',
.$metric=='ddown_cheat' ~ 'drawdown estimator v1.2, nu given',
.$metric=='sr_unbiased' ~ 'moment estimator, debiased',
.$metric=='sr' ~ 'moment estimator (SR)',
TRUE ~ 'error')) %>%
mutate(bias = meanvalue / pzeta,
zeta_pa=sqrt(ope) * pzeta,
serr = serr / pzeta) %>%
ggplot(aes(zeta_pa,bias,color=metric,ymin=bias-serr,ymax=bias+serr)) +
geom_line() + geom_point() + geom_errorbar(alpha=0.5) +
geom_hline(yintercept=1,linetype=2,alpha=0.5) +
facet_wrap(~nday,labeller=label_both) +
scale_y_log10() +
labs(x='Signal-noise ratio (per square root year)',
y='empirical expected value of estimator, divided by actual value',
color='estimator',
title='geometric bias of SR estimators')
print(ph)
```