Markets are full of feedback loops, so you can't expect the same results with "paper" backtesting or "paper" forward testing as with real trades, especially in larger amounts.
With such paper testing you can discover and fool yourself with amazing high-probability, high-earning strategies that come with the hidden surprise of low probability catastrophic losses.
For example, many naive gamblers think that a strategy with a 45% chance to win, combined with betting to cover losses, will nearly always succeed because the odds of losing ten times in a row seem low. However, when the inevitable 11th loss occurs, it can be devastating.
eg I could also say that you could fool yourself believing that investing in the S&P 500 index is a strategy with high-probability of performing well, with a hidden surprise of low-prob cat losses.
You ask a good question.
Consider a game like rock-paper-scissors. The goal is to predict your opponent's move. Among all possible strategies, the random strategy is unique because it can't be predicted. Any other strategy can be anticipated by a smarter opponent. Being random means you can't lose to a smarter opponent.
In trading, buying low and selling high requires prediction. Any strategy that relies on prediction becomes predictable to smarter opponents. Buy-and-hold index investing is the only strategy immune to this exploitation because you're effectively betting on everything. With passive index investing, your investments grow at the rate of business growth, making this strategy special.
After two years of research, you have developed a trading strategy that, compared to the S&P 500, "has underperformed, especially the way the market has been"? If your strategy depends on buying / selling it will trigger taxes each time this happens, yes? Short term gains taxes? When people switch from sp500 to your strategy how much less will then earn over a decade? Yet you claim your strategy "works"?
> The expected performance of the strategy is somewhere between 10% and 30%, but it is not clear what exactly.
If a profitable pattern exists, what makes you think nobody else has found it and that it will continue to exist in the future? Is it because few look for these profitable patterns? Are they poorly funded? Do they lack incentives to do it?
> I don't think it's necessarily fair to call others "dumb money". They can hold for longer and still profit with the market.
"Dumb money" refers to hobby traders who believe stock trading will make them rich. Some rely on gut feelings, while others use advice from astrology or technical analysis. Why do you suppose professional traders pay for order flow from such traders?
> The market is not a zero sum game because the market capitalization is several multiples of the total invested funds. The market is a money printer in its own way.
For every trade, there is a counterparty on the other side, and your gain is their loss and vice versa. This makes trading a zero-sum activity. Furthermore, for every trade, there are middlemen who charge fees, making trading a negative-sum activity.
Long-term buying and holding is positive-sum investing and does not require a trading strategy or searching for profitable patterns.
> Precious metals as a currency do not work because the government doesn't have the freedom to mint an unlimited supply of hard assets.
The purpose of a currency is to facilitate transactions and maintain stable prices in the short term while losing value in the long term to discourage hoarding. Gold cannot achieve this because the amount of gold cannot be adjusted to match the size of the economy as it grows or shrinks. When the economy grows, using gold as money will cause it to stall because people will save up gold as it becomes more valuable instead of spending it. Deflationary currencies like gold are detrimental to economic growth, which is why modern countries no longer use them.
> The government holds a gun to people's heads, locking them up if they don't use the highly inflationary national currencies.
Are you surprised by inflation? Saving your wealth using a currency does not make sense since it is designed to lose value over time.
If you prefer precious metals, what prevents you from keeping your wealth in gold (a negative-sum asset) and converting it to fiat currency only when needed?
> Countries are run by politicians, bureaucrats, and armed police mafia who care about their paycheck and pensions which wouldn't be so big if not for free moneyprinting at the expense of the citizenry. The people do not have freedom of their choice of money.
Countries are "owned" by their voters. Government debts are your debts. Inflation is just another tax like other taxes. If you are dissatisfied with the current situation, get involved to change it.
You make it sound like you are improving yet with all traders like you "chasing it anyway, refining the strategy at each failure" what exactly are you improving? When you play a game like "rock paper scissors" what refinement is possible?
Due to market feedback loops the patterns you look for profits are transient. When they do persist do you ever wonder about them being like fake "tells" in poker that lure you to bet big setting you up to be smashed later?
High frequency trading / front running information and arbitrage may be profitable but require low latency costly access.
Apart for that my impression is that technical analysis based short term trading (as opposed to long term investing) is on net a negative sum activity. Like a game of "rock paper scissors" there is nothing to improve.
Many of these strategies stopped working in 2008 because the markets became too crowded with players exploring them. Especially the ones that have low drawdowns attract a lot of competition. The writing was already in the wall with the quant bloodbath of 2007.
During the period from 2000 to 2013, the S&P 500 did experience significant drops, particularly during the dot-com bubble burst in the early 2000s and the financial crisis of 2007-2008. The largest drawdowns in this period were:
1. *Dot-com Bubble (2000-2002)*: The S&P 500 fell significantly after the peak in March 2000, dropping about 49% until it bottomed out in October 2002.
2. *Global Financial Crisis (2007-2009)*: The index again suffered a major drop, losing approximately 57% of its value from its peak in October 2007 to its low in March 2009.
However, these drawdowns did not last continuously for 13 years, nor did they result in a cumulative drawdown of 60% sustained over that entire period. The S&P 500 recovered from these lows and even reached new highs within the timeframe specified.
Where does the data come from?
https://data.nasdaq.com/databases/SFA
It's a great survivorship-bias-free dataset.
Regarding tools, I use Python. I wrote the backtesting software many, many, many years ago during my Master's degree, and I've been refining it ever since.
It's an event-driven engine (they are slower than vector-based engines, but they are easier to write strategies for, understand, and debug) with all the bells and whistles, similar to the late Zipline. In fact, I tried most of the Python backtest engines that exist, and that's why I prefer to use what I built over the years: I have 100% understanding of what’s happening and 100% control.
I’m thinking about open-sourcing it… anyway, the logic is not that complicated.
