There are many aspects of trading and mathematics that are counterintuitive. I find these particularly intriguing and important because our instincts are incorrect when assessing mathematical concepts like variance, probability, and the distribution of returns. Today, we talk about variance and distribution.
How the choice of frame impacts the viewer’s impression of a painting.
Flat
In Alpha Trader, I wrote a section on counterintuitive math facts, plus a big section discussing P&L variance, because I think that there are many concepts in math and trading that are understood at a superficial level but are hard to grok completely. The idea that an excellent trader can have a losing year not because of bad trading, but because of variance, is hard to accept.
Before I get into today’s piece, let me set the stage with an excerpt from the section on variance from Alpha Trader, in case you have not read the book.
Alpha Trader Excerpt begins
Let’s talk about variance for a few pages because it is a crucial concept in trading. First, here’s a quote that sums it up nicely:
The laws of probability, so true in general, so fallacious in particular.
– Edward Gibbon, 1774
In statistics and probability, variance is the concept that in a series of numbers with a given average, individual data points will be spread out away from the average. So even if your average daily P&L over the last 10 years is +$23,000, you might still lose $25,000 five days in a row sometimes. This may not mean you did anything wrong. It’s just variance.
Variance is a prominent feature in poker. A well-rested professional poker player will inevitably beat a drunk amateur over a long period, say 12 hours. But over any 10-minute or even 30-minute period, the drunk amateur might prevail. The key to understanding variance is that it is most visible in short samples. The larger the sample, the closer the aggregate data will approach the true average.
Luck is a short-term thing. In the long run, skill prevails. But the long run Is longer than most people think.
As Keynes famously said… “In the long run, we are all dead.” A very solid trading strategy with significant edge generates substantial variance, even year to year. This is why it is incredibly difficult to evaluate traders, portfolio managers and hedge funds. Was it luck or was it skill? Michael Mauboussin’s book: The Luck Equation: Untangling Skill and Luck in Business is the go-to book on this topic.
To give you a sense of how variance impacts a year of performance in trading, let’s run a few simulations. For simplicity, we’ll use a static risk management approach where the trader does not change their risk appetite during the year, regardless of performance. That said, I strongly advocate increasing your risk as your P&L rises.
I include these simulations because I think very few people, even expert traders, truly appreciate just how dramatic the impact of variance can be on performance. It is probably a defense mechanism. If we truly appreciated how important variance was, we would feel like we barely control our own fate. And humans love to believe they are in control.
The first simulation approximates the performance of a retail trader. She makes money 50% of days and loses money 50% of days. Her average winning day (net P&L) is +$2,000 and her average losing day is minus $1,200. Based on these figures, she will expect to make $100,800 in an average year. Here is her cumulative YTD P&L simulated 100 times.
The same trader using the same strategy in a world with the same probabilities might make $50,000 one year and $150,000 the next, with variance—and literally nothing else—to blame.
The second simulation represents a bank trader. He makes money 55% of days and loses money 45% of days. His average winning day (net P&L) is +$150,000 and his average losing day is minus $120,000. Based on these figures, his expected value is +$7.2 million. But here is how he fares in 100 simulations:
You can see that despite employing a process with substantial edge and positive EV, a trader’s annual performance is all over the place simply due to variance. This is a big reason why it is so flipping difficult for management to properly assess traders and for traders to evaluate themselves. It’s easy to be fooled by randomness!
One of the most difficult and most important skills in trading is the ability to distinguish between variance and bad decision making. If you are losing money right now, is it because your process has broken down, your discipline sucks and the market has completely changed? Or is it because of the inevitable variance within the P&L distribution?
When you are on a bad streak, you need to ask yourself this. The longer you have been trading, the easier this is to answer (usually). For me, I can usually look back on a losing streak and determine with a fair amount of confidence whether it was variance. When you are starting out, it’s almost impossible to tell.
Here are some examples of how I look at a losing streak and determine whether it’s a product of variance or bad process.
Variance
When I am wrong and I lose money, I barely care. That’s trading. As long as I have been sticking to my risk management system, I shrug and move on. Being wrong is where variance comes from in trading. Don’t worry about it. On the other hand, some things can’t be attributed to variance.
Not Variance
If losses stem from any of 1 to 9 above; that’s not variance. That is bad trading, bad process or bad mindset. Cut all your risk and figure out how to get things back on track. When you trust yourself to do the right thing, you are trading well. It should not matter all that much whether you make or lose money on any given day.
When you are trading with a good mindset, you see each trade and each day as a series of independent events with a probabilistic nature and you understand that some trades and some days you will lose money. Each day, you come in and do your best without getting too high or too low. You invest most of your energy in the process and very little in the outcome. Each trade, you go through a thoughtful process and execute in a rational and systematic way.
If you trade sloppy, don’t say “I suck” or “I traded SO bad today”. Instead ask: “Why was I so sloppy today? Am I burnt out? Discouraged? Disengaged?” Robotic application of a process keeps you sane and lets you know you can trust yourself. If you break your stop loss repeatedly, you should quickly figure out why. Bad discipline? No automation of stop loss? Don’t care anymore?
If you go through the above lists (variance vs. not variance) and are still not sure whether a recent losing streak is caused by variance or bad trading, take a step back. The more experience you have, the easier it is to make this determination but if you are new, it can be very difficult, especially if you tend to be critical of yourself.
Traders with a healthy mindset who are confident in their process have a much easier time dealing with losing streaks because they can confidently identify the cause as variance and move on. Traders with a less healthy mindset or less confidence will tend to assume that a losing streak is the result of bad decisions and a lack of trading skill.
Alpha traders with a solid process and excellent risk management still hit losing streaks. In the long run, a solid process will win but in the short run, there are always going to be ups and downs due to variance. If you make money 50% of days, you can expect to lose money 6 or 7 days in a row once in a while. That is fine! When you lose money six days in a row and determine that it is simply because of variance, you need to double down on your process and believe in what you are doing more than ever. Variance can be managed, but never controlled.
Get comfortable with variance.
If you understand your edge and can confidently explain it, you will be much more comfortable with variance. Furthermore, if you continually spend time thinking about the current regime and how to adapt to present market conditions, you will reduce your variance and maintain steadier performance over time. No matter how good you are, though, you still need to understand that variance is a thing.
End of Alpha Trader excerpt
Much as variance in P&L can be hard to fathom, the variance in hourly returns in FX can also be hard to fully internalize.
When we watch FX markets, we have an intuitive sense that they are sitting there doing nothing quite a lot and then they make meaningful moves on data, huge flows, and spurts of activity. Quoting one more time from Alpha Trader:
In World War 1, there was a famous quote:
“Modern warfare is months of boredom punctuated by moments of extreme terror”.
I would paraphrase here and say modern trading is hours of boredom punctuated by minutes of extreme terror. Things are quiet most of the time but what matters is how you act and react when the guano hits the rotating blades.
So, you know EURUSD is mostly just sitting there going nowhere. But what does the distribution of hourly returns actually look like? If we take all the 2459 hours in 2024 and calculate the change in EURUSD, then sort it and plot it, we get this:
Alternatively, we can make a histogram that looks like this:
These charts will look similar for most of G10 (ex-USDJPY of course) but EM will have more skew, like this:
Note you probably remember USDMXN moved much more than 2% that crazy night in April, but the hourly change was only 2% because it went up and down so fast
I was surprised by the stackedness in the middle of the distribution. My takeaways from this data are:
Finally, note that if you conduct this same exercise with daily data, you get something similar, but not nearly as skewed and not nearly as stacked in the center. Food for thought.
No first-tier data this week but loads of very high second-tier stuff. Tomorrow is a good day for events.
Sol made an interesting point: Much as DOGE peaked at 70 cents the night Elon Musk went on Saturday Night Live… Could the GLP-1 basket top out on May 24, as public awareness hits a peak and the theme is certainly mainstream à la magazine cover indicator? When a comedy show is referencing a market theme, you know the narrative is priced in.
“Another success – ‘possibly my favorite’, says Schade – was the reframing of Meindert Hobbema’s The Avenue at Middelharnis (1689). It was previously displayed in an 18th-century gilt frame punctuated by ornate decoration, originally designed for a portrait.
‘For landscapes, such frames don’t really work because there is no real emphasis in the composition at these points, which pulls up the horizon’s edges,’ he says, gesturing to the decorative elements. The painting is now in a narrow black frame that ‘lets the horizon stay down. The road seems to spill out into the room, whereas the previous frame made a kind of barrier.’”
This piece is mostly for people who are interested in backtesting
