When it comes to analyzing smart beta indices and ETFs, heavy reliance on backtesting results can be misleading if potential investors are unaware of the limitations of simulated results, according to research conducted by Research Affiliates LLC of Newport Beach, Calif.
Research Affiliates found that about two-thirds of smart-beta index track records are backtests, and that most real life track records extend no longer than a decade due to the newness of many products. Most of the investment outcomes reported by smart beta providers are therefore from simulations, Research Affiliates says in a recent report.
Research Affiliates says a backtest is a frequently used tool used to frame forward-looking return expectations, but backtests are subject to “overfitting” or “data-snooping.” This means firms are selectively mining for data and time periods that show a positive bias. Backtesting also typically ignores transaction costs that in actuality would affect investor returns.
“Like the producers of the Oscars and other live shows such as Saturday Night Live, investors don’t have the luxury of retakes — investing committed capital is ‘live’,” says the report. “Backtests, like big movie productions, can be misleading.”
The outperformance observed before a typical smart beta index is launched virtually disappears once it’s live, yet most investors are making decisions on backtest results, the report says.
Research Affiliates examined the performance of 125 U.S. equity smart beta indices on which ETFs defined as ‘strategic beta’ by Morningstar Inc. of Chicago are based. The average available history of this universe was seven years.
Research Affiliates found that prior to launch the indices tended to have superior performance relative to a market-capitalization-weighted benchmark, with performance peaking about six months ahead of the launch date. The outperformance was strongest during the three-year period ahead of the launch.
After the indices officially launched, their performance relative to their benchmark index appears to hover around the base line, exhibiting virtually none of the outperformance demonstrated before they were live.
Only 12 of the 125 indices showed significantly negative relative performance in the selected backtests, while the number of underperformers almost tripled once the indices were actually launched. The results indicated that many index-creators and ETF providers basing product on these indices appear to “trend-chase” performance in creating their products.
“We all know there are no ugly backtests,” the report says. “More precisely, ugly-looking backtest results are rarely published in journals or client-facing materials. In the academic world, publication bias is well-recognized, meaning that statistically significant results are three times more likely to be published than insignificant ones.”
By expecting lower performance than backtest results show, questioning how those results were achieved, and selecting a strategy built on sound economic theory, smart beta investors can form more realistic performance expectations, the report says.
“Investors nearly always base their decisions on backward-looking, frictionless results,” the report says. “We have no problem with that, if it’s done with eyes wide open.”
The report also suggests that seeking simplicity in smart beta strategies may reduce risk for investors. There is more “freedom in data mining” when it comes to complex methodologies, thus index providers have more “knobs to turn” in seeking to show outperformance in the outcomes shown in their back tests.
Photo copyright: hafakot/123RF