Regulators finally released their long-awaited “mystery shop” report in mid-September. Unfortunately, what could have been a fascinating research exercise is completely undermined by the lack of data that it produced.

The joint research project undertaken by the Ontario Securities Commission, the Investment Industry Regulatory Organization of Canada (IIROC) and the Mutual Fund Dealers Association of Canada was supposed to investigate the client/advisor experience from the client’s point of view. An outside research firm fielded a team of mystery shoppers to test the advice experience at investment dealers, mutual fund dealers, portfolio managers and exempt market dealers (EMDs). The regulators aimed to carry out 150 mystery shops, with researchers posing as prospective clients of randomly selected advisors.

Despite having 580 firms and more than 47,000 advisors to choose from, the shoppers managed to carry out only 105 mystery shops, and only 88 of those produced sufficient data for analysis.

Apparently, the researchers had a particularly tough time getting meetings with the portfolio managers and EMDs, as only 13 portfolio managers and 11 EMDs generated enough data for the research.

Of the 88 total mystery shops, just 24 resulted in actual recommendations from the advisors – 11 from fund dealer reps, seven from EMDs and just six from IIROC reps. Given the paucity of the data – a sample rate of less than 0.2% – it is impossible to draw any meaningful conclusions from the data that was collected in this exercise.

This is disappointing because the actual investor experience is often what’s missing from debates about appropriate regulation for the retail investment business. Ensuring that proper “know your client” information is collected from clients, and that required disclosures about conflicts and compensation are made, is the objective of so much regulatory effort. Yet, there’s so little real-world data informing these activities.

Unfortunately, the mystery shop project looks like a major missed opportunity to change that fact.

© 2015 Investment Executive. All rights reserved.