Canada’s big banks reported record annual profits for 2013, buoyed by a strong domestic market, but stormier seas may lay ahead as the financial institutions go into the new year.

With Canada’s housing market anticipated to cool, and consumers already carrying record amounts of debt, the banking sector will face challenges in finding growth.

“I think it is going to be pretty sluggish growth for loans here, especially if the housing market starts to slow down,” said Ian Nakamoto, a portfolio manager at MacDougall Investment Counsel.

“I also think the demographics are starting to bite them in Canada. Meaning, as you get older, people tend to borrow less, they tend to pay down their obligations.”

It has been a strong year for Canadian bank stocks with the sector up roughly 20 per cent for the year, however shares in several of the banks were sold off this week as the results came in.

Bank of Nova Scotia (TSX:BNS), the last of the big Canadian banks to report its quarterly and annual results, capped off the sector Friday with a record annual profit of nearly $6.7 billion, but fell just shy of investor expectations for its latest quarter.

The bank earned a fourth-quarter profit of $1.7 billion or $1.30 per share, up from $1.52 billion or $1.18 per share a year ago. However, Scotiabank’s core earnings were $1.31 per share, compared with an average analyst estimate of $1.32 compiled by Thomson Reuters.

As a group, the Big 5 Canadian banks — Royal Bank (TSX:RY), TD Bank (TSX:TD), Scotiabank, Bank of Montreal (TSX:BMO) and CIBC (TSX:CM) — earned $29.25 billion in their 2013 financial year, up from $27.81 billion in 2012.

But even as the earnings piled up, several of the banks trimmed staff in the fourth quarter.

The Bank of Montreal cut the equivalent of nearly 1,000 positions in the fourth quarter in a bid to reduce expenses, while CIBC’s number of full-time equivalent employees dipped by more than 450.

Royal Bank had about 1,100 fewer full-time equivalent employees compared to the third quarter.

“They are probably reading the writing on the wall as to what is going to be happening over the next several years,” Nakamoto said.

But despite the concerns, Nakamoto said the effects will be felt less by Canadian banks that have diversified operations from retail banking to insurance, wealth management and capital markets operations.

Stan Wong, a portfolio manager at investment firm Richardson GMP, said the Canadian bank stocks and their growing dividends will likely remain attractive to investors looking for cash.

“With interest rates potentially on the rise, you want to find dividend stocks that are growing their yields — companies that are able to grow their yields — and banks are a very good target for those types of investors,” Wong said.

“Amongst the banks, I would say that TD and Scotiabank are probably the ones that will grow their dividends the quickest over the next several years.”

Driving the latest quarterly results at Scotiabank was strong growth at its Canadian banking operations, boosted by the acquisition of ING Direct Canada as well as improved margins, lower loan loss provisions and asset growth.

Return on equity for the quarter was 15.7 per cent, down from 16.4 per cent a year ago.

Scotiabank president and chief executive Brian Porter said the 2013 results reflected strong performance from its global wealth management and insurance business, Canadian banking and international and capital markets business.

“We continue to benefit from well balanced and diversified contributions from each of our business lines across our unique geographic footprint,” Porter said.

The bank’s Canadian banking group earned $593 million for the quarter, up from $481 million a year ago, while its international operations earned $420 million for the quarter, up from $401 million in the fourth quarter of 2012.

The international operations also saw a $25 million after-tax gain on the sale of a non-strategic business in Peru this quarter.

Barclays analyst John Aiken noted Scotiabank’s international business saw its margins squeezed.

“We do not view the combined earnings negatively, and believe that there are several positives outside of the margin issues in international banking,” Aiken wrote in a note to clients.

The bank’s global wealth management business earned $318 million, up from $294 million a year ago on growth in its mutual fund and insurance business. Global banking and markets earned $336 million, down from $395 million.

For the full year ended Oct. 31, Scotiabank reported a profit of $5.15 per share, compared with a profit of $6.47 billion or $5.22 per share a year ago.

Revenue for the full year was $21.3 billion, including $5.4 billion in the fourth quarter. A year earlier, Scotiabank had $19.7 billion of revenue for fiscal 2012, including $1.83 billion for the fourth quarter.