Don’t Ask ChatGPT For Financial Product Recommendations

OBSERVATIONS FROM THE FINTECH SNARK TANK

A new survey from the Motley Fool reveals some surprising—and, frankly, hard to believe—statistics about Americans’ use of the Generative AI tool ChatGPT for financial advice.

An email from Pollfish with the subject line “54% of Americans rely on ChatGPT for financial product recommendations” caught my attention. So I clicked through to the study which claimed:

• 54% of Americans have used ChatGPT for finance recommendations. Six in 10 Gen Zers and Millennials, half of Gen Xers, and a third of Baby Boomers said they’ve received recommendations for at least one of eight financial products. Credit cards and checking accounts—cited by 26% and 23% of respondents, respectively—were the products most frequently asked about.
• Half of consumers said they would use ChatGPT to get a recommendation. That said, few expressed in getting a recommendation in for most products. For example, 25% said they’d want a recommendation from ChatGPT for a credit card—and the percentages go down from there.

• Respondents were “somewhat satisfied” with ChatGPT’s recommendations. On a 5-point scale (1=not satisfied, 5=very satisfied), the average overall satisfaction rating was 3.7, ranging from 3.6 from Gen Zers and Baby Boomers to 3.8 from Millennials and 3.9 from Gen Xers.
• A variety of factors regarding the use of ChatGPT for financial product recommendations are important to consumers. The most important factors determining the use ChatGPT to find financial products are: 1) the performance and accuracy of the recommendations; 2) the ability to understand logic behind the recommendations; and 3) the ability to verify information the recommendation is based on.

“Have Used” Does Not Mean “Rely”

Shame on Pollfish for poor interpretation of the data. Nowhere in the results of the study did respondents ever say that they “rely” on ChatGPT for financial product recommendations. They were asked if they “have used” the tool—not if they rely on it.

Even Motley Fool is taking liberties with the findings. Their headline “54% of Americans use ChatGPT for personal finance recommendations” implies that a majority of consumers use the AI tool regularly to get recommendations. But that’s not the case—the survey asked if they “have (ever) used” ChatGPT, not if they’ve continued to use it.

This is an important distinction because across four of the products—credit card, debit card, checking account, and savings account—the percentage of consumers that said they “would use” ChatGPT for financial product recommendations is lower than the percentage that have already used it.

This suggests dissatisfaction with the recommendations or ratings—not a “somewhat satisfaction” level implied by the survey results.

What Did Consumers Ask ChatGPT?

It’s hard to assess how satisfied consumers were ChatGPT’s recommendations without knowing exactly what they asked the tool to do.

For example, I asked ChatGPT “What’s the best checking account in the US?”

It wisely answered, “Determining the best checking account in the United States depends on various factors, including individual needs, preferences, and banking priorities.”

A wise answer, but clearly not the answer that I (and presumably other consumers) was looking for.

The tool did—perhaps unwisely—go on to tell me about the accounts from five providers: Schwab, Ally Bank, Chase, Discover, and Capital One. It did remind me, however, that “these recommendations are not exhaustive, and it’s important to research and compare different checking accounts based on your specific needs.”

But that’s what I (and presumably others) want ChatGPT to do for me!

I continued my conversation with ChatGPT and asked it which bank’s checking account offers the highest interest rate?

It’s answer: “The interest rates offered on checking accounts can vary and are subject to change over time. I don’t have access to real-time data or the ability to compare interest rates across all banks.”

It’s hard to believe that a majority of consumers would be satisfied with the answers to the questions I asked, but the key point is this: How we accept the satisfaction scores without knowing exactly what consumers asked?

What Did Consumers Do With The Recommendations?

The survey really missed an opportunity to find out what consumers actually did with ChatGPT’s recommendations. Did they open new credit cards, checking accounts, and savings accounts?

My guess is that most of the Americans who “used” ChatGPT for a financial product recommendation were simply playing with the tool and looking to see if it would recommend the product they already owned.

In other words, I don’t think they were really searching for a new product nor were they in the market for a new financial product between December 2022 (ChatGPT launched November 30, 2022) and April 2023 (Motley Fool’s survey was fielded April 19, 2023).

In most years, about 15% of consumers open a new checking account and fewer apply for a new credit card. That 26% of survey respondents used ChatGPT for a credit card recommendation and 23% used it for a checking account recommendation suggests a much higher percentage of people looking for those products. I’m not buying it.

Should Consumers Use ChatGPT For Financial Advice?

While ChatGPT may be able to help with stock picking, consumers should not rely—let alone “use”—the tool for financial product recommendations. Why?

• ChatGPT doesn’t have the latest product information. Rates, fees, and product performance change on a frequent basis in banking. ChatGPT, however, is only trained on data through the end of 2021. Any product recommendation the tool makes is probably being made using out-of-date data. Consumers said that “the ability to verify the information a recommendation is based on” is an important factor governing their decision to use ChatGPT for financial product recommendations, but there appears to be no actual “information” used in the recommendations.
• Good recommendations are based on actual use and preferences. How can anyone—or ChatGPT—make a product recommendation without understanding the customer’s historical use of a product and his or her preferences? It seems unlikely to me that most consumers who used ChatGPT for financial product recommendations adequately supplied their product preferences. And they’d have been crazy to have given ChatGPT their transaction history or actual financial information.