Chat GPT is usually wrong on financial matters

Fortune said:
I think AI is fine for this type of financial 'coaching', but it's not to be trusted where there's an analysis of numbers required.
Click to expand...
I disagree based on extensive recent experience using ChatGPT to analyse some household budget and medium/long term early retirement scenarios. As ever I did double check things and correct it if it got something wrong but, by and large, it was useful and accurate.
 
Dr Strangelove said:
I’d say that’s pretty specific
Click to expand...
I'd disagree. It's the question that is specific.

Most of the advice there relatively objective, and doesn't really analyse the numbers. A quick glance at the numbers and I can see some errors (the tax relief jumps out). They don't materially affect the advice given (as it's not based on the numbers) but are wrong all the same. I note it also states that rental property investments are liquid.
 
ClubMan said:
I disagree based on extensive recent experience
Click to expand...
Let's look at the example posted.


"Eliminating a €200k mortgage at 3.5% would save approximately €7,000 annually in interest, totaling €140,000 over 20 years."

Without even thinking hard about it at all, the €7k figure could only be correct for an interest only mortgage. Some quick calculations give a saving of c€78k, not €140k.

"This aligns with the “debt avalanche” method, targeting high-interest obligations first."
Hardly true of a mortgage.

"Tax Relief: Contributing €30k to a pension would yield €12,000 in tax savings (40% relief on €50k income), effectively costing €18,000 net."

Standard rate cut off is €44k, meaning not all of this contribution will get tax relief at 40%. No mention of backdating to the precious year.

"With a €100k pension at age 40, the user is below the recommended benchmark (1x annual salary by 30, 3x by 40)."
This is just rubbish
 
Fibbernacci said:
Did you sense any pattern in the type of questions it got wrong or right?
Click to expand...
I didn't see any particular pattern. Numbers were sometimes right, sometimes wrong. It was almost as if it was someone not great with numbers doing the calculations (I suppose that's likely the case in the days it was trained on). Sometimes it hallucinated regulations that didn't exist, or just got them backwards (something allowed wasn't allowed, etc).
 
I did a test myself there with a specific question related to my industry and it gave a wrong answer. When I enquired why, it said that it doesn't parse text every time and uses 'generative' mode to make predictions. I then asked if I had specifically worded my question to force it to parse the text, would it have done that and it said it would have switched to 'parsing' mode. If that's true I wonder if the user could possibly increase accuracy of the responses by wording the questions in a particular way.

(but still not take financial advice from it lol)
 
Fibbernacci said:
If that's true I wonder if the user could possibly increase accuracy of the responses by wording the questions in a particular way.
Click to expand...
I’ve been using LLMs since late 2022.

You can materially improve the quality of the replies by changing how you do with the prompts. All I can advise is for people to play around a little bit with it. The more precise your prompt often the better quality the response. But if you badger an LLM it will hallucinate a false response in many cases.
 
Back
Top