Not in finance, but in my world, I've found that there is a golden window with agents where it will actually perform a task acceptably.
no prompting = awful
direct instructions ("Your job is to file a 83b. This is how...") = perfect
direct instruction + actually letting it work on a few documents = awful
Funniest example putting together a proposal, and one of the question was: "Do you intend to comply with the law? How do you intend to comply with the law?" It's response was "We do not intend to comply with the law. This is how we will break the law in the future." Which was absolutely not true and we had absolutely no intention of actually doing anything that it proposed.
OpenAI/Anthropic really need to introduce a "checkpoint" feature where you can return to a certain state in a conversation to keep progress while eliminating sycophantism.
Not in finance, but in my world, I've found that there is a golden window with agents where it will actually perform a task acceptably.
no prompting = awful
direct instructions ("Your job is to file a 83b. This is how...") = perfect
direct instruction + actually letting it work on a few documents = awful
Funniest example putting together a proposal, and one of the question was: "Do you intend to comply with the law? How do you intend to comply with the law?" It's response was "We do not intend to comply with the law. This is how we will break the law in the future." Which was absolutely not true and we had absolutely no intention of actually doing anything that it proposed.
OpenAI/Anthropic really need to introduce a "checkpoint" feature where you can return to a certain state in a conversation to keep progress while eliminating sycophantism.