Why an AI bot may reply twice: what to check before launch
In our own product, a retry covered the model request, saving the reply and sending a notification. A failure in the last step could restart the whole sequence. A code review caught the risk before release.
Kirill Shkolenko, founder of IOANN
6 min read
The retry covered the whole step
A user sends text. The model prepares a reply, the system saves it and notifies the user. If the model has not replied within 15 seconds or the provider returns an error, the code makes another attempt. In the ordinary checks, the model replied in two or three seconds, so this retry never ran.
A colleague noticed that any error inside the step could trigger the retry. Suppose the reply is saved and the notification has reached the user, but the notification service returns an error. The system reruns the step, gets a different model reply and contacts the user again. One message can then receive two replies.
This was a failure scenario identified in review. It did not reach users. The extra call would also add model cost, but we did not measure savings from the fix.
What we changed
We limited the retry to the model request. Saving the reply and sending a notification are outside that retry.
We tied the saved reply to the original message. The database rejects a second reply for that message. This rule also matters when the first attempt replies late, after the second has already started.
The notification uses the saved reply. If sending fails, only that delivery is retried. Sending it again does not require another model call.
What the pre-release check showed
We delayed the model response beyond 15 seconds and deliberately broke notification delivery. In that check, each message had one saved reply and one notification.
We kept one automatic retry. In the scenario described, a user could wait about 30 seconds for a reply. We do not yet have data on whether users prefer waiting for the retry or receiving an immediate failure message.
Injected failures establish behaviour under those conditions. They do not measure failure rates with real users or guarantee a single notification under every delivery failure.
An acceptance table for your AI feature
The rows below are proposed checks for your project. We are not claiming that all of them were performed in our example. Agree the expected behaviour before development and record the result of each check.
| Condition | Expected behaviour | What to record |
|---|---|---|
| The model replies after the deadline | A late reply does not create a second saved result for the same message | Call count, saved reply count and wait time |
| The provider returns an error | Retries stay within the agreed attempt limit; after failure the user sees a clear status | Attempts, final status and cost |
| The reply is saved but the notification was not sent | Delivery retry uses the saved reply without another model call | Saved result, model calls and delivery attempts |
| The notification arrived but its acknowledgement was lost | Duplicate protection is checked and channel limitations are stated | What the recipient saw and how the system decided whether to resend |
| The system receives a message with the same identifier again | The same identifier does not produce a second saved reply | Message identifier and result count |
| The model replied but saving failed | The system does not report success before saving; recovery behaviour is agreed | Record state and whether another model call was needed |
| All permitted attempts are exhausted | The user can understand the failure and safely continue | UI message, retry option and total cost |
How to include the checks in the first release
Choose one complete scenario. Describe what the user sends, where the result is stored and how they learn about success or failure. For each step, specify which actions may be repeated.
Check failure handling separately from reply content. A good model reply does not establish that it was saved or that an action happened once. Acceptance needs both results.
For cost evaluation, count every attempt needed to complete the action. Agree wait limits, retry counts and acceptable cost for your own scenario. Our 15-second deadline is not a universal setting.