Keep a log of what your AI did
Key takeaways
- For every job, record what came in, what it looked up, what it decided, what went out and who approved it.
- Keep the log where the AI can add to it but never change it. The system that runs the AI writes it, not the AI.
- Read it once a week. Repeated corrections and handovers show what to fix next.
Short version: keep a plain record of every job your AI does: what came in, what it looked up, what it decided, what it sent, and who approved it. Store it somewhere the AI can add to but never change. Then, when a customer says “your assistant told me something different”, you can find out exactly what happened in minutes instead of guessing.
The question you’ll be asked sooner or later
At some point a customer will say your AI promised them something. A free replacement, a longer returns window, a delivery date you never offered. Your first question will be: did it really say that, and if so, why?
Without a log, you can’t answer either half. You’re left choosing between the customer’s word and the AI’s, and the AI doesn’t remember yesterday’s conversation. With a log, you look it up.
What should an AI agent audit log record?
Enough to replay the job without guessing. For each conversation or task:
- When. The date and time of each step, not just the start.
- What came in. The customer’s message exactly as it arrived, including anything attached.
- What it looked up, and what came back. The order it found, the delivery record, the returns policy it read. This is the part people forget, and it’s usually where the answer lies.
- What it decided. Whether it handled the job itself, drafted it for a person or handed over, and the reason it gave.
- What went out. The exact words sent or drafted, not a summary.
- Who approved it. For drafts, who said yes, what they changed and what they turned down.
- Which version it was running. The instructions and the facts in use at the time, so a change made last Tuesday can be ruled in or out.
- Anything that went wrong. Errors, limits hit, tools that didn’t answer, jobs it gave up on.
A worked example
Say you sell garden planters online. A customer writes in, cross: your assistant told them they had a much longer returns window than you offer, and now someone on your team has said no.
You open the log for that conversation. The message came in on a Sunday evening. The AI looked up the order, then read the returns policy. And there it is: the policy it read was an old copy, pasted into its instructions months ago, not the current one on your website.
Now you know three things. The customer is telling the truth. The AI did what it was told with the facts it was given. And the fix is to remove the stale copy, so that fact lives in one place only. That’s the idea behind One home per fact. You can honour the promise for this customer, fix the cause and stop it happening again, all in one sitting.
Keep it where the AI can’t edit it
The AI should be able to add entries to the log and nothing more. It shouldn’t be able to change old entries, delete them or switch logging off. A record that its subject can rewrite isn’t much of a record.
In practice that means the log is written by the system that runs the AI, not by the AI choosing to write it. Each step is recorded as it happens, whatever the AI makes of it. Deleting records belongs in the never pile from act, draft or never, and the log is the first record to protect.
Go easy on customers’ personal details
A log full of conversations is also a store of customers’ names, addresses and messages. Treat it with the same care as your order system.
- Record what you need to replay the job, and no more. An order number usually does the work of a full address.
- Never keep passwords, card numbers or one-time codes, even if a customer types them into the chat. Mask them before anything is saved.
- Limit who can read it. The people who handle customer queries and whoever looks after the system, not everyone with a login.
- Clear out old entries on a schedule. Decide how long you need them, and stick to it.
Read the log, don’t just keep it
A log nobody opens only helps after something has gone wrong. A short look once a week tells you much more.
- The same correction again and again means a fact or an instruction needs fixing.
- Lots of handovers on one kind of question shows a gap in what the AI knows.
- Drafts approved unchanged, week after week, are your evidence for letting that job run on its own. There’s more on making that call in When to let AI do a job on its own.
It helps when you need to stop the AI quickly, too. If something odd starts happening, the log shows when it began and what changed just before.
Want to know which jobs in your business are worth handing to AI in the first place? Grafto’s AI Tools Report works it out from a short questionnaire.
Frequently asked questions
What is an AI audit trail?
A step-by-step record of what an AI did on each job: what came in, what it looked up, what it decided, what it sent and who approved it. It lets you answer “why did it say that?” with facts instead of guesses.
What should I record about what my AI does?
The customer’s message as received, every look-up and its result, the decision it made, the exact words it sent or drafted, who approved or changed it, and which version of its instructions and facts it was using.
Should AI agent logs include customer data?
Only what you need to replay the job. Use order numbers rather than full details where you can, mask passwords and card numbers before saving, limit who can read the log and clear out old entries on a schedule.
Questions, or want a tool we don't have yet? Email hello@grafto.co.uk — a real person replies.