A reviewer fixes the agent’s draft before it goes out, the fix lives in one Slack thread, and next week the agent makes the same mistake for a different prospect. Nothing carried the correction back to the instructions the agent drafts from, so the reviewer has quietly become the playbook, one thread at a time.
Empact Partners, a consultancy that builds agents like this one into the RevOps work it runs for B2B software partners, closes that gap on its own outbound with one loop. Every reply in Instantly is classified first, a person reviews every draft in Slack, and once a day a calibration turns the corrections worth keeping into dated rules in a playbook file.
This page builds the same loop in your workspace with Claude Code, from one prompt, and nothing in it can send an email. On our run, five of its first eight drafts went to conversations no sales reply should touch, and eleven corrections from our own team became exactly one rule. Both results set the order of the steps.
emails:read, leads:read, accounts:read and campaigns:read.chat:write, channels:history (groups:history for a private channel) and, if you want names on the decisions, users:read.This builds the half of reply handling that drafts, records and learns. It never sends, it does not chase silence, and it does not touch your CRM.
Scope And Classify Every Reply Before Anything Gets Drafted
Step 1: Create a read-only key and test it
In Instantly, open Settings, then Integrations, then API Keys, and click Create API Key. Name it after the loop, tick the four scopes above, click Create, and copy the key into your password manager, because Instantly shows it once.
In a terminal window you will close afterwards, run export INSTANTLY_API_KEY= followed by the key. Then, from an empty folder, run:
curl -s "https://api.instantly.ai/api/v2/emails?email_type=received&limit=1" \
-H "Authorization: Bearer $INSTANTLY_API_KEY" -o first-reply.json && python3 -c "import json; print(sorted(json.load(open('first-reply.json'))['items'][0]))"The command saves your most recent reply to first-reply.json and prints only the field names, so nothing about the prospect lands on screen. You should see two dozen of them, thread_id, lead, eaccount and i_status among them. A 401 means the key is wrong, and a 403 means a scope is missing.
Look for two fields that are not there. Instantly’s documentation lists is_auto_reply and lead_id on every email, and the list we got back carried neither. The auto-reply flag only arrives when you ask for one email by its ID, which is why our own reply lane makes a second call for every reply.
The lead’s fields have to be found by the lead’s address instead. Both facts are written into the build prompt in step 3, because a loop built from the documentation alone would treat every out-of-office as a person.
Step 2: Write the ladder into playbook.md
A reply agent that drafts before it classifies treats an unsubscribe and a pricing question as the same job: something to answer. The ladder is the decision about each kind of reply, made before any draft exists, about which replies get a draft, which go to a person with no draft, and which get nothing.
Save this as playbook.md in the folder. Rewrite the shapes and house rules in your own words and keep the headings, because the scripts read the table and the calibration edits the sections by name:
# Reply playbook
The reply loop reads this file before it classifies or drafts anything. People edit it. The daily
calibration proposes changes, and a person merges them.
## The ladder
| Reply type | What it looks like | What the loop does |
|---|---|---|
| auto_ooo | Out of office, on leave, back on a date | Nothing. Note the return date. |
| auto_other | Any other automatic message: a bounce, a ticket, a calendar notice | Nothing. |
| unsubscribe_request | Asks to be removed | No draft. A person adds them to the block list. |
| rejection_hostile | Angry, calls it spam, mentions a lawyer | No draft. A person decides whether to answer at all. |
| rejection_polite | No thanks | A two-line thank-you that leaves the door open. |
| not_now | Interested later, the timing is wrong | A short reply. If they gave a date, it names the date. |
| interested_meeting | Wants a call, or sent a booking link | A reply that books the call. |
| interested_info | Wants details, examples or pricing | A reply that answers exactly what they asked. |
| referral | Points to a colleague | A reply to the colleague, thanking the person who referred. |
| wrong_person | Not them, and no name given | A reply asking who owns this. |
| existing_relationship | Already knows us or works with us | No draft. Whoever owns that relationship answers. |
| other | None of the above | A draft, and a question for the reviewer. |
Every draft goes to the review channel. A person sends it, from Instantly, or nobody does.
## What stays with a person for good
Even if the loop is one day trusted to send some reply types on its own, these never leave the
review channel:
- a complaint, a threat or a legal question;
- a price that is not written in this file, a discount, or a change to what we offer;
- anything someone has to sign;
- an unsubscribe request.
## How every reply is written
- The first sentence answers what they asked.
- Under 120 words, unless they asked for detail.
- One next step, never two.
- Signed by the person whose inbox the email went out from.
- Never a fact about their company that the thread or the lead's fields do not hold. Ask instead.
## Voices
One short paragraph per person who sends, written from their own edits.
## Rules
Dated rules the calibration added, each naming the decisions it came from.The twelve reply types are the ones our own lane sorts every reply into. When our team validated them in August, a person agreed with the classifier on 89% of the replies in the test set.
Two cheaper layers sit in front of the model: Instantly’s own auto-reply flag, and fixed text rules for out-of-office notices, bounces, ticket systems and calendar robots. Nothing reaches a model that a regular expression can sort.
The list under “What stays with a person for good” comes from our own review ladder. Today nothing our lane suggests goes out without a teammate’s go in the thread, and the types that would never send alone at any level of trust are fixed: a complaint, anything legal, an unlisted price, a concession, a change of scope, anything to sign, an unsubscribe.
Step 3: Build the loop with one prompt
The build refers to prompts/calibrate.md, the calibration prompt in step 8. Save that file first, so the calibration script has something to send. Then type claude in the folder and paste this:
Build an outbound reply loop in this folder, in Python with the standard library only. It drafts replies to our cold email in Instantly and never sends one. Read playbook.md first: it holds the reply types, what the loop does with each, and the rules.
1. pull.py
- GET https://api.instantly.ai/api/v2/emails with email_type=received, sort_order=asc, limit=50 and min_timestamp_created set to the watermark in state.json (seven days back on the first run). Page with next_starting_after.
- Send Authorization: Bearer $INSTANTLY_API_KEY and a browser User-Agent header. The API sits behind Cloudflare, which refuses Python's default one.
- Wait 3.5 seconds between calls: the email list allows 20 requests a minute.
- The list leaves out is_auto_reply, so GET /api/v2/emails/<id> for each new reply and keep that flag.
- Save each reply to inbox/<id>.json unless it is already there, then move the watermark.
2. draft.py
- Skip replies with is_auto_reply true.
- Classify the rest in one batch with claude -p --model haiku --effort low --tools "" : one reply type from the playbook's table per reply, as JSON.
- Where the ladder says the loop writes a reply, draft it with claude -p --model sonnet --effort medium --tools "" from the playbook, the whole thread (GET /api/v2/emails?search=thread:<thread_id>) and the lead's fields (POST /api/v2/leads/list with {"contacts": ["<lead email>"]}). If the draft needs a fact none of those hold, it asks the reviewer instead of guessing.
- Write drafts/<id>.json: the reply type, what the ladder says, the draft, the question for the reviewer.
3. review.py
- post: send each new draft to the Slack channel $SLACK_CHANNEL_ID with chat.postMessage and $SLACK_BOT_TOKEN: who wrote, their words, the reply type, the suggested reply, the question. Keep the message ts in the draft file. With --dry-run, write the payload to outbox/ instead.
- collect: read each posted draft's thread with conversations.replies. A reply of yes, send or ok is approved. A reply starting with edit: is edited, and the text after it is what the person sent. hold is held. Anything else is a note. Append one line per decision to decisions.jsonl: id, reply type, kind, reviewer, the draft, the final text, the note, the time.
4. calibrate.py
- Send prompts/calibrate.md, playbook.md and the decisions since the last calibration to claude -p --model opus --effort high --tools "".
- Apply an edit only if: its old text appears exactly once in playbook.md; the new text adds no email address, phone number, currency amount, or name or company found in inbox/ or decisions.jsonl; no heading disappears; the file does not shrink by more than 30%.
- Commit the edits that pass to a new git branch calibration/<date>, the reasons in the commit message. Print each refused edit with why. Save the diagnoses to calibrations/<date>.json.
5. Nothing in this folder calls an Instantly endpoint that sends, replies, forwards or changes anything. GET requests, plus the one POST that lists leads.
6. A README.md: how to run each script, the environment variables, and a crontab line that runs pull, draft and post every 15 minutes and calibrate once a day.
Test each script on the real API as you build it, but run review.py only with --dry-run and do not run calibrate.py.On our run Claude Code worked for twenty minutes and came back with five scripts and a README. pull.py saves new replies to inbox/, draft.py classifies and drafts into drafts/, review.py posts to Slack and reads the threads back, and calibrate.py proposes and checks playbook edits.
Open .gitignore before anything else. Every folder that holds a prospect’s email belongs in it, and on our run every one was.
The one stall was the bill. A plain claude -p call carries your whole Claude Code setup, its connected servers, settings and instructions, and on our machine a one-line classification cost about 570,000 tokens. Claude Code added four flags to every call:
--strict-mcp-config --disable-slash-commands --no-session-persistence --setting-sources ""With them the same call came down to about 3,500 tokens. If your build left them out, add them before the cron line goes in. Claude Code also put review.py collect in the cron line, which our prompt forgot, and without it nothing writes the decisions the calibration reads.
Building agents of this shape for partners is what we call marketing engineering: one agent, one job, and a consultant who owns what it writes. The build is the short part. The steps after it are where the agent earns its keep.
Step 4: Pull a week, then cut it to your sales campaigns
Run python3 pull.py. The first run reaches seven days back, waits 3.5 seconds between calls and fetches every reply twice for its auto-reply flag, so a busy week takes a quarter of an hour. You should end with one file per reply in inbox/, and about a quarter of ours carried the flag.
Check what the pull caught before you draft anything. Five of our first eight drafts went to conversations that were never a prospect answering our sales email, such as publishers’ threads about links. A workspace holds every campaign you run, and more than four in five of the replies a person typed that week came to campaigns this loop should never answer.
Run this to list your campaigns, one ID and name a line:
curl -s "https://api.instantly.ai/api/v2/campaigns?limit=100" \
-H "Authorization: Bearer $INSTANTLY_API_KEY" | python3 -c "import json,sys; [print(c['id'], c['name']) for c in json.load(sys.stdin)['items']]"The list returns a hundred at most, and with more you add &starting_after= and the last ID it printed to get the next page. Copy the IDs of the campaigns this loop answers into sales_campaigns.txt in the folder, one a line, then paste this into Claude Code:
Our workspace holds more campaigns than the ones this loop answers. Make draft.py skip every reply whose campaign_id is not listed in sales_campaigns.txt (one campaign ID per line; lines starting with # are comments), and say in the README how to fill that file. Change nothing else, and do not run anything that calls Slack or the calibration.Run python3 draft.py --limit 1 to check it took. Ours printed that it skipped 177 replies from other campaigns and 15 automatic ones before it classified the one reply it was allowed.
Our own lane draws the same line with two conditions, and a reply meets both before anything is suggested: it reached one of our own inboxes, and the company is an open deal in our CRM. Everything else in the workspace belongs to another process, our partners’ programs included. A campaign list is the version of that test a team can set up in an afternoon.
Read What The Drafter Asked Before You Read What It Wrote
Step 5: Draft the sales replies, and open the questions first
Run python3 draft.py. Every reply from a sales campaign gets a file in drafts/: its reply type, what the ladder says, the draft where the ladder allows one, and a question for the reviewer where the drafter needed something it did not have.
The twelve most recent sales replies of our week came out like this:
| Reply type | Replies | Drafts written | Questions for the reviewer |
|---|---|---|---|
| interested_meeting | 3 | 2 | 3 |
| other | 4 | 1 | 4 |
| existing_relationship | 2 | 0 | 0 |
| rejection_polite | 1 | 1 | 1 |
| not_now | 1 | 1 | 1 |
| auto_other | 1 | 0 | 0 |
Five drafts and nine questions, and the questions were the useful half. Open them before the drafts:
The first is why the build prompt sends the drafter the whole thread instead of the reply alone. A loop that drafted from the reply would have queued four second emails to people we had already answered, the failure that sent one prospect two emails from our own lane in October.
Our lane now reads the thread before it drafts and again before anything is sent. The fourth question is a gap in the playbook, and the reviewer’s answer to it is the first decision worth keeping, because the next prospect who asks will need the same fact.
Step 6: Post to the review channel and collect the decisions
Set SLACK_BOT_TOKEN and SLACK_CHANNEL_ID, run python3 review.py post --dry-run, and open one file in outbox/. It holds what the reviewer will see, in the order they need it: who wrote, their words, the reply type, the suggested reply, the question. Then run python3 review.py post.
Create the Slack app for your own workspace only. Slack holds apps distributed to other workspaces, outside its Marketplace, to one thread read a minute, and collect then stops and picks up on its next run.
The reviewer answers in each draft’s thread, and python3 review.py collect writes each answer to decisions.jsonl:
yes, send or ok, and the draft is recorded as approved as written.edit: followed by the text, which is recorded as the version the reviewer sent.hold, and the draft waits.Nothing in the folder sends. The reviewer sends the reply from Instantly, the only place in this setup an email can leave from.
Two things about this step took our team longest to get right. Our lane reads approvals with fixed rules in plain code: a short, unconditional yes sends, an edit has to match the text exactly, and anything conditional becomes a note the next draft is rewritten with.
And before anything goes out, our send step reads the thread again and checks that the yes sits under the exact version of the email it approves. Since 23 October our lane has sent nothing on its own.
Most Edits Are Not Rules
Everything so far produces a log. What turns the log into drafts that improve is deciding which entries deserve to change the playbook, and in our own outbound the answer is very few. Our daily calibration has run six times since 24 October and changed a document three times.
.png)
A loop that turns every edit into a rule ends up learning the reviewer instead of the job. One person’s preference on one afternoon becomes an instruction for every prospect, the playbook grows a line a week, and three months later nobody can say which line made the drafts worse.
Step 7: Diagnose each decision before it touches the playbook
Every decision that is not a plain approval is one of six kinds, and only two of them belong in the playbook. Read decisions.jsonl with this table beside it:
| Kind | What it looks like | Where the fix goes | An example from our outbound |
|---|---|---|---|
| Wrong type | The reply went down the wrong path of the ladder | The ladder’s descriptions | An automated follow-up filed as a request for a meeting |
| Missing context | The draft lacked a fact that the thread, the lead or a call held | The data the draft reads | A booking reply that needed a different consultant’s calendar link |
| Bad data | A field in the sending tool was wrong | The tool, never the playbook | Our own follow-up recorded as the prospect’s reply |
| Rule | The draft did what the playbook allowed, and the reviewer wants something else every time | The section of the playbook that owns it | A package named in a reply without a sentence saying what it is |
| Voice | How one sender writes | Their paragraph under Voices | A senior consultant’s own shape for a proposal email |
| One-off | A fact about one company, or a judgment about one conversation | Nowhere | Holding a second email until one prospect answers |
Bad data looks the most like a writing correction, because the reviewer fixes it in the draft. If the same field keeps coming back wrong, the prompt is the wrong place to fix it.
What Reddit says
“Log every draft you edit before sending. That diff is where you actually see what it invents, and in sales mail it’s the commitments, dates and prices, not the wording. Tighten the transcript tool on just those and you stop rereading whole drafts. Trust should be per field, not per draft.”
u/seekworld, r/SaaS, Sep 2026
“if the same field keeps coming back wrong after a few rounds, that is usually the tool handing back two near identical values, not the model drifting. fix it in the tool, not the prompt.”
u/seekworld, r/SaaS, Sep 2026
Step 8: Run the calibration
Save this as prompts/calibrate.md, if you did not in step 3:
You maintain playbook.md, the file a reply agent reads before it drafts answers to sales prospects. A person reviews every draft and sends it or not. Below are the decisions people made on the agent's drafts since the last calibration: approved as written, edited before sending (both versions), held, or a note.
First, diagnose every decision that is not a plain approval. Give it one of these kinds:
- wrong type: the reply was put in the wrong reply type, so the ladder sent it down the wrong path;
- missing context: the draft lacked a fact the thread or the lead held, and it never reached the draft;
- bad data: a field in the sending tool was wrong or ambiguous, so the fix belongs in the data, not the playbook;
- rule: the draft did what the playbook allowed and the person wanted something else, and would want it every time;
- voice: how one person writes, which belongs under their name in Voices;
- one-off: a fact about one company or a judgment about one conversation.
Then propose edits, only for rule and voice. One correction on its own, with no reason given, is not a rule. The same correction twice is, and so is an instruction that says it applies from now on. Write each rule in your own words, as the smallest change to the section that owns it, dated, naming the decisions it came from. Never put a price, a person's name other than a Voices heading, a company's name or a quote longer than six words into the playbook.
Answer with JSON only:
{"diagnoses": [{"decision": "<id>", "kind": "...", "why": "one sentence"}],
"edits": [{"old": "exact text in playbook.md, at least one whole line", "new": "the replacement", "decisions": ["<id>"], "why": "one sentence"}]}Then run python3 calibrate.py. It prints each refused edit with the reason, saves the diagnoses to calibrations/, and commits whatever passed to a branch named for the day.
We ran it on the fifteen decisions our team made on our own lane’s suggestions between 26 October and 4 November, in the loop’s decision format. It took one call and thirty-nine seconds.
Of the eleven that were not plain approvals, it called six one-offs, two missing context, two voice and one a rule. It proposed two edits and committed one, under “How every reply is written”:
- Never name one of our offers or packages without one plain sentence on what it is and what they get. (2026-11-04, from fb_20261102T141200_2af861 and the earlier comment it repeats)That rule came from a senior consultant correcting the same omission a second time, in her words “you forgot to apply the previous comment I made”. Our own lane’s calibration wrote the same rule into our playbook on 3 November, from the same correction.
The calibration also left alone what ours left alone: a remark that one sender’s drafts all carried the same emoticon, which was one passing remark and not yet a voice, and a held draft whose only reason was which of us takes the call.
.png)
The edit it refused was the one we wanted most. The senior consultant had rewritten a proposal email and asked for the rewrite to be learned, and the calibration wrote a good paragraph for her under Voices. The name check refused it, because her surname is in the inbox, on the emails she sends herself.
A false refusal, and the safe direction for a check to fail in. The fix is a short list of the people who send, which the check allows as Voices headings. Our own lane keeps each teammate’s voice on a page of its own, so its check never mistakes a colleague for a stranger.
Step 9: Merge the branch, or delete it
Read what changed, with your own date in place of ours:
git diff main calibration/2026-11-04 -- playbook.mdgit merge calibration/2026-11-04, if you would defend each rule to the reviewer who caused it.git branch -D calibration/2026-11-04, and the playbook stays as it was.Keep the merge in your own hands for the first month, and add the calibration to cron once a day, after the afternoon’s last review.
Our lane merges its own calibration once the checks pass, because every change lands in a dated log, a line goes to Slack every Friday, a failed check stops the merge and messages me, and any day can be undone with one revert.
A team can stop reading every branch at the point it can trust the loop with more, and the runs somebody had to rescue tell you when that is. On a partnership, getting there is the part our consultants own:
Every Rule Needs A Date, Because Rules Expire
On 27 October our calibration wrote a rule from my own instruction: answer a price question at once and let the report follow. Three days later I changed how we answer price questions, and the rule was replaced by hand, with a line naming what it replaced. The calibration could not have known: the decision was made outside the log it reads.
So before you change how your team answers a kind of reply, search the playbook’s Rules for that reply type and retire what your change replaces, in the same edit. The date and the decision IDs on every rule are what make that search possible.
If your reviewers are fixing the same drafts week after week and the agent is not getting better, book a call with me, and let’s work out whether it is something we should build with you.
