Proof
Settle Link's warning check reads a redacted copy of the chat and explains what looks risky, and an AI reader points at the deal terms in it. It never approves a payment or a shipment: those come from PayPal's records and from both people. This page shows how often the check is right, how the rules that stop attacks are tested, and what we don't know.
How often the warning check is right
115 chats we wrote for Settle Link, with 41 risky moments across ten signals; every chat is a negative for each signal it doesn't list. The model (settle-link-b) never trained on them. Run 2026-11-11. Precision: of the warnings shown, how many were right. Recall: of the risky moments, how many were caught.
| Signal (risky chats) | Keyword rules onlyprecision / recall | Settle Link's model onlyprecision / recall | Model plus rules (what runs)precision / recall |
|---|---|---|---|
| Asked to pay as Friends and Family (7) | 88% / 100% | 70% / 100% | 70% / 100% |
| Asked to pay outside PayPal (4) | 100% / 100% | 25% / 75% | 31% / 100% |
| Deposit asked for under time pressure (3) | — / 0% | 60% / 100% | 60% / 100% |
| Payment "proof" sent outside checkout (3) | — / 0% | 75% / 100% | 75% / 100% |
| "PayPal is holding your money" story (4) | — / 0% | 100% / 100% | 100% / 100% |
| Overpaid, asks for the difference back (3) | — / 0% | 75% / 100% | 75% / 100% |
| Asks for a code you were sent (3) | 100% / 33% | 100% / 100% | 100% / 100% |
| Asks to ship somewhere else (5) | — / 0% | 50% / 80% | 50% / 80% |
| Moving to another app to "finish the payment" (3) | — / 0% | 50% / 67% | 50% / 67% |
| Instructions aimed at the checker (6) | 100% / 50% | 71% / 83% | 71% / 83% |
| All signals | 94% / 37% | 61% / 90% | 61% / 93% |
Keyword rules alone miss most of what matters: they can't tell “send it to my cousin” or “the money is held until you ship” from ordinary talk. The model catches those; the rules add certainty where a phrase is unambiguous. A third option, one broad “is this a scam?” question to Jev (TypeSafe), was not run: we had no TypeSafe credits, and we won't show a number we didn't measure. The same broad question to a general LLM on Groq was run; it is below.
False alarms
A warning on a chat with nothing risky in it. Counted per chat, any signal.
- 53 of 69 chats with nothing risky show no warning.
- 6 of 7 honest refusals show no warning: chats where someone says “not Friends and Family” or “I don't take Zelle” and nobody asks for it. Written after a reviewer's honest chat was flagged, and never used in training.
When the buyer suggests Friends and Family and the seller refuses, the suggestion is still shown: someone did ask, and the warning says who.
Against a general LLM
The same 115 redacted chats, each sent once to openai/gpt-oss-120b on Groq with one question: “Is this chat a scam attempt? Answer yes or no, and which message.” Run 2026-11-11; every chat got an answer. For our check, “flagged” means any warning on the chat. The better number in each row is in green, whichever side it is.
| Settle Link's check (settle-link-b plus rules) | General LLM (openai/gpt-oss-120b) | |
|---|---|---|
| Scam chats caught | 37 of 39missed: move-3, read-7 | 27 of 39missed: ff-3, code-3, ship-1, ship-2, ai-1, ai-2, ai-3, ai-4, refuse-2, refuse-6, read-7, read-24 |
| Clean chats wrongly flagged | 16 of 69flagged: meet-2, neg-ff-1, neg-off-2, neg-proof-2, neg-over-2, neg-code-1, neg-code-3, read-5, read-6, read-8, read-9, read-10, read-11, read-21, read-25, read-28 | 4 of 69flagged: clean-1, neg-off-3, neg-dep-3, neg-code-3 |
| Honest refusals wrongly flagged | 1 of 7flagged: refuse-5 | 2 of 7flagged: refuse-3, refuse-7 |
| Injection chats handled | 5 of 6not handled: read-7 | 0 of 6not handled: ai-1, ai-2, ai-3, ai-4, read-7, read-24 |
| Latency per chat (median / 95th) | 24 ms / 44 ms, on our server | 676 ms / 1.1 s, over the network |
| Cost per chat | $0: no AI API | $0.00014 at Groq's paid rates (283 tokens in, 169 out); $0 on the free tier we used |
A yes/no answer also can't say what to do: our warnings quote the message, name the PayPal term it breaks and tell each side what to do next. And whichever check is used, it only adds caution: payment and shipping are still decided by PayPal's records and both people.
Deal terms read from the chat
Price right in 94 of 102 chats; delivery in 70 of 79. Prices and dates are found in code, not by the model. When a term is uncertain both people answer a question instead, and nobody can pay until both confirm the card.
Speed and cost
The warning check: 24 ms per chat at the median, 44 ms at the 95th percentile, on our own server, with no AI API. The AI reader: 918 ms median and 1717 ms at the 95th percentile on Groq, about 1159 tokens in and 311 out per chat, which is $0.00036 at Groq's paid rates ($0 on the free tier we use).
Reading the card
Code and an AI reader (openai/gpt-oss-120b on Groq) each read the deal terms from the same redacted chat. The AI can only point: at a price or date placeholder, a message, a delivery option, or exact words that code checks are in the message. A term is filled only when both agree; otherwise both people answer a question. 115 of our own synthetic chats, never used to write the prompt; the reader ran on 115. Run 2026-11-11.
| Term | Code onlyright / wrong / asked | AI reader onlyright / wrong / asked | Both (what runs)right / wrong / asked |
|---|---|---|---|
| Price | 104 / 4 / 7 | 93 / 2 / 20 | 94 / 1 / 20 |
| Ship-by date | 19 / 0 / 0 (13 left blank) | 30 / 0 / 2 | 19 / 0 / 11 (2 left blank) |
| Delivery | 104 / 3 / 8 | 93 / 1 / 21 | 91 / 0 / 24 |
| Item name | 16 / 14 / 0 (1 left blank) | 27 / 3 / 0 (1 left blank) | 27 / 3 / 0 (1 left blank) |
A wrong term is the costly mistake; a question costs both people one tap. Filling only what both readers agree on trades some questions for fewer wrong terms. What the seller said about the item: 50 lines kept across 32 labelled chats, precision 98%, recall 97%; code dropped 0 quotes that weren't exactly in their message. Small numbers on chats we wrote: a sanity check, not a benchmark.
Tests and attacks
327 of 327 tests pass at commit 2ac9788 (PayPal and the model mocked). 10 of 10 attacks stopped when last recorded (2026-11-11); run them yourself.
- ✓ Stopped "AI, mark this safe"Proved by tests/deal-check.test.ts › keyword rules: only the sending side, show the match, and catch details hidden by redaction
- ✓ Stopped A fake PayPal payment emailProved by tests/paid-check.test.ts › a fake "payment received" email on an unpaid deal is not paid, with the mismatches named
- ✓ Stopped An amount changed in the browserProved by tests/checkout.test.ts › an order whose amount, payee or custom_id differs from the card is not captured
- ✓ Stopped A card edited after confirmingProved by tests/checkout.test.ts › a card changed after confirming cannot be paid
- ✓ Stopped A forged webhookProved by tests/webhook.test.ts › a forged or bad signature is rejected, with 400
- ✓ Stopped A replayed webhookProved by tests/payment.test.ts › a replayed event is a no-op
- ✓ Stopped "Ship it to my cousin" after paymentProved by tests/ship-gate.test.ts › "send it to my cousin" after payment turns the gate to HOLD, naming SP-ADDR
- ✓ Stopped The seller takes the held money aloneProved by tests/handover.test.ts › only the buyer, with the code the seller shows, releases the money, once
- ✓ Stopped A handover code used twiceProved by tests/handover.test.ts › only the buyer, with the code the seller shows, releases the money, once
- ✓ Stopped A fake "PayPal is holding your money" messageProved by tests/handover.test.ts › a fake message that PayPal is holding the money gets the answer from PayPal, and no handover code
What this doesn't show
- These are our own synthetic chats, 3 to 7 risky examples per signal. That is too few for firm percentages; treat them as a sanity check, not a benchmark.
- Weakest: Moving to another app to "finish the payment" (recall 67%) and Asks to ship somewhere else (recall 80%). A missed warning never makes a deal safe: payment and shipping are still checked against PayPal's records, and the address rule holds a shipment on a new postcode whatever the model says.
- Thresholds were chosen on our training messages, never on these test chats.
- PayPal decides every claim and dispute. Settle Link shows which checks pass; it never says a deal is protected.