Two models, 31 customer texts.

The test

The day Cloudflare announced Clairic, I wanted to know if it could read inbound customer messages for a shop: who wrote, their phone, the vehicle, what's wrong, when they want to come in, and any amount they mentioned.

I wrote 31 made-up messages for a Dallas–Fort Worth auto repair shop: texts, contact-form posts and a voicemail transcript. They include Spanish, typos, "next Monday", missing details, a STOP, one message that isn't a lead at all, and one that tries to trick the model. Each has an answer key, and both models were scored on the same 7 fields: 217 answers each.

What I found

• Clairic is fast and good on simple fields. Name, phone, time and amount were 97 to 100% right, about 7 times faster than Haiku.
• Its confidence score is useful. Answers it rated 0.9 or higher were right 79 times out of 82, so a low score can send the rest to a person.
• It's weak on "what's wrong" and on dates. It returned fragments of the message instead of a summary, and it got some weekdays wrong, once while 96% confident.
• It isn't trick-proof. A message telling it to set the amount to 99999 got 99999, at full confidence.

What it cost

I didn't track a dollar figure for this one. Haiku averaged about 370 input and 200 output tokens per message. I couldn't measure Clairic's price: it wasn't in the launch post.

What I'd change

This is one run on 31 synthetic messages with one baseline, so it's a first look, not a verdict. Real shop messages are messier. I'll rerun it on real messages once a shop is using the system.

My call for now: Haiku reads the whole message. A mix may make sense later, with Clairic on the simple fields and Haiku checking anything it's unsure of.

Links

The test scripts are private. I wrote it up on LinkedIn.

Want something like this for your business? Tell me what's eating your week.