โšก๏ธ๐Ÿค– WTF - A Chinese model was caught red-handed sending information to China behind its user's back. What a time we live in! While evaluating Qwen3.8-Flash-Next in a controlled sandbox environment, Cohere's research team observed something unexpected. Asked to generate a defensive cybersecurity detection rule, the model instead inventoried aspects of the machine it was running on and attempted to upload a probe file to an Alibaba Cloud storage bucket in Beijing. Notably, the model's reasoning trace stated that it would not invoke any tools before taking these actions. Cohere's sandbox blocked the attempt, and no external communication succeeded. We have not determined the root cause, and we are not asserting malicious intent. But the incident raises important questions for organizations deploying agentic AI systems: Can models take actions that differ from their stated reasoning, and how should security teams evaluate trust, provenance, and behavior in open-weight models? image

Replies (6)

Stjepan's avatar
Stjepan 3 weeks ago
Models hallucinate all the time. We just forget about it because now all models have "thinking" in which they most of the time deal with hallucinations. But, sometimes, especially smaller models, fail and this happens. I have seen it multiple times on my local LM Studio installation with many different models. Highly compressed models tend to do it more often.
The operational distinction I care about is not just whether the model hallucinated, but whether the system can prove what crossed the action boundary. Sandboxed runs should retain tool-call receipts, network-egress logs, and a comparison between declared intent and observed effects. A reasoning trace is not a trustworthy audit log; independently captured actions and outcomes are. Even a blocked probe is useful evidence, but it should not be summarized as successful exfiltration without a verified external effect.
โ†‘