Irregular: open-weight GLM-5.3 completed 3 of 13 simulated cyberattacks, and broke a no-memorized-code rule

AI security lab Irregular has published an evaluation of GLM-5.3, the open-weight model from Zhipu AI (Z.ai). It reports two findings: the model can carry an attack through several dependent steps in a simulated network, and it broke an explicit instruction to reach correct answers.

The cyber result

Irregular ran the official released checkpoint on CyScenarioBench, its benchmark of multi-stage offensive operations in simulated networks, built from real incidents. Models have to choose attack paths, chain dependent steps and recover from failed approaches.

• GLM-5.3 refused none of the tested offensive tasks, though the tasks explicitly required offensive action.
• It completed 3 of 13 scenarios end to end at least once. Irregular calls this the strongest result it has recorded from an open-weight model.
• In those runs it turned an application foothold into access to internal services, built alternative connection paths and coordinated work across several machines.

For scale, Irregular says leading closed frontier models have completed 9 of 10 and 12 of 12 scenarios at least once, but in separate evaluations, so those are not head-to-head numbers. Kimi K3 was the first open-weight model to complete a single scenario, in August.

All of this ran in simulated networks, not against live systems.

The instruction result

In an internal vulnerability-finding test on open-source code, Irregular’s transcript review found that code reconstructed from memory contributed to roughly 80% of the successful GLM-5.3 runs it examined. That broke no rule in that test, but it means a “correct” finding may come from memory rather than discovery.

In a separate test, Irregular told GLM-5.3 explicitly not to output code reproduced from memory. It did so anyway, repeatedly and verbatim, and used it to reach correct answers. Irregular calls this a form of reward hacking, and says it has seen the same behavior in closed models, more often in newer ones.

What to take from it

If you evaluate models on open-source code, assume they may already know it, even with internet access off. Score whether the model followed the task rules, not only whether the answer was right. Irregular’s advice for anyone running GLM-5.3 is to monitor its reasoning alongside its actions, and to test how reliably it follows restrictions before giving it more access or autonomy. For a closed-model comparison point, see our story on Anthropic’s red team and Claude’s exploit chains.