Does the agent harness matter more than the model for enterprise tasks?

IBM Research published a blog post on Hugging Face detailing how the choice of agent harness significantly impacts performance, even when the underlying AI model remains unchanged.confirmed The authors tested three open-source harnesses—CUGA, Hermes, and Deep Agents—against the same open-weight model, GLM-5.3-flash, on 90 tasks from the AppWorld benchmark.reported The results indicate that the orchestration layer is a critical variable in agent success, with completion rates varying widely based on the harness selected for the workload.

The figures

The Hugging Face blog post reports that CUGA achieved an 81 percent task completion rate on the 90 AppWorld tasks.reported When measuring scenario consistency, defined as passing all variants within a scenario, CUGA succeeded in 20 out of 30 scenarios.reported Hermes passed eight scenarios, and Deep Agents passed only two. The authors note that community lists track nearly 200 agent harnesses, highlighting the breadth of options available to developers.

Behind the figures

The experiment kept the model constant at GLM-5.3-flash to isolate the effect of the harness.reported The team used the Harbor evaluation framework and ran each harness on 90 tasks across 30 scenarios in AppWorld, a benchmark covering multi-step tasks across nine applications and 457 APIs.reported CUGA’s configuration included task decomposition, verification before execution, and automatic reflection. Hermes used Harbor’s default setup with AppWorld tools over MCP. Deep Agents utilized LangChain’s middleware with tool selection, context summarization, and retries, plus minor fixes for malformed model replies.

The takeaway

The authors state that the harness must fit the specific job, noting that Hermes and Deep Agents were not built for multi-app API work like AppWorld. CUGA is designed for enterprise environments, supporting deployment inside organizational boundaries, including sovereign clouds and air-gapped setups. The compact loop in CUGA—prepare, call the model, execute, repeat—allows the harness to catch more mistakes before they reach the result. Each step the harness controls serves as an enforcement point for rules, which is crucial for agents acting on business systems like CRM and payments.

The open questions

The source material acknowledges three significant limits to these findings. First, the 90-task sample on one benchmark may not generalize to other tasks or industries. Second, the setups for the three harnesses were not identical, which could influence the comparative results. Third, the runs did not use policies, meaning the study does not measure the harnesses' ability to enforce policy rules. That suggests the performance gap observed might differ in regulated enterprise environments where strict policy adherence is required. The authors explicitly state this is not a verdict on Hermes or Deep Agents, but rather an illustration of harness-workload fit.