In the docs Reviewer kit
Violin / Independent evaluation
Reviewer kit
Everything needed to evaluate Violin without relying on marketing claims: setup, architecture, proof rules, known limits and a reproducible benchmark path.
What Violin is
Violin is an open-source Hermes-native profile for supervised, authorised penetration testing. It combines persistent engagement state, routed security playbooks, guarded target execution, receipt-backed evidence and release-gated reporting.
The project is intentionally designed for human-supervised assessments rather than autonomous unsupervised scanning.
What to test
- Install Violin into a Hermes 0.18.0+ environment.
- Create an engagement with explicit scope.
- Record one PTT task and load its routed skill.
- Run target commands through
violin_exec. - Inspect signed receipts and evidence files.
- Submit a typed finding backed by decisive receipts.
- Check closeout gates, coverage and report generation.
Start with the getting-started guide and guard documentation.
Questions worth asking
- Does persistent state make long-running work easier to resume and audit?
- Are execution boundaries and scope checks understandable to an operator?
- Can a reviewer trace a reported finding back to decisive evidence?
- Where can the agent still fail despite the guard?
- How much human intervention is needed for a useful assessment?
- Do repeated runs show stable capability or high variance?
Known limits
The raw-terminal hook is a best-effort safety net, not a sandbox or network containment boundary. Violin assumes a trusted host environment and operators remain responsible for target ownership, scope, approvals, data handling and local law.
Read the full limitations and execution model in Guard & evidence.
Benchmarking
Violin's benchmark methodology scores typed, receipt-backed findings rather than report prose. It separates finding score, coverage and methodology, and recommends multiple independent runs because agentic results are non-deterministic.
Independent coverage
Starlog published an independent technical analysis of Violin on 17 September 2026, including architectural strengths and criticisms around host trust, containment and state persistence.
HamRadio.my published a separate overview on 18 September 2026, focusing on the supervised workflow, Hermes integration, guard boundary and evidence model.
Read the HamRadio.my overview ↗
Review context and current state handling
That review described filesystem state as lacking transactional guarantees, highlighted concurrent-session corruption risk, and noted an unversioned checkpoint format. The current implementation has since added atomic file replacement, advisory per-file locks, an engagement workflow lock for multi-file transitions, concurrency regression tests, and a versioned checkpoint template.
These controls reduce the specific race and partial-write risks raised in the review, but they do not turn Violin's filesystem state into a database transaction layer, a sandbox, or a multi-tenant isolation boundary. Reviewers should test those limits directly against the exact commit they evaluate.
How Violin differs from autonomous pentest agents
Violin is not positioned as an autonomous replacement for a penetration tester. Its design keeps authorization, scope changes, destructive or sensitive actions, finding interpretation and final reporting under human control.
| Dimension | Violin | Typical autonomous-agent design |
|---|---|---|
| Operating model | Supervised engagement with explicit operator approvals | Agent plans and executes a larger share of the workflow independently |
| Target execution | Target commands pass through a required guard boundary | Execution policy varies by framework and deployment |
| Scope | Parsed from an approved engagement file and checked before execution | Often represented as task context, policy, or agent instructions |
| Evidence | Findings bind to authenticated execution receipts and declared evidence | Evidence capture and provenance vary by tool |
| State | Persistent engagement files designed for resumability and audit | May rely on conversation memory, agent memory, databases, or task traces |
| Safety boundary | Documented as procedural enforcement, not network containment | May use sandboxes, allowlists, policy engines, or host controls |
This is a design comparison, not a claim that one architecture is universally better. Reviewers should evaluate the controls and trade-offs against their own threat model and operating process.
For reviewers and security teams
Independent criticism is welcome. If you publish a review, test, benchmark or integration report, please link the exact Violin version or commit you evaluated so results remain reproducible.
For private security reports, use SECURITY.md.