[repository-quality] Repository Quality Improvement — Engine Feature Parity (2026-04-22)

[github-actions[bot]]

🎯 Repository Quality Improvement Report — Engine Feature Parity

Analysis Date: 2026-04-22
Focus Area: Engine Feature Parity and Integration Completeness
Strategy Type: Custom (first run, no history)
Custom Area: Yes — gh-aw supports 6 AI engines (copilot, claude, codex, gemini, opencode, crush). As a multi-engine tool, consistency in how engines are wired into the compilation and runtime infrastructure is critical for reliability and future engine onboarding.

---

Executive Summary

gh-aw supports six AI coding engines, implemented through the CodingAgentEngine interface and a BaseEngine struct providing defaults. The mature engines (copilot, claude, codex) are well-tested and fully integrated; the newer engines (gemini, opencode, crush) show meaningful capability and testing gaps.

Key findings: (1) only copilot and claude override APM targets while four engines fall back to "all", which may mis-pack dependencies; (2) OpenCode and Crush have only 7 test files each vs 70–252 for older engines; (3) Crush declares web search in its hardcoded config JSON but supportsWebSearch is false, blocking the neutral tools system from using it; (4) no engine-capability reference table exists in the user-facing docs.

Full Analysis Report

Focus Area: Engine Feature Parity

Current State Assessment

Metrics Collected:

Metric	Value	Status
Total engines	6 (copilot, claude, codex, gemini, opencode, crush)	✅
Experimental engines	2 (opencode, crush)	✅
Engines with custom APM target	2 / 6	⚠️
Engines with ParseLogMetrics override	3 / 6 (copilot, codex, gemini)	⚠️
Engines with supportsToolsAllowlist=true	4 / 6	✅
Test files referencing copilot	252	✅
Test files referencing claude	167	✅
Test files referencing codex	70	✅
Test files referencing gemini	25	⚠️
Test files referencing opencode	7	❌
Test files referencing crush	7	❌
Crush web search: config enabled vs neutral-tools exposed	enabled / not exposed	⚠️

Findings

Strengths

• All six engines implement the full CodingAgentEngine interface via BaseEngine defaults — no missing method stubs
• Engine capability flags (supportsMaxTurns, supportsBareMode, etc.) are documented with inline comments explaining why each is true/false
• Engine constants and EngineOptions catalog are well-structured in pkg/constants/engine_constants.go
• engine_definition_test.go covers all 6 engines for catalog resolution and prefix matching
• Claude has a dedicated JS log parser (parse_claude_log) wired via GetLogParserScriptId()

Areas for Improvement

1. APM target defaulting to "all" for 4 engines ⚠️ — BaseEngine.GetAPMTarget() returns "all". Only CopilotEngine ("copilot") and ClaudeEngine ("claude") override this. Codex, Gemini, OpenCode, and Crush all default to "all", potentially packing unnecessary apm primitives.

2. OpenCode and Crush test coverage gap ❌ — Both have only 7 test files vs 70+ for other stable-tier engines. Core scenarios (tool expansion, network config, model env var injection, firewall generation) are under-tested.

3. Crush web search not surfaced ⚠️ — crush_engine.go line 226 shows the hardcoded Crush config JSON explicitly enables "websearch": "allow", yet supportsWebSearch is false. This means gh-aw cannot offer web-search as a neutral tool capability to Crush users even though the underlying CLI supports it.

4. No engine capability matrix in docs ⚠️ — Users cannot easily compare engines to know which features (max-turns, web-search, tools allowlist, bare-mode, max-continuations) are available per engine. This creates confusion during engine selection.

5. Gemini ParseLogMetrics uses base no-op i️ — gemini_engine.go contains JSON log parsing logic in comments but ParseLogMetrics falls through to the base no-op. Metric extraction is missing at the Go level (though a JS-level parser may exist).

Detailed Analysis

APM Target: microsoft/apm-action packs engine-specific primitives to speed up dependency installation. Returning "all" forces all primitive types to be packed regardless of engine, inflating job time and artifact size for Codex, Gemini, OpenCode, and Crush workflows.

Test Coverage: The 7-file coverage for OpenCode and Crush covers only basic engine construction and model env var injection. Missing test scenarios include: MCP config generation, firewall step generation, tool allowlist enforcement (or its absence), and log collection step generation.

Crush WebSearch: Line 226 of crush_engine.go hard-codes "websearch":"allow" in the Crush agent config JSON. The gh-aw neutral tools system checks SupportsWebSearch() before routing a web-search tool request to an engine. Updating supportsWebSearch: true would allow users to enable web search for Crush workflows without any other changes.

---

🤖 Tasks for Copilot Agent

NOTE TO PLANNER AGENT: The following tasks are designed for GitHub Copilot coding agent execution. Please split these into individual work items for Claude to process. Each task is self-contained.

---

Improvement Tasks

Task 1: Enable Engine-Specific APM Targets for Codex and Gemini

Priority: Medium
Estimated Effort: Small
Focus Area: Engine Feature Parity

Description:
BaseEngine.GetAPMTarget() returns "all". Only CopilotEngine and ClaudeEngine override this with engine-specific values. Codex and Gemini should return values matching their toolchain identity so apm-action packs only necessary primitives.

Acceptance Criteria:

• CodexEngine.GetAPMTarget() overrides and returns "codex" (or "openai" if that matches apm-action's supported targets)
• GeminiEngine.GetAPMTarget() overrides and returns "gemini" (or "google")
• Both overrides are documented with comments consistent with the existing pattern in copilot_engine.go and claude_engine.go
• engine_definition_test.go is updated to assert the expected APM target per engine
• make test-unit passes

Code Region: pkg/workflow/codex_engine.go, pkg/workflow/gemini_engine.go, pkg/workflow/engine_definition_test.go

In `pkg/workflow/codex_engine.go`, add a `GetAPMTarget()` method to `CodexEngine` that returns `"codex"`. Model it after `CopilotEngine.GetAPMTarget()` in `pkg/workflow/copilot_engine.go`.

In `pkg/workflow/gemini_engine.go`, add a `GetAPMTarget()` method to `GeminiEngine` that returns `"gemini"`. Same pattern.

Then update `pkg/workflow/engine_definition_test.go` to assert the correct APM target value for each engine (copilot→"copilot", claude→"claude", codex→"codex", gemini→"gemini", opencode→"all", crush→"all").

Run `make fmt` and `make test-unit` to validate. Follow the patterns in `agentic_engine.go` and existing engine files.

---

Task 2: Expand OpenCode Engine Test Coverage

Priority: High
Estimated Effort: Medium
Focus Area: Test Coverage — OpenCode Engine

Description:
OpenCodeEngine has only 7 test files referencing it, covering basic construction and model env var injection. Critical scenarios like MCP config generation, firewall step generation, network config, and engine-specific compilation output are untested.

Acceptance Criteria:

• A new test file pkg/workflow/opencode_engine_compilation_test.go is created
• Tests cover: MCP config generation with opencode engine, firewall step generation, model env var injection into compiled YAML, supportsToolsAllowlist=false behavior (no tool filtering applied)
• Tests use table-driven pattern (t.Run) with descriptive names
• Build tag //go:build !integration is present at top of file
• make lint and make test-unit pass

Code Region: pkg/workflow/opencode_engine.go, pkg/workflow/opencode_engine_test.go

Create `pkg/workflow/opencode_engine_compilation_test.go` with `//go:build !integration` at the top.

Add table-driven tests using the existing patterns in `pkg/workflow/codex_engine_test.go` and `pkg/workflow/gemini_engine_test.go` as reference. Cover these scenarios:

1. Compiling a minimal workflow with `engine: opencode` produces valid YAML output
2. Model env var (`OPENCODE_MODEL`) appears in the compiled job's env block when `engine.model` is set in frontmatter
3. `SupportsToolsAllowlist()` returns false for OpenCode (so tool allow-listing is skipped)
4. MCP server config is generated when `tools` are specified

Use `NewCopilotEngineForTest()` or equivalent test helper patterns from the package. Run `make fmt` and `make lint` before committing.

---

Task 3: Expand Crush Engine Test Coverage

Priority: High
Estimated Effort: Medium
Focus Area: Test Coverage — Crush Engine

Description:
CrushEngine has only 7 test files. The engine has a unique behavior: it uses a hardcoded configJSON with all tool permissions enabled (bash, edit, read, glob, grep, write, webfetch, websearch) but supportsToolsAllowlist=false. This behavior is currently untested.

Acceptance Criteria:

• A new test file pkg/workflow/crush_engine_compilation_test.go is created
• Tests cover: compiled YAML includes hardcoded Crush config JSON, model env var injection (CRUSH_MODEL), supportsToolsAllowlist=false behavior, firewall-related step generation
• Build tag //go:build !integration at top of file
• make lint and make test-unit pass

Code Region: pkg/workflow/crush_engine.go, pkg/workflow/crush_engine_test.go

Create `pkg/workflow/crush_engine_compilation_test.go` with `//go:build !integration` at top.

Add table-driven tests using `pkg/workflow/crush_engine_test.go` and `pkg/workflow/codex_engine_test.go` as reference. Cover:

1. Compiling a minimal workflow with `engine: crush` produces valid YAML output containing the Crush agent config JSON
2. Model env var (`CRUSH_MODEL`) appears in compiled job env when `engine.model` is set
3. `SupportsToolsAllowlist()` returns false for Crush
4. The hardcoded config JSON (from `crush_engine.go` line ~226) is present in compiled output, verifying websearch and bash permissions are included

Run `make fmt` and `make lint` before committing.

---

Task 4: Expose Crush Web Search via Neutral Tools System

Priority: Medium
Estimated Effort: Small
Focus Area: Engine Feature Parity — Crush

Description:
crush_engine.go already enables "websearch":"allow" in its hardcoded agent config JSON (line ~226), meaning the Crush CLI supports web search. However, supportsWebSearch is set to false, preventing users from declaring web-search: true in workflow frontmatter for Crush workflows. This flag should be enabled, with any necessary wiring in the neutral tools expansion path.

Acceptance Criteria:

• CrushEngine.BaseEngine.supportsWebSearch is changed from false to true with an updated comment
• Validation test confirms that web-search: true in frontmatter compiles successfully for engine: crush
• If there is a neutral tools expansion path for web search (expandNeutralToolsToXTools), Crush's path handles it correctly (or the base behavior is sufficient)
• make test-unit passes

Code Region: pkg/workflow/crush_engine.go

In `pkg/workflow/crush_engine.go`, change `supportsWebSearch: false` to `supportsWebSearch: true` and update the comment from "Has built-in websearch but not exposed via gh-aw neutral tools yet" to reflect that it is now exposed.

Check whether any neutral-tools expansion is needed by examining how Claude and Codex handle web search in their engine files (`claude_tools.go`, `codex_engine.go`). If the base behavior of `SupportsWebSearch()` returning true is sufficient (the Crush config JSON already enables websearch), no further changes are needed beyond the flag.

Add a test to `pkg/workflow/crush_engine_test.go` asserting `engine.SupportsWebSearch() == true`.

Run `make fmt` and `make test-unit`.

---

Task 5: Add Engine Capability Matrix to Documentation

Priority: Low
Estimated Effort: Small
Focus Area: Documentation — Engine Comparison

Description:
Users choosing between engines have no single reference showing which gh-aw features each engine supports. A capability matrix table in the docs would reduce confusion and support questions.

Acceptance Criteria:

• A new file docs/src/content/docs/reference/engine-capabilities.md is created (or an existing engine reference page is updated)
• The page includes a table with columns for each engine and rows for key capabilities: Tools Allowlist, Max Turns, Max Continuations, Web Search, Native Agent File, Bare Mode, Experimental
• Table is accurate based on BaseEngine struct values in each engine implementation
• Doc page is linked from docs/src/content/docs/reference/ navigation or an existing engines reference page

Code Region: docs/src/content/docs/reference/

Create `docs/src/content/docs/reference/engine-capabilities.md`.

Include a capabilities matrix table. Use the following values derived from the `BaseEngine` structs in `pkg/workflow/`:

| Feature | Copilot | Claude | Codex | Gemini | OpenCode | Crush |
|---------|---------|--------|-------|--------|----------|-------|
| Tools Allowlist | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| Max Turns | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ |
| Max Continuations | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ |
| Web Search | ❌ | ✅ | ✅ | ❌ | ❌ | ⚠️ (built-in) |
| Bare Mode | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ |
| Experimental | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ |

Add brief descriptions for each row explaining what the feature does. Use GitHub Flavored Markdown. Ensure the page uses h3 (###) or lower for headers.

---

📊 Historical Context

Previous Focus Areas

Date	Focus Area	Type	Custom	Key Outcomes
2026-04-22	Engine Feature Parity	Custom	Yes	5 tasks: APM targets, test coverage for OpenCode/Crush, Crush web search, engine docs

---

🎯 Recommendations

Immediate Actions (This Week)

1. Expand OpenCode engine test coverage — Priority: High
2. Expand Crush engine test coverage — Priority: High

Short-term Actions (This Month)

1. Add APM targets for Codex and Gemini engines — Priority: Medium
2. Enable Crush web search via neutral tools — Priority: Medium

Long-term Actions (This Quarter)

1. Add engine capability matrix to docs — Priority: Low

---

📈 Success Metrics

Track these to measure improvement in Engine Feature Parity:

• OpenCode test file count: 7 → 15+
• Crush test file count: 7 → 15+
• Engines with specific APM target: 2/6 → 4/6
• Crush web search: not exposed → exposed via neutral tools
• Engine capability docs: 0 pages → 1 reference page

---

Next Steps

1. Review and prioritize tasks above
2. Assign Task 2 and Task 3 (test coverage) to Copilot coding agent — highest immediate impact
3. Task 1 (APM targets) is a quick win that can be bundled with Task 2/3
4. Re-evaluate engine parity after experimental engines (OpenCode, Crush) graduate to stable

---

References:

• Workflow Run §24780312949

Generated by Repository Quality Improvement Agent · ● 1.5M · ◷

• expires on Apr 23, 2026, 1:27 PM UTC

[github-actions[bot]]

This discussion has been marked as outdated by Repository Quality Improvement Agent.

A newer discussion is available at Discussion #28091.