Automator capability matrix¶
Spectre exposes the same automation ideas over three transports:
| Transport | Client type | Module |
|---|---|---|
| In-process | ComposeAutomator |
:core |
| HTTP | HttpComposeAutomator |
:server |
| Agent attach | AttachedAutomator |
:agent |
They deliberately do not share one Kotlin interface. The agent crosses a
reflection/classloader boundary with its own CBOR DTOs; HTTP has JSON payloads and
HTTP status semantics; in-process returns live AutomatorNode graphs. Forcing a single
runtime type leaks those boundaries.
The contract is a shared contract-test corpus
(AutomatorContractCorpus in :testing) driven against each transport's own client.
Cell states live in Kotlin as CapabilityMatrix — also in :testing — and this page
is the human-readable view of that source of truth.
Cell states (multi-state, not binary)¶
| State | Meaning |
|---|---|
| Supported | Claimed working for that op × transport × platform. Must list executable evidence (test source + CI workflow). A Supported cell without resolvable evidence fails CapabilityMatrixEvidenceTest — fail-closed. |
| Experimental | Works in some environments; not a 1.0 guarantee. |
| Unsupported by design | Will not be offered on this transport (e.g. needs live JVM objects). Always includes a rationale. |
| Not yet CI-executed | Intended for the 1.0 surface, but no CI task has executed the cell yet. Docs must not claim "supported" until the state flips and evidence lands. |
Platform rows are prerequisites, not just OS names: headless JVMs, Linux Xvfb, Wayland sessions, and interactive desktops behave differently for Robot and Compose.
Shared desktop input has an additional cooperative boundary. In-process Required, in-process
JUnit PerTest, CLI, and injected-agent input participate in the per-user desktop coordinator.
Synthetic pointer/keyboard and headless query work do not; synthetic clipboard use does. Current
HTTP callers participate when their supplied automator uses Auto/Required. Cross-request remote
lease tokens and launch-and-attach whole-test leases remain a follow-up, so the matrix must not
claim a multi-request remote transaction today. The entire coordination surface is
Experimental and opt-in; see Experimental desktop input coordination.
Contract form decision (#198)¶
| Option | Outcome |
|---|---|
Single shared interface in :core |
Rejected for 1.0 — agent reflection and HTTP status semantics do not fit cleanly. |
Shared contract-test corpus in :testing |
Chosen. Each transport implements AutomatorContractDriver and runs the same scenarios against the real client. |
Corpus runners today:
| Transport | Test | CI evidence |
|---|---|---|
| In-process | InProcessContractCorpusTest |
.github/workflows/ci.yml (./gradlew check) |
| HTTP | HttpContractCorpusTest |
.github/workflows/ci.yml |
| Agent | AgentContractCorpusTest (+ AgentAttachIntegrationTest) |
.github/workflows/validation-linux.yml (Xvfb, fail-closed JUnit XML), .github/workflows/macos-check.yml. Windows: transport/ACL tests on hosted CI; full UI attach e2e is opt-in on physical desktops (-Pspectre.agent.attachE2e.allowWindows=true) and pre-tag release smoke. |
Intersection ops (current shared surface)¶
These operations exist on all three clients and are the corpus core:
windows/ surface listingallNodesfindByTestTagclick(by canonicalsurfaceId:ownerIndex:nodeIdon remote transports)
Headless CI exercises transport liveness (empty trees OK; unknown-key click must fail). Fixture-backed semantics (non-empty windows, known tags, click/type/screenshot) are claimed for the agent transport under Linux Xvfb and macOS desktop. Windows agent surface listing and click are Experimental (native AF_UNIX works; UI e2e is physical/opt-in, not hosted fail-closed).
Deliberate exclusions¶
These stay in-process only — they need live JVM objects that cannot cross a transport boundary without a different design:
| Operation | Why |
|---|---|
registerIdlingResource / idling resources |
Live callbacks in the UI JVM |
withTracing |
Live tracer hooks |
waitForIdle (idling-resource registration) |
Live callbacks — use fingerprint wait over attach instead (#362) |
Remote waits (waitForNode over agent — #201, also CLI wait-for-node / MCP
wait_for_node; waitUntilGone over agent — #438, also CLI wait-until-gone / MCP
wait_until_gone; waitForIdle fingerprint wait over agent — #362), selectors
(findByText / role / content-description — #202), and input verbs (doubleClick /
swipe / scrollWheel — #203) are Supported on agent under Linux Xvfb and macOS desktop
via AgentContractCorpusTest / reflective wait suites against agent-test-fixture. Agent
pressKey is Supported on Linux Xvfb (fail-closed after focus retries) and
Experimental on macOS desktop (hosted runners may soft-skip OS keyboard focus loss after
retries — same class as typeText). Agent focusWindow (#364) is Supported on Linux
Xvfb and macOS desktop (raises the window hosting a node before real keyboard input); HTTP
focusWindow is Unsupported by design for this issue. HTTP selector entry points are covered
by headless HttpContractCorpusTest. In-process tree / printTree are Supported on
headless (empty snapshot / empty dump; InProcessContractCorpusTest). HTTP
clearAndTypeText and node screenshot are
Unsupported by design on headless (same bar as HTTP typeText / screenshot: the
happy path needs a live node and display-backed Robot/capture). Display-backed HTTP
cells for those ops, and HTTP tree / printTree, remain Not yet CI-executed — the
same bar as HTTP findByText / role / content-description — until a fixture exercises
nested WindowTreeDto conversion or a successful capture/replace. findOneBy* /
structured TextQuery envelopes are covered by HttpTransportExpansionTest (no separate
matrix ops). Some HTTP input cells and agent longClick /
waitForVisualIdle remain Not yet CI-executed. Idling-resource registration over attach
stays Unsupported by design.
The HTTP transport exposes no wait route at all — SpectreServer has none — so every HTTP wait
cell (waitForNode, waitUntilGone, waitForVisualIdle) is Not yet CI-executed because it is
unrouted, not because a routed op is waiting on a display-backed fixture. waitUntilGone was added
only where waitForNode already reaches: in-process, agent attach, CLI, and MCP.
For waitUntilGone, the agent cells are claimed on the strength of two corpus scenarios that run
inside AgentContractCorpusTest: a selector that matches nothing returns instead of burning its
budget, and a selector that is still on screen times out with taxonomy timeout and a message
still naming the selector, the timeout, and the still-present node count. The second scenario is the
one that matters for a transport claim — a boundary that flattened those diagnostics into a bare
"timed out" would fail the cell rather than quietly pass it.
How to read a cell¶
- Find the operation and transport of interest.
- Check the platform prerequisite that matches your environment.
- If the state is Supported, the evidence column (in
CapabilityMatrix) names the test file and workflow that must execute it — CI must not silently skip that cell. - If you add a new op or flip a cell to Supported, update
CapabilityMatrixfirst, land the corpus/test evidence, then refresh this guide if the human summary changed.
Source of truth¶
| Artifact | Location |
|---|---|
| Matrix data + states | testing/.../contract/CapabilityMatrix.kt |
| Corpus runner | testing/.../contract/AutomatorContractCorpus.kt |
| Fail-closed evidence test | testing/.../contract/CapabilityMatrixEvidenceTest.kt |
| Epic | GitHub #197; matrix issue #198 |
Machine-check: ./gradlew :testing:test --tests "*CapabilityMatrixEvidenceTest*".