Architecture¶
Module Structure¶
spectre
├── core/ — shared automation model and desktop automation primitives
├── input-coordinator/ — published local protocol, identity, client, and fencing tokens
├── input-coordinator-server/— published external FIFO coordinator process and recovery state
├── server/ — optional transport layer for cross-JVM access
├── cli/ — agent-facing CLI / daemon / MCP entrypoint
├── recording/ — screenshot / recording API and common JVM implementation
├── recording-macos/ — runtime-only macOS ScreenCaptureKit helper artifact
├── recording-linux/ — runtime-only Linux capture helper artifact
├── recording-windows/ — runtime-only Windows Graphics Capture helper artifact
├── testing/ — test fixtures and JUnit-facing helpers built on top of public APIs
├── sample-desktop/ — small manual-test harness app for spike validation
└── sample-intellij-plugin/ — sample IntelliJ plugin embedding Spectre in a Jewel tool window
Dependency Direction¶
sample-desktop ─┐
sample-intellij-plugin ├──> core
server │
testing ─┴──> recording (failure-video #206 uses AutoRecorder)
↑
recording-macos ─┐
recording-linux ├──> recording (runtime helper artifacts)
recording-windows ─┘
The modules follow these dependency rules:
coredefines reusable automation concepts.coreimplementation-depends only oninput-coordinator; CLI, testing, and the attaching-side agent bringinput-coordinator-server. The injected agent relocates only client/protocol classes and must never contain a server main or launcher.servertransportscoreoperations. It must not implement automation a second time.sample-desktopprovides a test harness. Production automation belongs in the library modules.testingshould use public APIs and reusable fixtures. Access private internals only when a test cannot verify the behaviour through a public path.- The shared contract-test corpus in
:testingchecks transport parity across in-process, HTTP, and agent clients. It usesAutomatorContractCorpusandCapabilityMatrix. See the capability matrix and epic #197. recordingisolates OS and native library behaviour from core query and input APIs. Platform helper artefacts carry runtime binaries. The API jar does not.
Spike Constraints¶
The implementation must account for these Compose Desktop constraints:
- Read semantics through the public
ComposeWindow.semanticsOwners,ComposeDialog.semanticsOwners, andComposePanel.semanticsOwnersproperties.DialogWindowcreates aComposeDialog. - Scope public node identifiers to their
SemanticsOwner, or use compound identifiers.SemanticsNode.idis unique only within one owner. - Search for popup roots inside each window. Compose can keep popup layers on the same canvas.
- Convert Compose semantics coordinates before passing them to Robot or AWT screen APIs. HiDPI scaling affects that conversion.
- Check for a native
windowHandlebefore selecting window recording. An embedded Swing-hosted Compose surface may not expose one.
Planned Responsibility Split¶
core¶
Expected long-term responsibilities:
- window/surface tracking
- semantics tree reading and normalisation
- owner/root-aware node identity
- selector/query API
- coordinate mapping between Compose, AWT, and Robot
- Robot-backed interaction primitives
- wait/synchronisation semantics
server¶
Expected long-term responsibilities:
- request/response DTOs
- serialisation
- embedded HTTP server
- remote client
Keep transport-specific fields out of the core data model.
Desktop input coordinator¶
A local process serialises desktop-interaction for each normalised OS user and desktop
key. Only its owner can access it. The coordinator manages FIFO acquisition, bounded
queues, heartbeats, epochs, fencing, restart quarantine, compare-and-revoke, and
audited unsafe takeover. Core input guards use a driver-bound JUnit lease first, then
a coroutine-scoped lease. They acquire an operation lease if neither exists. Queries,
waits, screenshots, and recording do not acquire that automatic lease. The agent runs
lease waiters on a bounded input worker to protect the general query pool.
cli¶
The cli module provides the spectre client and the per-user session daemon contract.
It also owns the MCP entry point. Both frontends exchange versioned CBOR handshake messages
with the daemon before they send session commands.
The CLI and daemon protocol stays outside agent-runtime. The daemon can use jdk.attach
and agent transport. The runtime jar must remain small enough for arbitrary target JVMs.
recording¶
Expected long-term responsibilities:
- screen/region capture orchestration
- platform recorder selection and helper extraction
- recorder lifecycle and output plumbing
Keep native capture interfaces small. Test pure logic separately from OS integration.
The published spectre-recording jar is API/common-only; it must not contain generated
native/... helper resources.
recording-macos, recording-linux, and recording-windows¶
Expected long-term responsibilities:
- publish runtime-only helper resources for the recording API
- keep native binary payloads out of
spectre-recordingand its sources jar - mirror the runtime resource paths the extractors probe:
native/macos/SpectreCaptureHelper.app,native/linux/<arch>/spectre-wayland-helper, andnative/windows/<arch>/multi-file Windows Graphics Capture helper tree (exe + companions; framework-dependent)
Current backends:
FfmpegRecorderprovides deprecated, explicit region capture through a systemffmpegbinary.FfmpegBackend.detect()choosesavfoundationon macOS,gdigrabon Windows, andx11grabon Linux Xorg. It rejects Linux Wayland; use the portal-backed Linux helper there.windows.WindowsGraphicsCaptureRecorder— Windows-only MP4 capture via the shared .NET Windows Graphics Capture helper packaged by:recording-windowsfor x64 and arm64. Window mode follows movement automatically and ignores occluders; region mode records a fixed monitor rectangle and is also used for fullscreen recording.FfmpegWindowRecorder— deprecated legacy explicit Windows-only window-targeted capture viagdigrab title=. Window movement is followed automatically; occlusion doesn't matter.windows.WindowsWindowScreenshotter— Windows-only still window screenshots via a shared framework-dependent .NET Windows Graphics Capture helper packaged by:recording-windowsfor x64 and arm64.LinuxX11Recorder— Linux Xorg/Xvfb region and named-window capture via thespectre-recording-linuxhelper and GStreamerximagesrc. Window mode resolves an XID from the title and fails if that window is missing.LinuxNativeScreenshotter— Linux still screenshots via the same helper: GStreamerximagesrcon Xorg/Xvfb, and one-frame portal/PipeWire capture on Wayland.FfmpegRegionScreenshotter— deprecated legacy explicit Linux X11 still screenshot fallback via one-framex11grabregion capture; the target must be visible and frontmost because this is not true window capture.screencapturekit.ScreenCaptureKitRecorder— macOS-only window-targeted and region capture via a Swift helper (recording/native/macos/). The helper is built by Gradle on macOS, staged underrecording/build/generated/screenCaptureHelper/..., and packaged by:recording-macos.screencapturekit.ScreenCaptureKitScreenshotter— macOS-only still window screenshots through the same Swift helper in--mode screenshot.portal.WaylandPortalRecordercaptures Linux Wayland throughxdg-desktop-portal. Region and monitor capture share a long-livedspectre-wayland-helper --sessionprocess with real OS input. That process uses RemoteDesktop, ScreenCast, hostRegistry.Register, and a Spectre-ownedrestore_tokenunder$XDG_STATE_HOME/spectre. Window-source capture uses a separate one-shot ScreenCast helper. The:recording-linuxartefact packages the helper. It passes the PipeWire FD togst-launch-1.0.AutoRecorderselects a backend for eachstartWindow(...)orstartRegion(...)call. It uses the Wayland portal on Wayland, macOS SCK on macOS, Windows Graphics Capture on Windows, and the Linux helper on Xorg/Xvfb. Each route selects a window or region backend for the requested mode.AutoScreenshotter— high-level still screenshot router: macOS SCK window source, Windows Graphics Capture, and Linux helper still capture on Xorg/Xvfb and Wayland.
testing¶
Expected long-term responsibilities:
- JUnit rules/extensions
- reusable fixtures for validation of public APIs
- focused contract-test helpers for transport, selectors, and geometry
sample-desktop¶
Expected long-term responsibilities:
- tiny exploratory app for manual spike validation
- reproducible surfaces for popup, focus, scrolling, and coordinate tests
Move reusable production logic out of the sample app and into its owning module.
sample-intellij-plugin¶
The sample IntelliJ plugin verifies in-process automation of Jewel tool windows.
It is never published. The runIde checks cover popup discovery and ComposePanel
semantics. Production logic belongs in the library modules.
Run via ./gradlew :sample-intellij-plugin:runIde, then Tools → Run Spectre Against the
Sample Tool Window. The action drives the in-process ComposeAutomator against the Jewel
tool window and dumps the discovered semantics tree to idea.log.
./gradlew :sample-intellij-plugin:uiTest runs the automated counterpart through
intellij-ide-starter (#42). It starts IntelliJ Ultimate in a child process and
installs the locally built plugin zip. The Driver API fires RunSpectreAction.
The test checks that idea.log contains every tagged Compose node from
SpectreSampleToolWindowContent. It uses the manual smoke assertions. This opt-in
test is outside :check. CI runs it through .github/workflows/ide-uitest.yml
when plugin, core, or recording sources change.
Architectural Invariants¶
These should remain true as the codebase grows:
corestays usable in-process without requiring the server.- Selector/query logic belongs in
core; the sample app and transport layer use that logic. - Transport modules serialise public/core-facing models. A strong compatibility reason is required to introduce a parallel model.
- Platform-specific integrations should be isolated behind small interfaces at module boundaries.
- Research-only shortcuts are acceptable in the sample app. Move reusable behaviour into the proper module before it is treated as part of the product surface.
- Never substitute a different mechanism for the one the caller asked for. If the requested mechanism is slow, needs consent, or fails, fix it within that mechanism, or fail with a clear error. Do not quietly do something else that only looks similar. The only exception is a substitution that the maintainer explicitly asks for or approves.
- A window capture is always a real window capture: the OS captures that window's own pixels, through the platform's window-capture API. It is never replaced by a capture of the screen or monitor cropped to the window's bounds (#355, #559). The two use different OS APIs and give different results. A crop of the screen includes whatever covers the window, and it breaks when the window moves or is hidden.
- Cropping a real window capture to a surface inside that window (for example a Compose panel inside its host window) is still a window capture, so it is allowed.
- A screen or monitor capture is correct only when the caller asked for the screen.