Driving input¶

The interaction layer of ComposeAutomator sits on top of RobotDriver and dispatches mouse, keyboard, and clipboard input to whatever surface the target node lives in.

All interaction methods (click, doubleClick, longClick, moveTo, moveBy, swipe, scrollWheel, typeText, clearAndTypeText, pressKey, pressEnter) are suspend — call them from a coroutine. Real java.awt.Robot work runs inline when the caller is already off the AWT event dispatch thread, and hops to Dispatchers.IO only when needed to keep EDT callers from blocking the UI. Internal sleeps use delay rather than Thread.sleep, so a cancelled coroutine cancels mid-longClick / mid-swipe rather than parking the worker thread until the hold completes.

The explicit screenshot(region) overload stays sync — it is a single framebuffer read, with no blocking I/O to bury behind a coroutine boundary. Window- and node-targeted screenshots can instead launch a native capture helper and wait for a frame; use them from a coroutine-friendly test context when that latency matters.

The snippets below are written as if they sit inside a suspend block (e.g. a JUnit test wrapped in runSpectreTest { … }).

Cooperative desktop input leases¶

Experimental and opt-in

This surface requires @OptIn(ExperimentalSpectreInputCoordinationApi::class) and may change in any release. See Experimental desktop input coordination for dependencies, JUnit setup, recovery controls, platform boundaries, and graduation criteria.

Real mouse, keyboard, focus, and system-clipboard work can be coordinated across participating Spectre processes. Opt in on the driver with InputLeasePolicy.Required, or hold one reentrant lease across a larger sequence:

@file:OptIn(ExperimentalSpectreInputCoordinationApi::class)

import dev.sebastiano.spectre.input.ExperimentalSpectreInputCoordinationApi
import dev.sebastiano.spectre.core.InputLeaseOptions
import dev.sebastiano.spectre.core.InputLeasePolicy
import dev.sebastiano.spectre.core.RobotDriver

val automator = ComposeAutomator.inProcess(RobotDriver(InputLeasePolicy.Required))

automator.withExclusiveInput(InputLeaseOptions(ownerLabel = "fills login form")) {
    focusWindow(username)
    click(username)
    typeText("alice")
    click(submit)
    automator.waitForIdle()
}

Auto coordinates capabilities that touch shared OS state when the coordinator runtime is available; Required fails closed if it is unavailable; Off is the explicit escape hatch for externally coordinated or hermetic callers. The no-argument driver keeps the pre-coordinator default while the feature is experimental; select Auto or Required deliberately.

The lease covers real pointer/keyboard/focus operations and pasteText's system clipboard use. Synthetic pointer and keyboard work stays parallel. Synthetic pasteText coordinates only when the synthetic driver was created with an explicit Auto or Required policy; the ordinary RobotDriver.synthetic(rootWindow) factory remains Off. Screenshots, semantics reads, waits, and recording do not acquire automatically; put a visibility-sensitive capture inside withExclusiveInput when it must not race input.

Coordination is cooperative: it prevents participating Spectre clients from interleaving. It cannot block a human, an older Spectre version, or another automation framework. A contended EDT call fails immediately with ContendedEdtInputLeaseException; acquire the scope from a non-EDT test thread or use JUnit per-test isolation instead of blocking the AUT's event thread. Auto also avoids opening a new coordinator session from the EDT: without an already-connected session it proceeds uncoordinated. Choose Required when that fallback is unacceptable.

The attach path uses Required whenever the target can coordinate at all — an injected agent drives the target's default synthetic driver (or real OS input if that target opted in) into a process you do not own, so a coordinator it cannot reach fails coordinated verbs rather than quietly stopping policing the desktop. When the coordinator itself is broken and that leaves you with no way forward, there is one deliberate opt-out; see When the coordinator cannot be reached.

The exception is a target whose preinstalled Spectre core predates InputLeasePolicy. The agent falls back to the legacy no-argument driver there, which coordinates nothing — see Attach. Injected targets and current-core targets are unaffected.

Mouse: clicks and drags¶

val send = automator.findOneByTestTag("Send") ?: error("button missing")
automator.click(send)
automator.doubleClick(send)
automator.longClick(send, holdFor = 600.milliseconds)

All click helpers resolve the node's centerOnScreen and dispatch through RobotDriver, which compensates for HiDPI/display scaling.

Undelivered input fails loudly on Windows and X11¶

On Windows, node-targeted click, doubleClick, longClick, swipe(from, to) and scrollWheel(node) check that the event actually arrived. On X11, the same check runs for clicks (click, doubleClick, longClick, swipe) but not for the wheel. If Spectre dispatches the input and no matching mouse event reaches the target JVM, the call throws IllegalStateException in process, or reports inputRejected over the attach path. Earlier versions returned normally without doing anything.

Windows discards injected input when the calling process is not on the session's active input desktop. The common causes are a locked workstation, or a process running on a desktop that is not the input one — including a non-interactive session such as Windows session 0, an SSH shell, or a scheduled task started while logged out. Input that lands somewhere else looks the same from inside the target: another window covering it, or a scroll delivered to the focused window rather than the one under the pointer. On X11 the same silent no-op can happen for a click that never reaches this JVM.

The check does not run on macOS, where AppKit swallows the click that activates an application, or for X11 wheel events, which arrive as button presses.

If you hit this, fix the environment rather than the test: run on an unlocked, interactive desktop. See #460 for the Windows investigation.

Mouse: hover and pointer moves¶

val send = automator.findOneByTestTag("Send") ?: error("button missing")

// park the pointer on a node (hover / tooltip) without clicking
automator.moveTo(send)

// raw screen coordinates (same space as swipe)
automator.moveTo(x = 240, y = 80)

// offset from the last Spectre-issued pointer position
automator.moveBy(deltaX = 12, deltaY = -4)

moveTo / moveBy never press or release a button. moveBy is relative to the last Spectre-issued move on this automator's RobotDriver (click, doubleClick, longClick, swipe, scrollWheel, moveTo, or a previous moveBy) — it does not read the OS cursor. If no Spectre move has happened yet, moveBy throws IllegalStateException.

Mouse: swipes and scrolling¶

val list = automator.findOneByTestTag("MessageList") ?: error("list missing")
val first = automator.findOneByText("First message") ?: error("first row missing")
val last  = automator.findOneByText("Last message")  ?: error("last row missing")

// node-to-node drag
automator.swipe(from = first, to = last)

// raw coordinates (HiDPI-corrected)
automator.swipe(
    startX = 100, startY = 400,
    endX = 100, endY = 100,
    steps = 16,
    duration = 200.milliseconds,
)

// mouse-wheel scrolling — drives Modifier.scrollable / LazyColumn on desktop
automator.scrollWheel(list, wheelClicks = 5)   // scroll down
automator.scrollWheel(list, wheelClicks = -5)  // scroll up

scrollWheel is the right helper for desktop scrollable containers — they respond to wheel events rather than touch-style drags.

Keyboard: typing and key events¶

import java.awt.event.InputEvent
import java.awt.event.KeyEvent

val input = automator.findOneByTestTag("MessageInput") ?: error("input missing")

// click-then-type
automator.click(input)
automator.typeText("Hello, Spectre!")

// click-clear-type in one go (uses key events, not the clipboard)
automator.clearAndTypeText(input, "replacement text")

// clipboard paste for large or Unicode text
automator.pasteText("こんにちは, Spectre!")

// raw key events
automator.pressKey(KeyEvent.VK_TAB)
automator.pressKey(KeyEvent.VK_S, modifiers = InputEvent.CTRL_DOWN_MASK) // Ctrl+S

// shorthand
automator.pressEnter()

pressKey's modifiers parameter takes an AWT modifier mask (InputEvent.CTRL_DOWN_MASK, InputEvent.SHIFT_DOWN_MASK, …) — not a KeyEvent constant. The driver translates the mask into the right modifier-key presses around the main keyCode.

typeText dispatches key press/release pairs and does not touch the clipboard. It is intentionally conservative: ASCII letters, digits, space, newline, and common US-keyboard punctuation. Requested letter case is independent of ambient Caps Lock: Spectre reads the Caps Lock LED and inverts Shift on letter strokes when it is on (global lock state is left unchanged). Use pasteText for large strings or arbitrary Unicode; it stashes the previous clipboard contents, writes the requested text, dispatches the platform paste shortcut (Cmd+V on macOS, Ctrl+V elsewhere), waits for the paste handler to drain, then restores the previous clipboard contents. See Troubleshooting for macOS clipboard and apple.awt.UIElement=true caveats.

Screenshots¶

import dev.sebastiano.spectre.core.WindowScreenshotResult
import java.awt.Rectangle
import java.io.File
import javax.imageio.ImageIO

// whole virtual screen
val full = automator.screenshot()
ImageIO.write(full, "png", File("screenshot.png"))

// a single window's Compose surface
val mainWindow = automator.screenshot(windowIndex = 0)
ImageIO.write(mainWindow, "png", File("main.png"))

// several physical windows, listed from back to front (main window, then dialog/popup)
val scene = automator.screenshotWindows(listOf(0, 1))
scene.windows.forEachIndexed { index, result ->
    if (result is WindowScreenshotResult.Success) {
        ImageIO.write(result.image, "png", File("window-$index.png"))
    }
}
scene.composite?.let { ImageIO.write(it.image, "png", File("composite.png")) }

// a single node
val send = automator.findOneByTestTag("Send") ?: error("button missing")
val sendShot = automator.screenshot(send)

// arbitrary screen region
val region = automator.screenshot(Rectangle(0, 0, 800, 600))

Returns a BufferedImage you can save, hash, or compare against a baseline.

screenshotWindows is intended for heavyweight popup renderers such as Swing/Jewel dialogs, menus, and tooltips. Each requested physical window is captured separately, then placed into a transparent canvas using its screen position. The index list is painter's order (back to front), so put an owner before its popups. The canvas uses the union of all requested bounds rather than the owner's bounds, so a popup that extends above, below, or beside its owner is not clipped. Successful source captures include their screen bounds, device density, and capture order. If one window cannot be captured, its entry is a WindowScreenshotResult.Failure and composite is null; successful individual images remain available for diagnostics.

Window captures prefer native backends

screenshot(windowIndex) and screenshot(node) prefer a native, window-scoped backend when spectre-recording is on the runtime classpath; screenshot(region) remains an explicit screen-region capture. If a native backend is unavailable or cannot identify the selected window unambiguously, Spectre fails with an actionable error; it never substitutes a screen framebuffer crop. Linux Xorg/Xvfb window capture reads visible screen pixels, so keep the target visible and frontmost there. If another app overlaps the window, those overlapping pixels can appear in the image. For top-level windows, spectre-recording also exposes AutoScreenshotter, which uses native/window-targeted backends on macOS and Windows, and the Linux helper on Xorg/Xvfb and Wayland.

Captures are normalised to sRGB

The returned BufferedImage is always sRGB (TYPE_INT_ARGB with an sRGB ColorModel), regardless of the source display's colour profile. Capturing on a wide-gamut display (Display P3 on a modern Mac, Adobe RGB, etc.) goes through the OS's display pipeline and lands in the buffer as sRGB pixels. This keeps captures portable — a baseline collected on one machine compares meaningfully against a capture from another — but it means the captured pixel values are post-display-pipeline, not the raw Color(...) your Compose code passed. Plan for ±1–2 per-channel rounding noise from the gamma round-trip when you assert on colour, and use a tolerant comparator (see below).

Bitmap comparison needs tolerance

Don't compare screenshots byte-for-byte against a baseline. Identical-looking frames routinely differ at the pixel level because of:

Encoder/decoder round-trips (PNG re-saves can shift LSBs).
Text rendering: subpixel positioning, hinting, font fallback, font version.
Antialiasing on edges, gradients, and blurs.
OS- and GPU-driven differences in compositing, gamma, and colour profiles.
HiDPI scaling at non-integer factors.

Always compare with a tolerance — perceptual diff (e.g., a small ΔE threshold), a per-channel allowance, or a structural metric like SSIM. Region-mask the parts of the UI that are inherently noisy (timestamps, cursors, animations).

Spectre intentionally doesn't ship a screenshot comparison suite — it returns BufferedImage and lets you wire whatever comparator fits your stack. If there's demand, a built-in tolerant comparator could land later; open an issue describing the use case if you'd find it valuable.

For test output that records continuous video rather than per-step images, see Recording.

Real vs. synthetic input¶

The RobotDriver your automator wraps governs how input is actually dispatched. The public surface:

RobotDriver.synthetic() / RobotDriver.synthetic(rootWindow) — the default. ComposeAutomator.inProcess() defaults to synthetic AWT events posted straight into the live window hierarchy. No real cursor motion, no global focus, doesn't fight with other processes. The no-window factory hit-tests every visible top-level window; passing rootWindow pins the starting tree. Mouse and wheel events hit-test against that tree plus other visible top-level windows. Key events go to the current AWT focus owner when one exists; when AWT has no focus owner (for example, a macOS helper JVM launched with apple.awt.UIElement=true), Spectre falls back to the key-listening AWT descendant under the last pointer target or Compose host. That lets Compose Desktop's internal focus model route typeText into focused TextFields even when the host window is not the OS-foreground app. screenshot() under a synthetic driver still uses the OS framebuffer via Robot.createScreenCapture, so screenshots show the pixels the display compositor currently exposes rather than a Swing repaint of the Compose host. On macOS this still requires Screen Recording permission and an unlocked screen, but synthetic input itself does not need Accessibility permission. On Linux Wayland, that framebuffer read goes through the same long-lived spectre-wayland-helper session as real OS input — not a per-JVM java.awt.Robot portal.
RobotDriver() — the explicit real-OS opt-in. On X11, macOS, and Windows this uses a fresh java.awt.Robot plus the system clipboard. On Linux Wayland it routes pointer, keyboard, and region capture through spectre-wayland-helper so compositor clicks restack without XTest and so parallel JVMs share one RemoteDesktop grant. Moves the real cursor, takes system-wide keyboard focus, and is visible to other applications. This is what end users experience; pass it to ComposeAutomator.inProcess(robotDriver = RobotDriver()) when you need that fidelity. On macOS, the first input or screenshot call lazily probes TCC permissions (Accessibility for input, Screen Recording for capture) and throws IllegalStateException with remediation guidance when either is denied — see Troubleshooting. On Wayland, the first seated run shows one Share + Remember / Allow remote interaction dialog; later Spectre processes reuse the helper session. See Linux Wayland consent.
RobotDriver(robot) — same as the no-arg real-OS form but reuses an existing java.awt.Robot you've already constructed (e.g., one targeted at a non-default GraphicsDevice).
RobotDriver.headless() — for read-only flows in headless CI where real OS I/O is unavailable. Every input, clipboard, and screenshot call throws UnsupportedOperationException so an accidental automator.click(...) / typeText(...) / screenshot(...) surfaces at the call site instead of silently dropping. Semantics-tree reads still work — pair this with ComposeAutomator.performSemanticsClick(node) if you need to fire clicks without going through the OS. See The automator for the full picture.

ComposeAutomator.inProcess() already uses RobotDriver.synthetic(). Pin a window or opt into real OS input via the factory:

import dev.sebastiano.spectre.core.ComposeAutomator
import dev.sebastiano.spectre.core.RobotDriver
val pinned = ComposeAutomator.inProcess(
    robotDriver = RobotDriver.synthetic(rootWindow = composeWindow),
)
val realOs = ComposeAutomator.inProcess(robotDriver = RobotDriver())

Synthetic input is the right default for parallel test JVMs, a machine that also runs unrelated UI work, IntelliJ/Jewel-hosted Compose, and macOS test helpers launched with apple.awt.UIElement=true to avoid a Dock icon. That macOS mode is safe for per-character typeText with RobotDriver.synthetic(rootWindow = ...), but clipboard-backed pasteText still requires a foreground-capable app (apple.awt.UIElement=false). Opt into real OS input for end-to-end smokes where the realism of the input matters (e.g., validating that a system shortcut reaches the app). See Running on CI for the macOS CI trade-off.