Recording limitations

Two recording modes

Before reading the rest of this page, it's worth being explicit about what "recording" means here. The recording module exposes two structurally different ways of producing a video file, and each has its own limitations:

• Region capture — records a fixed Rectangle of the virtual desktop, frame by frame. The source of pixels is the OS framebuffer (or the platform's nearest equivalent — ScreenCaptureKit on macOS, Windows Graphics Capture on Windows, GStreamer ximagesrc on Linux Xorg/Xvfb, or the Wayland compositor's PipeWire stream on Linux Wayland). Whatever is showing on screen inside that rectangle goes into the file, regardless of which window is there. Region capture is the only path available when the Compose surface has no adaptable top-level OS window, such as a ComposePanel inside an IntelliJ tool window.
• Window-targeted capture — captures a specific OS window's pixels directly, not the screen rectangle the window happens to occupy. The source is the window's own backing store (ScreenCaptureKit on macOS, Windows Graphics Capture on Windows) or a compositor-provided window stream on Wayland. Because the source is the window itself, movement, occlusion, and off-screen position don't break the recording. This path is the right choice when you have a top-level ComposeWindow.

Backend → mode mapping:

Backend	Mode
FfmpegRecorder	Deprecated legacy explicit region capture (avfoundation/gdigrab/x11grab).
FfmpegRegionScreenshotter	Deprecated legacy explicit Linux X11 still-image region fallback (x11grab).
LinuxX11Recorder	Linux Xorg/Xvfb region and named-window capture via GStreamer ximagesrc.
LinuxNativeScreenshotter	Linux Xorg/Xvfb screenshots via ximagesrc, and Linux Wayland screenshots via portal/PipeWire one-frame capture.
WaylandPortalRecorder	Region capture, sourced from the Wayland portal (SourceType.MONITOR).
WaylandPortalWindowRecorder	Window-targeted (Linux Wayland, portal SourceType.WINDOW — only the picked window's pixels are captured).
ScreenCaptureKitRecorder	Region and window-targeted capture (macOS).
ScreenCaptureKitScreenshotter	Window-targeted still screenshots (macOS).
WindowsGraphicsCaptureRecorder	Window-targeted and region capture (Windows Graphics Capture).
FfmpegWindowRecorder	Deprecated legacy window-targeted (Windows, gdigrab title=).
WindowsWindowScreenshotter	Window-targeted still screenshots (Windows Graphics Capture helper).
AutoRecorder	Routes to the right one based on TitledWindow?.
AutoScreenshotter	Routes still screenshots by platform and window.

The rest of this document is mostly about region capture's constraints, because that's where the failure modes are. The window-targeted backends exist to sidestep those failure modes, and the section below (Window-targeted capture) explains how they do it.

Wayland restore_token (agent non-interactive runs)

CI and hermetic tests can redirect token storage with:

• SPECTRE_WAYLAND_RESTORE_TOKEN_DIR — directory for per-source token files (mode 0700). Files are named wayland-screencast-restore-token-{key} (e.g. …-monitor-embedded).
• SPECTRE_WAYLAND_RESTORE_TOKEN_PATH — base path (not a single flat file). Spectre writes {basename}-{key} next to that path (mode 0600), so keys stay separate. Prefer DIR when you only need to relocate the directory.

Headless runners without a seated GNOME session will not expose org.freedesktop.portal.ScreenCast; unit tests cover persist/reuse/clear without a live portal. Interactive first-consent still requires a real compositor seat.

On Linux Wayland, the ScreenCast portal consent dialog is the analogue of macOS TCC: interactive and not automatable. Spectre's spectre-wayland-helper uses portal persist_mode=persistent and stores a restore_token under $XDG_STATE_HOME/spectre/wayland-screencast-restore-token-{key} (mode 0600, directory 0700; override with SPECTRE_WAYLAND_RESTORE_TOKEN_PATH or SPECTRE_WAYLAND_RESTORE_TOKEN_DIR).

• First run (no token, or compositor rejects a stale token): the portal dialog is shown. A human must accept once.
• Later CLI/agent sessions: the helper reuses the stored token so no dialog appears (validated on GNOME/mutter; other compositors are best-effort).
• Invalidation: if Start/SelectSources rejects the token, Spectre clears the file and retries interactively once, printing a clear message on stderr.

Platform

• macOS — ScreenCaptureKit region and window capture through the spectre-recording-macos helper. Requires the Screen Recording permission. AutoRecorder no longer falls back to FfmpegRecorder/avfoundation when that helper artifact is absent or when callers set SCK-incompatible options such as a custom RecordingOptions.codec. RecordingOptions.screenIndex is the SCK display index: primary display first, then by display frame minX and minY. The explicit FfmpegRecorder remains available only as a deprecated legacy backend.
• Windows — native window recording, region/fullscreen recording, and still window screenshots use the shared .NET helper from spectre-recording-windows. The helper uses Windows Graphics Capture, is published for x64 and arm64, and does not require ffmpeg. It is framework-dependent, so Windows users need .NET 8 Desktop Runtime and Windows App Runtime 1.8 installed. AutoRecorder.startRegion(...) no longer falls back to FfmpegRecorder/gdigrab when that helper artifact is absent, or when the caller sets Windows region options that WGC cannot honour (RecordingOptions.codec or screenIndex); those cases fail loudly instead. FfmpegRecorder remains available only as a deprecated explicit legacy backend.
• Linux Xorg/Xvfb sessions — helper-driven GStreamer ximagesrc capture. Reads DISPLAY. LinuxX11Recorder records fixed regions or a named X11 window; LinuxNativeScreenshotter writes one-frame PNGs through the same helper. Routine validation has only been on Ubuntu 22.04's Xorg session (one machine, one X server build) and on CI under xvfb-run (Xorg protocol over a virtual framebuffer, no GPU). Other Xorg WMs/distros fall under the "Linux is best-effort, contributions welcome" line in the README. AutoRecorder and AutoScreenshotter no longer use ffmpeg for this route; install spectre-recording-linux and the GStreamer packages instead. The helper's recording pipeline currently supports only RecordingOptions.codec = "libx264" or "x264enc"; unsupported codec strings fail at start(...) with a targeted error rather than building a broken GStreamer pipeline. The explicit ffmpeg classes remain available only as deprecated compatibility escape hatches.

Xvfb validation proves Linux X11 capture only — not Wayland portal/PipeWire consent, restore-token reuse, or compositor-specific portal behaviour. CI and release-smoke under xvfb-run exercise the X11 helper path (ximagesrc). Keep a separate real Wayland desktop run when claiming portal capture.

Session detection (#397): residual wayland-* sockets under XDG_RUNTIME_DIR (or inherited WAYLAND_DISPLAY / XDG_SESSION_TYPE=wayland from a login session) must not hijack an active pure-X11 DISPLAY such as Xvfb. Spectre prefers a pure-X11 display probe (Xvfb process match or xdpyinfo without the XWAYLAND extension) and only treats a residual compositor socket as Wayland when DISPLAY is unset (SSH into a Wayland host). Real Wayland+XWayland still routes to the portal. Override with SPECTRE_CAPTURE_BACKEND=x11 or wayland when probes cannot decide.

Native Wayland sessions with XWayland are different: Mutter does not expose the composited desktop through the XWayland root framebuffer, so root-region ximagesrc capture can be black except for the cursor. AutoRecorder detects Wayland before this backend and routes region capture through the portal; do not force the X11 backend in that environment (SPECTRE_CAPTURE_BACKEND=x11 is an escape hatch for nested Xvfb only). - Linux Wayland sessions — gst-launch-1.0 driven through the xdg-desktop-portal ScreenCast interface, with the PipeWire FD passed to the encoder by a small Rust helper binary from spectre-recording-linux (spectre-wayland-helper, sources at recording/native/linux/). Two JVM-side recorders share the helper: WaylandPortalRecorder (region-targeted, portal SourceType.MONITOR — user picks a monitor, helper crops to the requested rectangle) and WaylandPortalWindowRecorder (window-targeted, portal SourceType.WINDOW — user picks a specific window, the granted PipeWire stream contains only that window's pixels, the helper crops to the window's pixel size, #85). LinuxNativeScreenshotter uses the same helper for one-frame portal/PipeWire PNG capture. All of these paths spawn the helper and talk to it over stdin/stdout via newline-delimited JSON. First call within a session pops the compositor's "share your screen" dialog (asking for a window or a monitor depending on which recorder is running); subsequent calls reuse the grant via the portal's restore_token. Why the helper: a pure-JVM attempt hit a dbus-java UnixFD-unmarshalling bug that wasn't fixable trivially, and Rust's std::process makes FD inheritance into gst-launch a one-liner where the JVM's ProcessBuilder doesn't expose the necessary fcntl(F_SETFD, ...) knob. The helper-as-subprocess shape also matches the spectre-recording-macos SCK helper (recording/native/macos/) on the JVM side.

Window-targeted Wayland needs xprop on PATH and GTK/Mutter-style frame extents. Mutter renders window-source-type streams with the WM-imposed invisible-shadow extents around the visible window — typically 25 px on each side for GTK CSD windows on GNOME — but AWT's Frame.getBounds() reports only the inner window. WaylandPortalWindowRecorder queries the X11 _GTK_FRAME_EXTENTS(CARDINAL) property on the JFrame's XWayland window via the system xprop binary to compute the right stream-relative crop; without it the close-button icon ends up clipped off the title bar. If xprop is not on PATH (it's part of x11-utils on Debian/Ubuntu — installed by default on the desktop image, separate package on minimal/server images), the WM doesn't publish _GTK_FRAME_EXTENTS (older Mutter, non-GTK CSD, server-side decorations like KDE Plasma's default), or the call to WaylandPortalWindowRecorder.start is given a TitledWindow with a null/blank title, the recorder throws IllegalStateException rather than producing a silently-misaligned recording. The fallback is explicit region capture: call AutoRecorder.startRegion(...), or instantiate WaylandPortalRecorder directly. Installing xprop only fixes the missing-binary case; it will not make Qt, JavaFX, Electron, SDL, KDE, or wlroots windows publish the GTK-specific property.

Wayland still screenshots use the same portal grant model as video. They are supported only when the compositor's ScreenCast portal can provide the requested monitor or window stream and the user accepts the dialog. In unattended SSH/headless runs, a hidden or unanswered portal dialog surfaces as a helper timeout rather than a silent blank PNG.

Validated end-to-end on Ubuntu 22.04/GNOME 42/mutter, Ubuntu 24.04/GNOME 46/mutter, and Ubuntu 26.04/GNOME 50/mutter (real-pixel mp4 with the smoke runner, 2026-05-02 and 2026-05-03 — the 24.04 run also confirmed cursor_mode=Embedded composites the system cursor into the captured frames when RecordingOptions.captureCursor=true (#87), and the 26.04 run confirmed window-source-type cropping produces a window-sized mp4 with no leakage from other apps and all WM decorations visible (#85)). KDE/Plasma, sway, wlroots-based compositors, non-Ubuntu distros, and other Ubuntu versions aren't part of the routine validation matrix yet — the xdg-desktop-portal interface is standardised across compositors but the _GTK_FRAME_EXTENTS query is GNOME/Mutter-specific, so window-targeted recording on KDE / sway will likely throw the "Could not determine WM frame extents" error and the caller should fall through to WaylandPortalRecorder until we wire compositor-specific frame-extent lookups. See the README for the contribution invite.

Frame-rate fidelity tracks what the compositor delivers. The pipeline runs the PipeWire stream through a videorate element clamped to RecordingOptions.frameRate (default 30). When the source delivers fewer frames than the target, videorate pads the gaps by duplicating the last frame; when it delivers more, the excess gets dropped. On real hardware with GPU-side compositor composition this is a no-op — the source comfortably sustains 30 fps and the output is byte-clean. On a Hyper-V/VirtualBox VM with a software-rendered virtual GPU, the source rate dips into the 5–25 fps range and the output mp4 contains visible duplicate-frame runs even though the file metadata reports a flat 30 fps. This is faithful capture, not a recording bug — the compositor genuinely had no new frame to deliver during those gaps. If your scenario captures from a VM and the visible stutter matters more than the frame-rate metadata, lower RecordingOptions.frameRate to match what your VM actually produces (15 is usually a safe floor for software-rendered VM GPUs).

Region capture

• Region capture is the last resort, not the default for embedded surfaces. Region capture records a fixed Rectangle of the virtual desktop — whatever pixels the screen happens to be showing inside that region land in the file. The region is bound at Recorder.start(...) time and does not follow a window. Prefer window-targeted capture (ScreenCaptureKitRecorder / WindowsGraphicsCaptureRecorder) when you have a top-level window, and window+crop when only an embedded surface lacks its own native handle.
• Embedded ComposePanel surfaces: prefer host window + crop over region capture. When the panel lives inside a top-level host Frame/JFrame (IntelliJ tool window, Swing host, etc.), use AutoRecorder.startWindow(window, output, cropInWindow = surfaceBoundsInWindow, scaleX = …, scaleY = …). That captures the host window's pixels and crops to the panel (macOS sourceRect, Windows pre-encode crop). Benefits: occlusion-immune (overlapping apps are not recorded) and the host window still moves with the OS compositor. v1 crop is fixed at start — if the panel is moved/resized mid-recording, frames may misalign until you stop and restart. A stderr warning is printed when crop is used. Linux X11/Wayland do not implement window+crop yet (evaluated for parity; not wired). startRegion(...) remains the explicit last resort when no host window handle exists. Practical consequences of region fallback:
• Anything that visually overlaps the panel — other windows, the menu bar, OS notifications, a floating popup that escapes the panel's bounds — appears in the recording.
• The captured region is the panel's screen-space bounds at start. If the host window moves or resizes while recording, the panel's pixels drift out of the captured rectangle and you record whatever is now under the original rectangle (often empty desktop).
• Off-screen panels record black frames.

Window movement and popups under region capture

These limitations are specific to the region path; the window-targeted backends below solve them.

• Window movement isn't followed. Move the host window after startRegion(...) and the recording keeps capturing the original screen rectangle. Stop and restart to follow the new position.
• Popups that escape the captured region are clipped. Compose Desktop's OnWindow popup layer renders in a separate top-level OS window; if that popup appears outside the recorded rectangle the recording does not include it. OnSameCanvas and OnComponent popups stay inside the panel / window bounds and are recorded as long as the panel itself is.

Window-targeted capture

ScreenCaptureKitRecorder (macOS) and WindowsGraphicsCaptureRecorder (Windows Graphics Capture) target a specific OS window rather than a screen rectangle, which removes the region-capture failure modes described above:

• Window movement is followed automatically. Drag the window across the screen while recording and the file keeps capturing its pixels — no need to stop and restart.
• Occlusion doesn't matter. Other windows passing in front of the target window do not appear in the recording. The capture reads the target's own backing store, not the composited screen.
• Off-screen target keeps recording. A window dragged off the visible desktop still has its pixels composed, so capture continues. (Compare the region path, which would record whatever's now under the original rectangle.)
• OnWindow popups are not captured by the target's recorder, because the popup is a separate OS window with its own title — it isn't the window being targeted. OnSameCanvas and OnComponent popups are part of the target window's surface and are recorded normally.

WaylandPortalRecorder is structurally a third path: the portal hands the capture a PipeWire stream from the compositor, scoped to a user-picked monitor, which then gets cropped to the requested region. The monitor-level source means it doesn't follow a specific window the way SCK and WGC do — moving the target window within the same monitor stays within the captured stream, but moving it off-monitor or to another display does not. Treat its movement-and-popup behaviour as closer to region capture than to window-targeted, with the bonus that the source is the compositor and not raw framebuffer reads.

HiDPI/Retina

• The recorded resolution is the screen-pixel size of the region, not the dp size. A 400×300dp region on a 2× display becomes an 800×600 pixel video.
• Rectangle coordinates passed to start(...) are in screen pixels. If you derive the region from AutomatorNode.boundsOnScreen you get the right thing for free; if you derive it from boundsInWindow you have to apply the density yourself.

Permissions and process lifecycle

• The JVM process needs macOS Screen Recording permission (MacOsRecordingPermissions documents the path). Without it, the SCK helper exits during startup and Recorder.start surfaces an error rather than handing back a handle.
• The macOS console session must be unlocked. A locked screen can make Robot screenshots or ScreenCaptureKit recordings fail, or produce black/incorrect frames, even when Screen Recording TCC is granted. Unlock the display and retry before treating the failure as a missing permission.
• The underlying encoder process/helper is spawned eagerly: by the time start(...) returns, frames are landing in the output file. A failure to spawn (binary/helper not available, codec unavailable, permission/device busy) surfaces as an exception from start(...), not as a silent "successful" handle.
• The handle MUST be stopped (RecordingHandle.stop(...)) for the file to be flushed cleanly. A JVM exit without stop leaves a partial/non-finalised file and an orphaned subprocess.

JUnit failure-video policy (#206)

ComposeAutomatorExtension / ComposeAutomatorRule can record the whole test and keep or delete the file at the end (FailureVideoPolicy: Off | OnFailureKeep | Always). See JUnit integration — Failure video.

Honest overhead notes (policy defaults to Off for a reason):

• Per-OS cost — the same backends as AutoRecorder run for the full test duration (ScreenCaptureKit helper on macOS, Windows Graphics Capture helper on Windows, GStreamer X11 / Wayland portal on Linux). CPU, encoder, and helper-process cost scale with test length and resolution; capability and permission limits on this page still apply.
• Disk under OnFailureKeep — the video file is written during every invocation (including green tests) and only deleted after stop+finalize on pass. Parallel suites and long-running tests can therefore spike disk use during the run even when nothing remains after a green suite.
• Stop before delete — Spectre always stops the recorder and finalizes the file before applying keep/delete. A JVM exit without teardown can still leave a partial file and an orphaned helper (same as any other RecordingHandle — see process lifecycle above).
• In-process JUnit only — this policy does not cover agent/attach / CLI spectre record paths; those remain separate entry points.

Current non-limitations

• Cursor capture is configurable via RecordingOptions.captureCursor and works under the region path. The cursor pixels are baked into the frames; there is no overlay you can toggle in post-processing.
• Audio capture is intentionally absent — Spectre currently records video only.

Window-targeted recording

• Window-targeted capture via ScreenCaptureKitRecorder is the recommended path for top-level ComposeWindow surfaces on macOS. It removes the "anything overlapping the region appears in the video" class of problems and follows the window across the screen. Windows has the equivalent path via WindowsGraphicsCaptureRecorder.
• The embedded-panel path is still region capture only — there's no host window for SCK or WGC to target.