ffmpeg-skill execution contract

ffmpeg-skill contract --json (or python3 scripts/_contract.py --json) prints a machine-readable description of this skill: which tools exist, what each one needs, takes and writes, how its result is verified, and what an agent may assume about dry-run, input preservation and repeatability. It is the interface a planning agent consumes instead of reading SKILL.md, which stays written for a coding agent that follows the workflow by hand.

The contract is derived from the code that runs, not maintained beside it:

• the tool list is every script in scripts/ that does not start with _;
• every input_schema is generated from the script’s own argparse parser at the moment the contract is printed, so a new flag appears in the contract with no other edit;
• the facts a parser cannot express (role, required ffmpeg components, verification policy, visual-check policy) live in one table in scripts/_contract.py and are checked against the scripts, the MCP server and the installer by tests/test_contract.py.

Versions

A release that adds a tool or a flag keeps contract_version; a breaking change to the ToolSpec shape bumps it. Consumers pin on contract_version and read skill.version for provenance. Consumers should also pin ffmpeg-skill itself by npm version or git tag, not by tracking main — see README, “Development”, “Releasing”.

The prose tool count in this file, README, SKILL.md and package.json’s description is not generated (it reads naturally in a sentence), so tests/test_contract.py’s test_docs_tool_count_matches_the_real_tool_list checks all five against the real count from scripts/ on every CI run instead — a stale count fails a test rather than drifting silently. (SKILL.md was added to that check after its “the 28 scripts” sat stale through twelve tool additions while the other three files were correct.)

Stability guarantee (2.x)

1.0.0 was published on 2026-09-11 (by accident: see CHANGELOG.md’s 1.0.0 entry; the number is kept rather than burned). From 1.0.3 on, the number is treated as the promise it implies; 2.0.0 (2026-09-22) is the first deliberate break, listed under “What 2.0 changed” below. The same promise holds for the whole of 2.x:

Not covered: the exact wording of --help text, descriptions, stderr messages, and the details/notes free-text fields of JSON output; the internals under scripts/_*.py; the evals harness; the development skills under .claude/.

The promise is enforced, not remembered: tests/test_contract.py’s test_mcp_tool_surface_matches_the_frozen_1x_snapshot pins every tool’s argument names and which are required against tests/fixtures/mcp_tools.json. A removal, rename or newly required argument fails CI; an addition fails until the snapshot is regenerated (UPDATE_MCP_SNAPSHOT=1 python3 tests/test_contract.py), so the diff of the fixture shows a reviewer exactly what grew.

Deprecation policy

Something that has to go (an argument superseded by a better one, an output key that turned out to be misleading) is retired in three steps, never in one:

1. Deprecate in a minor release: the old form keeps working unchanged, a one-line warning naming the replacement is printed to stderr when it is used, the CHANGELOG entry says “deprecated”, and --help marks it (deprecated: use ...).
2. Keep it for at least two further minor releases or 90 days, whichever is longer.
3. Remove it only in the next major, listed in that release’s CHANGELOG under “Removed”, together with the version that first deprecated it.

A defect fix that changes behaviour is not a deprecation: it ships in a patch with a CHANGELOG line, and if the old behaviour was something a caller could reasonably have relied on, the line says so.

contract --json carries a top-level deprecated list, next to contract_version: one entry per thing the next major removes, {"what", "since", "replacement", "removed_in", "where"} with where naming the surface (cli, json, mcp, behaviour). It was empty in 2.0.0. Since 2.10.9 it lists one entry: running on Python 3.9 (where: behaviour, removed_in: 3.0.0). CPython 3.9 reached end of life in October 2025 and gets no security fixes, but raising the floor is a break, so 2.x keeps running on 3.9 (requirements.python stays >=3.9, and CI keeps testing it). It is deprecated, and Python 3.10+ is recommended: on 3.9, doctor and the npx installer print one stderr line, warning: Python 3.9 is deprecated since ffmpeg-skill 2.10.9 (...); it still works in 2.x and support ends in 3.0.0: use Python 3.10+. Like every deprecation warning it is stderr only: doctor --json’s document has no key for it. Since 2.14.0 there is a second entry, running on FFmpeg 5.x (where: behaviour, removed_in: 3.0.0), handled the same way: requirements.ffmpeg stays >=5.0 and CI keeps its 5.1.1 job until 3.0, which requires 6.1. When ffmpeg -version reads 5.x, doctor and the npx installer print one stderr line, warning: FFmpeg 5.x is deprecated since ffmpeg-skill 2.14.0 (...); it still works in 2.x and support ends in 3.0.0: use FFmpeg 6.1+. A version that is not a release number (a git snapshot) gets no line. tests/test_contract.py refuses a package.json major bump that still lists an entry that major should remove, or that comes before step 2’s window has passed – unless the waiver is written here, as 2.0.0’s is.

2.0.0 waived the 90-day half of step 2. Its deprecations date from 1.10.0 (2026-09-13), so the policy’s earliest 2.0.0 was 2026-12-12; on 2026-09-22 the maintainer decided not to wait. The other half held with room to spare: sixteen minor releases (1.11-1.26) carried every one of them, each already had a 1.x opt-in to the 2.0 behaviour (--quality, FFMPEG_SKILL_MCP_LEAN=1, hdr_signal, FFMPEG_SKILL_NO_OVERWRITE=1), and the one change a caller could not have prepared for mechanically – promoting result_v2 – was withdrawn rather than shipped early (below).

What 2.0 changed

contract --json’s removed list is the machine-readable form of this table, one entry per change with the version that deprecated it.

The v1 success keys were deprecated in 1.10.0 in favour of result_v2. 2.0 withdrew that deprecation instead: result_v2 decided whether a key landed in metrics or in details by its value (a number went to metrics, the same key holding null went to details), so a caller could not know where to look without knowing the value first, and every consumer of every tool’s JSON would have been rewritten for that. The flat keys, typed per tool in output_schema, stay.

Skill

{
  "contract_version": "1.0",
  "deprecated": [{"what": "running on Python 3.9 (the requirements.python floor)", "since": "2.10.9", "replacement": "Python 3.10 or later; ...", "removed_in": "3.0.0", "where": "behaviour"},
                 {"what": "running on FFmpeg 5.x (the requirements.ffmpeg floor)", "since": "2.14.0", "replacement": "FFmpeg 6.1 or later; ...", "removed_in": "3.0.0", "where": "behaviour"}],
  "removed": [{"what": "...", "since": "1.10.0", "replacement": "...", "removed_in": "2.0.0", "where": "cli | json | mcp | behaviour"}],
  "skill": {"id": "ffmpeg-skill", "version": "2.17.1", "execution_mode": "local", "kind": "execution",
            "entrypoints": {"cli": "...", "mcp": "...", "contract": "...", "doctor": "..."},
            "not_provided": ["AI reasoning", "decisions", "production plans", "project IR", "approvals", "network access", "transcription engine"]},
  "requirements": {"python": ">=3.9 (standard library only; 3.9 deprecated since 2.10.9, removed in 3.0.0; 3.10+ recommended)", "ffmpeg": ">=5.0 (5.x deprecated since 2.14.0, removed in 3.0.0; 6.1+ recommended)", "ffprobe": ">=5.0"},
  "execution": {"shell": false, "arbitrary_executables": false, "network": false,
                "model_downloads": {"when": "...", "in_process": {"faster-whisper": "..."},
                                    "child_process": {"parakeet-mlx": "...", "openai-whisper": "..."},
                                    "none": ["whisper.cpp", "parakeet.cpp"], "offline": "..."},
                "input_mutation": false}
}

ffmpeg-skill is an execution skill. It measures, transforms and verifies media with local FFmpeg. It does not reason, plan, decide, or hold a project model; those belong to the agent that calls it.

ToolSpec

One entry per tool under tools, sorted by id. Tool ids are stable: ffmpeg-skill/<script name> (ffmpeg-skill/cut, ffmpeg-skill/loudness, …).

Roles

Verification policy

The workflow in SKILL.md is “probe first, verify last”. The contract states it per tool:

requires_visual_verification is true exactly for the tools that change the picture (fit, caption, overlay, graphics, color, join, multicam, render). Audio-only tools and audio-only inputs never need look; the report line is Look: not needed. check rows carry kind: format (fix it) or kind: judgement (decide with the user). The audio row FAILs (WARN when no loudness target applies) when the track is present but silent (peak at or below -50 dBFS, from the loudness pass or, when loudness is skipped, one volumedetect pass). check --content (opt-in; render.py "check": {"content": true}) adds black, frozen and silence rows from one decode pass, and dropouts (2.17) from one audio-only pass; without it the row set is unchanged. Thresholds: docs/design-decisions.md (“check.py –content”).

Dry run

supports_dry_run is measured, not declared: tests/test_contract.py runs every tool with --dry-run behind a fake ffmpeg that records any call, and asserts that no call happened and no file appeared. Under --dry-run a tool prints the command lines it would run, reports dry_run: true, and never reports an output probe. The exceptions are stated per tool in the contract’s dry_run field: probe and check are read-only (ffprobe still runs); sync, multicam, scenes, cropdetect, report, silence, loudness and stabilize still run their ffmpeg/ffprobe measurements (the analysis is the tool’s job; only the artifact is skipped, including side files such as --edl, --sheet or a generated .ass), overlay measures the lines of a multi-line --text with one drawtext on a generated frame on FFmpeg 8.0, 8.0.1 and 8.1.0 (it draws them one by one there), and verify does not support dry-run (its steps run). SKILL.md and references/scripts.md repeat the same list; the contract is the authority.

--codec h264|hevc|av1|prores and --quality N (1.8) are on every tool that re-encodes, marked common. They are resolved in one place (_common.encoder_args()): hevc keeps an HDR source Main10 with its tags and writes 8-bit BT.709 for SDR; av1 is SVT-AV1 with libaom as the fallback; prores is 422 HQ and needs a .mov/.mkv output; h264 refuses an HDR source (kind: input). --quality is the CRF scale; its default is the tool’s own (18, proxy.py 30). 1.x also accepted --crf as an alias; 2.0 removed it (export.py keeps its own --crf). Without --codec the encoder is what it always was (x264 for SDR, x265 Main10 for HDR), so the flags add no behaviour to a caller that does not pass them. Every SDR line except prores is tagged BT.709; a source whose matrix is BT.601 (smpte170m, bt470bg, or fcc/smpte240m) is converted to it in the filter graph (colorspace, matrix only; _common.bt709_matrix_filter()), so the label and the pixels agree. An untagged or BT.709 source adds no filter, and the tags themselves never convert (bt709_tag_args()). An RGB picture (a PNG or JPEG still, a PNG sequence, a qtrle clip, an overlay --image logo, an emoji PNG, waveform --image) is converted to BT.709 YUV explicitly (scale=out_color_matrix=bt709; a JPEG is a bt470bg source); grid.py converts each cell by its own tags. Where a converted picture meets an untagged one in a mixing filter, the untagged one is labelled BT.709 (setparams, tags only), so FFmpeg 7.1+’s colourspace negotiation converts neither (join, multicam, broll and overlay --video too, beside a converted clip). A colour a tool draws from a hex or named value (a pad, tpad or rotate fill, drawtext and libass text and boxes, a graphics.py box or bar, background.py’s color/gradients, waveform.py’s plate and waves) lands in the same BT.709: from FFmpeg 7.1 an untagged picture is labelled for the drawing filters, before 7.1 the drawing runs on an RGB copy and only the pixels it touched are taken from it (_common.drawn_chain()); a grey changes nothing. cut.py --keep-hevc (opt-in) re-encodes an SDR HEVC source as x265 8-bit BT.709 instead of x264; an HDR or BT.2020 source keeps the Main10 line with its own colour tags, with or without the flag. The encoder each value needs is listed under the tool’s optional capabilities (--codec hevc and so on). export.py refuses --codec: its presets decide the codec.

--hw / --no-hw are on the same tools and on export.py, marked common; render.py and batch.py take them for every stage they run. They are two booleans in the input schema (hw, no_hw; MCP sends --hw for hw: true and --no-hw for no_hw: true), mutually exclusive. --hw moves an h264/hevc/prores encode to Apple VideoToolbox (Apple Silicon only; AV1 stays on the CPU); --no-hw keeps it on the CPU even when FFMPEG_SKILL_HW=1 makes the GPU this machine’s default. Neither changes anything for a caller that passes neither on a machine without the variable, with one exception: on a build whose ffmpeg -encoders has no libx264 (or no libx265) but has the matching VideoToolbox encoder, on Apple Silicon, an H.264 (or HEVC) encode takes VideoToolbox without --hw – export.py’s presets too, which FFMPEG_SKILL_HW never moves – because there is no CPU encoder to choose instead; --no-hw still asks for the CPU and gets what ffmpeg says about it. What ran is in the result’s encoder and hw (see the keys added after 2.2.2, and doctor’s gpu_encoders and covered_by below).

--plan FILE (1.6) is a dry run that also writes a plan document: {"plan_version": 1, "tool", "argv", "cwd", "inputs": [{"path", "size", "sha256_head_tail"}], "commands", "unbound_inputs", "output", "verify": [{"tool": "probe"}, {"tool": "check", "platform"}], "notes"}. It implies --dry-run, so the same execution rules apply; inputs covers the -i files of the planned commands, every existing file named in argv (a recipe, a still, an SRT) and the subtitle/LUT/font files a filter reads; unbound_inputs (absolute paths, additive to version 1) names the inputs that did not exist yet when the plan was made (an earlier step’s output), which therefore carry no fingerprint. Tools that print their document without --json (probe, the analysis tools) still write the plan at exit; verify.py and render.py refuse --plan (their steps run for real; a project file is already a plan). render.py FILE executes a plan: it refuses (kind: input, error.reason: plan_inputs_changed) when an input’s size or head/tail hash differs from the plan, with every input’s {path, status, expected, actual} in error.details.inputs (actual is "missing" for a file that is gone) beside plan_sha256 and checked_at, and appends one record to the audit log (below); a wrong plan_version, an unknown tool, a vanished cwd or a malformed argv is refused the same way under its own reason. It then runs the tool with the planned argv, then the verify steps (a check step may also carry "content": true, added by hand, for check.py --content), and reports plan, plan_sha256, tool, tool_result, check and unbound_inputs (each one also a diagnostics.warnings entry: nothing checked that file). plan_version is bumped when the document’s shape changes.

Repeatability

No tool keeps state or uses randomness. deterministic_inputs is false only for verify and cropdetect, whose output includes timings or machine state. idempotency_hint says what “same inputs” gives you:

The hints describe the CPU encoders. Under --hw, or FFMPEG_SKILL_HW=1 on the machine, a tool that encodes video runs VideoToolbox, whose bytes are not promised identical from run to run: treat such a result as content_equivalent at best. That includes background, whose hint stays bit_exact within 2.x. The variable belongs to the machine, so the same flags can give x264 bytes on one machine and VideoToolbox bytes on another; encoder in the result says which.

provides

provides lists these 42 tools by a cross-repository Capability id, for kajisho5/AI-video-production-OS’s CapabilityContract.provides (docs/SPEC.md there), matching the ids already assigned to this Skill in that project’s own docs/CAPABILITY_MATRIX.md section 9 (“ffmpeg-skill’s 21 raw tools … are Capabilities in their own right, independent of the higher-level Skills that delegate to them”): [{"id": "ffmpeg-skill.<tool>", "lifecycle": "EXPERIMENTAL", "tool_id": "ffmpeg-skill/<tool>"}, ...], one entry per tool, sorted by id. The Capability id uses a dot (ffmpeg-skill.cut) - the <domain>.<verb> shape every other Skill’s Capability ids use elsewhere in that project (video.trim, audio.gain, …), with ffmpeg-skill as the domain - while tool_id carries this contract’s own slash-shaped id (ffmpeg-skill/cut) unchanged. It is purely additive: derived from public_tools(), saying nothing tools[] doesn’t already say, only indexed by Capability id instead of tool name.

capability_map

provides re-indexes each tool by an id shaped like the tool name (ffmpeg-skill.cut); it doesn’t tell a caller that “I need to trim a video” resolves to cut. capability_map is the small, hand-authored table that closes that gap: [{"capability": "<domain>.<verb>", "tool_id": "ffmpeg-skill/<tool>", "params": {...}}, ...]. A planner that only knows an abstract goal (video.trim, audio.loudness, subtitle.burn, media.stream.inspect, media.frames.extract, media.proxy) looks it up here to find the tool, then builds and runs that tool’s own call from its input_schema exactly as it would have if it already knew the tool name - capability_map never executes anything itself, and this skill never picks a capability on the caller’s behalf.

Some entries also fix one or more params where the capability names a specific behaviour narrower than the whole tool: video.reframe maps to fit with params: {"fit": "crop"}, because fit.py also does duration-fit and letterbox padding, and only the crop mode is a “reframe”. A caller resolving video.reframe should treat those params as fixed inputs to that tool’s own schema, not as optional defaults.

This list is deliberately short and will stay short: a capability is only added when resolving it is a mechanical, no-judgment lookup. There is no video.highlight entry, for instance, because scenes.py --highlights ranks candidates by a measured proxy (audio energy or duration), never by understood content - offering it as a blindly-delegable capability would misrepresent what it does (see SKILL.md, “What this skill does and does not decide”). media.proxy (a low-bitrate, fast-decode proxy for downstream analysis/preview, distinct from export.py’s delivery presets) resolves to proxy - itself a mechanical resize + re-encode with no opinion on which asset should be proxied or what for.

Delivery table and templates (1.14)

scripts/_platforms.py is the one table every delivery tool reads. Per destination (tiktok, reels, shorts, youtube, youtube-hdr, youtube-av1, x, linkedin, facebook, podcast, plus the broadcast / custom compliance targets):

It is an internal module (leading underscore), not a tool: the public tool count is unchanged. check.py’s SPECS, export.py’s PRESETS (each platform preset’s frame and duration cap) and export.py’s PLATFORM_OF are all built from it, so the loudness export.py --normalize targets, the frame it writes, the cap it trims at and the spec check.py enforces are one value. Two presets deliberately differ from their destination’s row and say so in the code: youtube4k delivers to YouTube at 2160p, and no youtube* preset trims at YouTube’s 12-hour limit (check.py reports it instead). _platforms.resolve() is the one alias map – youtube-shorts/yt-shorts = shorts, yt = youtube, instagram/ig = reels, twitter = x, fb = facebook – and check.py --platform, export.py --preset, caption.py/graphics.py/overlay.py --platform, look.py --safe and render.py --template all accept those spellings.

New in the same release, all additive: export.py --preset tiktok|shorts|linkedin|facebook (real presets, not aliases of reels/youtube), --preset youtube-hdr (HEVC Main10 keeping the source’s HDR tags; kind: input on an SDR source) and --preset youtube-av1 (kind: missing_tool, error.reason: encoder_missing, when the build has neither SVT-AV1 nor libaom; kind: ffmpeg, encoder_list_unreadable, when its encoder list could not be read at all, which says nothing about the build); caption.py --platform and graphics.py --platform / --margin and overlay.py --platform (margins from the safe zone, an explicit --margin/--position wins); look.py --safe NAME; fit.py --fit blur; report.py --pack; graphics.py --template sticker|hook|meme; and render.py --template NAME INPUT (--cues/--srt/--logo/--title/--brand/--chapters/--fit/-o/--write-project/--list-templates), which fills a templates/<name>.json project shipped with the skill. --template all or a comma-separated list renders every named destination and writes a <stem>_pack.md table. A project may now carry "template" (the name it was filled from) and "frame": {"fit": ...}.

Capabilities

Names: ffmpeg, ffprobe, encoder:<name>, filter:<name>, bsf:<name>, external:whisper, external:parakeet. capabilities.required is the union of every tool’s required list; optional the union of the conditional ones. With detection (the default) available, missing and missing_optional are added from doctor, which reads ffmpeg -encoders / -filters / -bsfs and looks for a local whisper or Parakeet. Pass --static to omit detection. Nothing from the environment other than those lists, the ffmpeg/ffprobe/python versions and the paths of the ffmpeg/ffprobe binaries the tools run (ffmpeg_path, ffprobe_path; a font file’s path under fonts is the other one) is printed; no environment variables (doctor’s hw.default_on is a boolean derived from FFMPEG_SKILL_HW, never its value). The two binary paths can sit under a home directory, so read a pasted doctor before posting it.

filter:delogo is optional on redact (--mode delogo). The filter fills a fixed rectangle from a 1 px ring of the surrounding picture. It is a border blend, not inpainting, and its pixel formats are 8-bit YUV, so an HDR or 10-bit source is refused (color.py --to-sdr first). filter:boxblur stays required on redact, because --mode blur is the default.

external:whisper is optional for two tools since 1.17: caption.py (--transcribe) and silence.py (--filler --transcribe). Neither requires it — both take a transcript the caller already has (--srt/--words), and both refuse with the same install lines when asked to make one with no engine present. Whisper is never a dependency of this skill. external:parakeet is the same kind of optional capability for the same two flags: parakeet-mlx, or parakeet-cli with a .gguf model it can find. Either engine family satisfies --transcribe; --engine auto (the default, or FFMPEG_SKILL_ASR_ENGINE) runs Parakeet for English speech and Whisper for every other language. --model (both tools) is the model; without it FFMPEG_SKILL_ASR_MODEL is this machine’s default, else base. An explicit --model goes to whichever engine runs; a Parakeet model in the variable goes to the Parakeet engines only, and the Whisper engines keep base. doctor reports neither variable (it prints no environment values); the result’s transcription.model names the model that ran. When no --language is given and no detector can name it, a Whisper engine that can run takes it (whisper.cpp is passed -l auto: its own default is English). Parakeet then runs only on English assumed – when no Whisper engine is installed, or every one failed, or --model names an English-only Parakeet model – and the result’s top-level notes says the transcript is wrong if the speech is not English; a multilingual (v3) Parakeet model assumes nothing and gets no note. Every whisper.cpp command that writes a transcript is also passed -mc 0 (no previous-text context, so one hallucinated line is not repeated to the end of the file); its language detection is not. The result’s transcription says which engine, model and routing decision produced the cues (in every --mode, mux included). Every engine transcribes on the machine running the skill, and the audio is never uploaded. The skill’s own code opens no network connection (execution.network: false), but an engine may fetch its model weights on first use: parakeet-mlx downloads its default mlx-community/parakeet-tdt-0.6b-v2 from Hugging Face and openai-whisper its model from OpenAI, each in its own child process. faster-whisper is a Python library the skill runs inside its own process, so its first-run download from Hugging Face is made by the skill’s process; HF_HUB_OFFLINE=1 keeps it (and parakeet-mlx) on the cached copy after that. parakeet.cpp and whisper.cpp read a model file the user downloaded. execution.model_downloads says the same in the contract, engine by engine, so a caller that sandboxes the network can tell before a first --transcribe that faster-whisper’s fetch comes from the skill’s own process.

doctor has three states per capability. available and missing come from a listing that was read; unknown means the listing that would prove the capability could not be read (ffmpeg -filters in a layout the parser does not recognise, or ffmpeg exiting non-zero), and it is never folded into missing, so an installed filter is not reported absent, nor into available, so a failed detection is not a pass. detection gives the status (parsed, unparsed, failed, missing), row count and detail of each listing; errors lists the unreadable ones. filter:drawtext is also rendered once, with the fontfile= the tools pass by default: a crash there makes it missing; a crash only on a bare fontconfig lookup leaves it available and adds a notes entry (--font-file, not a bare --font <family>, on that machine). The filter parser recognises the FFmpeg 6/7 layout (three flag characters, ..C acompressor A->A) and the FFmpeg 8 layout (two, T. acompressor A->A) by the io-spec token, so the flag width does not matter; fixtures for both live in tests/fixtures/.

ffmpeg-skill doctor exits 0 when every required capability is available, 1 when one is missing, 2 when none is missing but a required one is unknown. ok is true only for 0. The keys of 0.9.0 (available, missing, missing_optional, ok) are unchanged.

One capability can be neither: encoder:libx264 (encoder:libx265) absent from the build while h264_videotoolbox (hevc_videotoolbox) is in it, on a machine that can run VideoToolbox (the check --hw uses: macOS on Apple Silicon). The tools encode with VideoToolbox there without --hw, so the capability is left out of available, missing and missing_optional – it is not in the build, and it blocks nothing – and is named in the additive covered_by: {"encoder:libx264": "encoder:h264_videotoolbox"} ({} for every other build, which reports what it always did). ok and the per-tool usable follow what will run, a tool that needs the covered capability lists it in its own covered_by, and notes says its output is not CRF-equivalent. The contract’s capabilities.required/optional lists still name encoder:libx264: they say what the tool needs from a build, not which encoder satisfies it on a given machine.

doctor’s tools field folds that same per-capability state into a per-tool answer: {"<tool>": {"usable": "yes"|"no"|"unknown", "missing": [...], "fix": "...", "unknown": [...], "covered_by": {...}}}. missing/unknown list only that tool’s own required capabilities that are in that state (missing is absent when there is none, same for unknown); fix is a one-line, plain-language remedy for each missing capability, joined with “; “ when there is more than one. This exists so a caller does not have to cross-reference available/missing against each tool’s own required capabilities by hand to answer “can I run caption.py on this machine right now” – doctor passing overall does not mean every tool is usable (a plain Homebrew ffmpeg on macOS is ok for tools that don’t need subtitles/drawtext/zscale, but caption.usable is "no").

doctor’s ffmpeg_path, ffmpeg_source, ffprobe_path and ffprobe_source name the binary the tools actually run (null when none is on PATH). *_source is "path" (what PATH resolves) or "homebrew-ffmpeg-full": on macOS, when the binary PATH resolves is Homebrew’s lean ffmpeg / ffmpeg@N formula (its real path is under <prefix>/Cellar/ffmpeg…) and the same prefix has the keg-only opt/ffmpeg-full/bin/<name>, the tools run that one (#317), because brew install ffmpeg-full does not put it on PATH and the lean build has no drawtext/subtitles/zscale. Nothing else is substituted. ffmpeg and ffprobe (version strings) keep their meaning; the four keys are additive.

doctor’s ffmpeg_security field compares the ffmpeg and ffprobe versions with a small table of high-severity FFmpeg CVEs and the first upstream release on each branch that carries the fix. The table is hand-maintained in scripts/_contract.py (FFMPEG_ADVISORIES), copied from https://ffmpeg.org/security.html and last checked on 2026-10-22; the document repeats both as source and checked. Nothing is fetched: the check is offline, like the rest of the skill. On that date it holds one row:

AISLE’s CVE-2026-64830..64835 were not on security.html on that date, so they have no row yet.

{"source": "https://ffmpeg.org/security.html", "checked": "2026-10-22", "ffmpeg": {"version": "...", "status": "older_than_fix"|"not_older"|"unknown", "advisories": [{"cve", "name", "component", "kind", "cvss", "first_fix", "fix_on_branch", "branch"}]}, "ffprobe": {...}, "warnings": ["..."]}. The leading upstream number is read from the version (8.0.1-3ubuntu2 is 8.0.1). A version on a branch the table lists is compared with that branch’s fixing release; a version on an older branch, which has no fixing release, with the oldest release that has it (fix_on_branch: false). A version with no release number (a git snapshot, N-12345-g…), or no binary, is unknown and never warned about. Each distinct version that is older gets one line in warnings, also printed to stderr (in plain and --json mode). The line says the build is older than the upstream fix and that a distribution package may have backported it without changing the version; it does not say the build is vulnerable, and it names an update command (apt for an Ubuntu/Debian version, else by platform). Like gpu_encoders and fonts, it never affects ok, any tool’s usable or the exit code.

doctor’s gpu_encoders field reports GPU-backed encoders (nvenc, videotoolbox, qsv, vaapi, amf) present in this ffmpeg build, read from -encoders alone — {"status": "parsed"|"unparsed"|"failed"|"missing", "present": [...]}. It proves the build shipped the capability, not that the GPU/driver on this machine will accept a job (that needs a real encode, which this introspection never runs). No tool declares or requires a GPU encoder, so gpu_encoders never affects ok or any tool’s usable (covered_by, above, is the one place a VideoToolbox encoder counts, and only for a libx264/libx265 the build lacks). --hw (and FFMPEG_SKILL_HW=1, which export.py’s delivery presets ignore) puts h264/hevc/prores encodes on VideoToolbox on Apple Silicon only, never by default (a build without libx264/libx265 excepted). A job VideoToolbox refuses (the encoder could not open, or its compression session failed) is re-encoded on the CPU and said so in the result’s hw.fallback/hw.notes; any other failure is not retried. When FFMPEG_SKILL_HW=1 rather than --hw chose the GPU and the GPU ran, hw.notes also says so, with the file-size cost and --no-hw as the remedy for a final deliverable. doctor’s hw field is {platform_ok, default_on} — normalised booleans, never the environment value.

doctor’s fonts field reports whether the default drawtext font (caption.py’s --animate/--karaoke, graphics.py’s templates — BRAND_DEFAULTS["font"], "DejaVu Sans") is actually installed — {"default_font": "...", "status": "available"|"missing"|"unknown", "detail": "..."}. drawtext’s font= is a fontconfig name lookup, and fontconfig silently substitutes the closest match for any name, known or not — a missing font never fails the encode, so drawtext’s own exit code cannot detect it. fc-match is queried instead: available when it resolves the name to itself, missing when it substitutes a different family, unknown when fc-match itself is not on PATH or fails. Like gpu_encoders, this is purely informational and never affects ok or any tool’s usable — a substituted font is not a broken tool, just a typeface the caller didn’t ask for.

Since 1.12 the same field also carries scripts: one entry per writing system the tools detect, {"ja": {"status": "available"|"missing"|"unknown", "file": "/path/to/font.ttc"|null}, "zh": ..., "ko": ..., "ar": ..., "he": ..., "hi": ..., "bn": ..., "ta": ..., "th": ..., "lo": ..., "ru": ..., "el": ...} (bn, ta and lo were added in 1.15). It answers “which languages can this machine actually render”, which no filter or encoder capability asks: available means fc-list :lang=<code> (Linux/macOS) or a known system font file (Windows) covers the script, missing means fontconfig knows none, unknown means there is no working fontconfig to ask (no fc-list on PATH, or it failed). caption.py, graphics.py and overlay.py --text resolve a font by script automatically and fail with kind: input rather than render boxes only for missing: unknown is not missing here any more than anywhere else in this document — the job runs with the font as given and a font_coverage_unverified warning says the coverage could not be verified. So a missing script here is a job that will not run until a font is installed — but, like the default font, it never affects ok or any tool’s usable (the tool works, this machine just has no glyphs for that language). The plain-text doctor summarises the whole map on one fonts: line, which also carries the default font’s detail in brackets when its status is not available and that detail is short enough to keep the line to one screen width; a longer explanation, and the per-script files, are --json only. --lang/--language (caption, graphics) is the hint that says whether Han-only text is Chinese, Japanese or Korean.

Since 1.15 fonts also carries emoji: {"mode": "color"|"png"|"mono"|"none", "color_font": "Noto Color Emoji"|null, "color_font_file": "..."|null, "libass_color": true|false|null, "assets": "/path"|null, "detail": "...", "fix": "..."}. libass_color comes from a render probe — one 64x64 frame with an emoji cue through subtitles=, chroma-tested — because an installed colour emoji family proves nothing: Noto Color Emoji installs cleanly on builds whose libass still draws a monochrome outline. null means the probe was not run: contract --json --static (and every other static/JSON-only path) skips it, exactly as it skips the rest of the environment detection. mode is color when the probe says colour, else png when an emoji assets directory resolves, else mono when some installed face has a glyph, else none. Informational like the rest of fonts: it never moves ok or any tool’s usable.

caption.py, graphics.py and overlay.py gained --emoji auto|color|png|mono|none, --emoji-assets DIR, --emoji-scale FLOAT and --emoji-max N in 1.15; graphics.py also gained --text-render auto|ass|drawtext and --write-ass PATH. New success keys: emoji ({"mode", "count", "clusters", "assets", "missing", "overlays"}) on caption.py and graphics.py, and text_renderer ("ass"|"drawtext"), script and ass (the generated file, when one was written) on graphics.py. --json-brief carries emoji.mode and emoji.count only. All additive: contract_version stays 1.0.

Invocation

Structured arguments are the canonical way to call a tool, on the CLI or through MCP. The mapping is stated in invocation.structured.argument_mapping: positionals in input_schema.positional order, key → --key with _ → -, booleans as bare flags, arrays repeated, output → -o, loudness.lufs → -I. --json is appended for every tool except probe (JSON by default) and look.

The MCP server also accepts {"argv": [...]} for CLI compatibility. That path is marked canonical: false: it is still bound to the named script and never reaches a shell, but an agent ecosystem should use the structured form. No tool, CLI or MCP, runs a shell, evaluates strings, or executes anything other than the named script, ffmpeg and ffprobe.

JSON output

Non-finite numbers never reach a --json document (2.2.6): every tool prints through one writer that turns -inf / inf into the strings "-inf" / "inf" (the spelling loudness.py already used for a silent input) and NaN into null, then serialises with allow_nan=False, so a strict JSON parser reads every document. A silent file’s loudness is therefore "lufs": "-inf", never -Infinity.

audio.py (2.2.6) measures the whole-file peak of every --music / --replace / --effects file and, under --duck, of the voice: at or below --silence-threshold (default -50 dBFS) the track is silent. --on-silent warn (default) mixes it and adds silent: [{flag, path, peak_db}] plus a notes line; --on-silent fail names it in the single kind: input refusal’s problems with reason "silent (peak X dBFS)". A real run measures every file that exists. export.py’s loudness gains silent: true for a silent output, whose note says “output audio is silent” instead of recommending loudness.py.

Per-tool keys added in 1.13: audio (audio.py) reports the mix it built — the --voice level, stereo_widen, whether an --effects bed was mixed, and with --music the music_volume plus a duck object naming the threshold (dB and linear), ratio, attack and release actually used, or null when --duck was not given. loudness.py reports measured (the input’s loudnorm measurement, including input_lra) and targets (the requested lufs / tp / lra). check.py --platform podcast adds two informational rows to checks, channels and chapters.

Per-tool keys added in 1.16, all additive (no 1.15 key is removed, renamed or given a different type):

Per-tool keys added in 1.17, all additive:

Per-tool keys added in 1.17.1, all additive:

Per-tool keys added in 1.18.0, all additive:

Per-tool keys added after 2.2.2, all additive:

Per-tool keys added after 2.10.11 (“nothing happens silently”), all additive:

caption.py --transcribe and silence.py --filler --transcribe: when a local speech engine ran and produced no cue, the refusal is "<engine> found no speech in <input>" with kind: input, reason: "no_speech" and engine in the failure document — distinct from the “no local speech-to-text engine found” refusal, which now only means no engine was found. When --engine auto found engines and none transcribed, the refusal is kind: input with reason: "engine_failed" (one ran and failed) or "english_only" (the only engines found were Parakeet engines auto passed over for a language that is not English), engine (the first one it names) and engines: [{engine, reason, detail}], where detail carries an engine’s own error line. A named --engine that is not installed is kind: missing_tool; one that is installed and failed is kind: input, reason: "engine_failed", with engine and detail.

check.py also gains an informational subtitles row on every platform: PASS when every soft subtitle stream carries a language tag, WARN when one does not (or when there are none). Like channels and chapters it is never counted in failed — no platform refuses a delivery over it.

The MCP audio tool publishes voice as {"type": "string", "enum": ["light", "medium", "strong"]}. A client that still sends the 1.12 boolean {"voice": true} keeps working: true emits the bare --voice, which is medium — the chain the flag has always produced. Send the string when you can; the boolean is accepted at runtime and means medium.

Success (exit 0): one document matching output_schema, always with status: "completed", output, dry_run, commands, and probe of the output when a file was written. probe prints its measurement document directly.

Every writing tool also reports what it verified itself (1.7): verification lists the steps ({"step": "probe", "ok": true}; loudness with the measured and target values for loudness.py and the platform presets of export.py; check with the platform for render.py; color_tags with the tags ffprobe read back for color.py --retag, which fails with kind: verification when they are not the requested ones), and verified is true only when the artifact was written, probed and every listed step met its target. A --dry-run document has verified: false and an empty list. export.py whose written file misses the platform’s loudness spec stays completed (the file is valid) with verified: false and the fix in notes, so a caller keys on one field.

Success is decided by verify_output in _common (scripts/_common/probe.py), not by the ffmpeg exit code alone: the file must exist, be non-empty and give ffprobe at least one stream. A tool that ran ffmpeg successfully but has no usable artifact fails with kind: output (a 0-byte file is removed so a later step cannot mistake it for a result).

Failure (non-zero exit; 127 when ffmpeg/ffprobe is missing): the message on stderr as before, and, when --json was given, on stdout:

{"status": "failed", "exit_code": 1,
 "error": {"kind": "input | ffmpeg | output | missing_tool | timeout | verification | interrupted", "message": "...",
           "code": "INPUT_INVALID | DEPENDENCY_MISSING | FFMPEG_EXECUTION_FAILED | OUTPUT_INVALID | TIMEOUT | VERIFICATION_FAILED | INTERNAL_ERROR",
           "retryable": false},
 "commands": ["ffmpeg ..."]}

message carries the script’s own reason (missing input, ffprobe failure, the last stderr lines of ffmpeg, the verification that failed); an optional error.hint names the flag change that would make a retry meaningful (never a diagnosis of the media); commands lists what was planned or run so the caller can retry or report without re-deriving the command; kind: ffmpeg failures add ffmpeg_returncode (ffmpeg’s own exit code; the process exits 1). code is a purely additive, statically-mapped relabelling of kind (never a new distinction kind doesn’t already make) for a caller that wants a stable enum instead of matching kind strings. retryable is currently always false: none of the kinds are distinguishable today from a deterministic failure that would fail identically on a blind retry, so nothing here claims otherwise until real exit-code/stderr sniffing exists to back that up.

A refusal also carries error.reason, a stable snake_case code for why (contract --json → json_output.error_reasons lists them: output_exists, output_is_input, output_is_directory, output_dir_missing, output_dir_not_writable, output_locked, render.py’s plan_* reasons, and --codec av1/prores’s encoder_missing and encoder_list_unreadable), and error.details, the facts the message is written from, so a caller never parses the prose:

"error": {"kind": "input", "message": "refusing to overwrite existing output 'out.mp4': ...",
          "code": "INPUT_INVALID", "retryable": false, "hint": "pass --overwrite, or choose another -o path",
          "reason": "output_exists",
          "details": {"outputs": [{"path": "out.mp4", "resolved": "/abs/out.mp4", "size": 48213, "mtime_ns": 1729660000000000000}]}}

A tool that runs another (render.py stages and plans) forwards the child’s reason and details in its own error. Both are absent on a failure that is not a refusal, like hint.

Diagnostics. What a run decided on its own – a warning, a fallback it took, a step it skipped – is recorded as it happens (a stderr line too) and lands in the document as diagnostics: {warnings, fallbacks, skipped}, each entry {code, message, ...facts}; it is present on success and failure documents alike, only when one list is not empty, and an entry a render.py stage reported carries stage. contract --json → json_output.diagnostic_codes lists every code with its bucket and meaning (a test reads every warn()/record_fallback()/ record_skip() call against it). Among them: the runner’s odd_dimensions_scaled_even (an encode re-run with an even scale; the frame loses a row or column), non_av_streams_dropped (with the first attempt’s error beside dropped_non_av_streams), measurement_input_pending (skipped: a dry-run measurement of an earlier stage’s output), ffmpeg_list_unreadable (ffmpeg -encoders or -bsfs could not be read; what needed it names the read failure, not a missing encoder) and timeout_env_invalid (an FFMPEG_SKILL_TIMEOUT that is not a number, or is negative); av1_encoder_libaom and crf_clamped; font_substituted (with font_used); drift_correction_skipped (with drift_skipped_reason); cropdetect_windows_failed (with windows_failed); ass_lines_unshifted / ass_lines_malformed; timeline_parse_failed; and every warning: line a tool used to print to stderr only (emoji fallbacks, silent or short inputs, memory-heavy jobs, --strip-dovi without Dolby Vision, render.py’s frame-from-preset and --normalize defaults). render.py’s plan, check and cache records (check_dry_run, check_output_unparsed, cache_entry_damaged, …), export.py’s loudness_not_measured, loudness_outside_spec and colour_not_converted, multicam.py’s camera_not_started, report.py’s report_section_missing and batch.py’s cache records (batch_cache_reset, batch_cache_output_changed, …) are listed there too. Every one of them still prints its stderr line; the record is additive.

A result that is not the one asked for says so. verified is the conjunction of every verification step, and a tool that runs others counts theirs: render.py lists each stage’s own steps beyond its intermediate’s probe, tagged with stage (a cached stage’s as the cache recorded them), and its notes carry each stage’s notes prefixed with the stage’s tool, so an export that missed its platform loudness leaves the render verified: false. A step that could not be measured is a failing step with a reason, never an absent one: export.py’s unreadable loudness is {step: loudness, ok: false, reason: not_measured} with loudness.measured: false, and report.py lists each section it built (loudness, compliance with passed, contact_sheet), one it could not with ok: false and the child’s reason/message. A deliberate change of result is a field: export.py’s trimmed: {from, to} when a platform maximum cut the clip, multicam.py’s cuts naming the camera rendered (camera 0 where the one asked for had not started, with a camera_not_started fallback). batch.py keeps ok as “every step completed” (a spec miss is completed + verified: false, not a failure) and adds verified and unverified: [{step, tool, failed}] to each row, one item verification step per row, a failed row’s error: {step, tool, kind, message, code, reason, details, hint, exit_code} from the failing step’s own document, and cache_miss when a cache entry was not served because its output changed or could not be checked; an interrupted batch has a row for every input (interrupted: true, or skipped: "interrupted" when it had not started) and total counts them all. verify.py’s steps keep the tool’s kind, reason, message and exit_code, and the steps that do not apply are listed under each file’s skipped with a reason.

Audit log. Events a later reader must be able to reconstruct are appended as JSON lines, {ts, tool, event, pid, plan_sha256, details} (plan_sha256 on plan events): plan_rejected, lock_takeover, output_unlocked, output_is_input_refused when a log was named, cache_entry_damaged (render.py --cache) and batch_cache_reset. The file is render.py --audit-log FILE when given; otherwise a refused plan’s <plan>.audit.jsonl next to the plan, a lock event’s hidden .<output>.ffskill-audit.jsonl beside the output, and a cache event’s .ffskill-audit.jsonl in the cache directory or the batch’s output_dir. Nothing is appended under --dry-run. A record that cannot be written is a diagnostics.warnings entry (audit_log_unwritable) and never fails or blocks the run; the refusal’s details.audit_log says {path, written} either way.

MCP relationship

mcp/server.py is a transport. It holds no tool table and no schema of its own:

argparse parser  →  ToolSpec.input_schema  →  contract  →  MCP tools/list inputSchema

At start-up the server builds the ToolSpecs (_contract.build(detect=False)) and derives each tools/list entry with _contract.mcp_tool: the name is the ToolSpec name, description is the ToolSpec’s own one-line description (unchanged from the CLI’s --help first line), the order is the contract’s sorted order, and inputSchema is _contract.mcp_input_schema(ToolSpec). The structured-arguments note that explains how to call a tool (_contract.MCP_STRUCTURED_NOTE) is server-wide, not per-tool: it is sent once, in initialize’s instructions field, rather than appended to all 42 descriptions (1.20.0). tools/call maps structured arguments to argv with the ToolSpec’s mcp.positional and mcp.argument_exceptions. A new public script, a removed one, or a changed parser therefore changes the MCP surface with no edit to mcp/; tests/test_contract.py proves this by copying the skill, adding, removing and editing scripts, and reading tools/list again.

Translation, and what JSON Schema cannot say

Two things are documented rather than encoded, because JSON Schema has no way to express them: when argv is present every other key is ignored (stated in the argv description), and MCP has no notion of positional order, so positionals are named properties whose order is only informative. %(default)s help interpolation is already applied when the ToolSpec is built.

The tools/list document is deterministic (byte-identical across processes and identical to the translation of contract --json), which the tests check.

No inputSchema lists json or progress: they are transport flags mcp/server.py sets itself – it appends --json for every tool but look and probe – rather than arguments a caller chooses. 1.10.0-1.26 listed them unless FFMPEG_SKILL_MCP_LEAN=1; 2.0 leaves them out for good (see “What 2.0 changed”). A client that still sends either is not refused: the server maps it like any other key.

Default tools/list surface: core 12, opt-in for all 42

By default tools/list returns only the core dozen in _contract.MCP_CORE_TOOLS (render, look, caption, export, check, fit, cut, audio, loudness, graphics, silence, probe) rather than all 42, so a session doesn’t pay context for 30 schemas it is unlikely to ever call directly – the set is the tools eval iterations 17-20’s ground-truth expect lists actually name most often across the corpus in evals/agent_prompts*.json. Every tool – including the other 30 – is still callable by name through tools/call regardless of what tools/list advertised; the contract (ffmpeg-skill contract --json) still describes all 42 unconditionally. Set FFMPEG_SKILL_MCP_FULL=1 (anything but “” or 0) in the server’s environment to make tools/list return all 42, as every version before 1.18.3 did.

prompts capability: five workflow recipes

Roadmap 1.20.0 “agent ergonomics”: the server advertises capabilities.prompts in initialize and answers prompts/list / prompts/get with five canned workflows an agent can ask for by name instead of composing the individual tool calls itself: reel, podcast, multicam, delivery_check, hdr. prompts/list returns each one’s name, description and arguments (from _contract.MCP_PROMPTS); prompts/get {name, arguments} fills the matching template and returns one user-role text message naming the actual command lines to run and what to check afterward – the same facts SKILL.md’s own request table states, addressed by workflow instead of by request phrasing. A missing required argument is a JSON-RPC error (prompts/get never guesses a path or a platform); an unknown prompt name is a JSON-RPC error too, distinct from “method not found” since prompts/get itself is a real method.

A line on stdin that is not JSON is answered with a JSON-RPC parse error (-32700, id: null) rather than dropped, so a client whose request was mangled does not wait for an answer that never comes; a JSON value that is not an object, and a notification, still get no response. An unknown method is -32601, any other handler error -32000. These are recipes, not new tool calls: every command line a prompt names is one tools/call (or the CLI) can already run.

Consuming the contract from an agent

A planning agent (for example video-production-agent’s SkillRegistry) can:

1. run ffmpeg-skill contract --json once and register the skill by skill.id and the tools by id;
2. resolve capabilities.required against capabilities.available before planning;
3. pick a tool by role, inputs/outputs, video_required and audio_only;
4. build the call from input_schema and the argument mapping, plan with --dry-run;
5. run, parse output_schema, then run verification.tools, adding look when requires_visual_verification is true.

The measurement documents (probe, check, scenes, sync) are ffmpeg-skill’s own shapes, not another system’s Observation model; convert them in the agent’s adapter. ffmpeg-skill contains no agent-specific code.

Where things live

• scripts/_platforms.py: the delivery table (destinations, specs, safe zones) read by check/export/render/caption/graphics/look
• templates/*.json: the shipped delivery templates render.py --template NAME fills
• scripts/_contract.py: the generator (--json, --static, doctor)
• bin/install.js: ffmpeg-skill contract and ffmpeg-skill doctor
• tests/test_contract.py: schema, consistency (scripts = MCP = installer), MCP inputSchema derived from the contract (equality, determinism, drift, round trips), dry-run, JSON shapes, verification policy, real-media run
• evals/contract/: questions an agent must answer from the contract alone, with the expected answers checked against the live contract
• tests/release_check.sh: runs the contract from the packed and installed copies before a release