Writing FrameX plugins

A plugin adds to FrameX without changing it. It can:

  • add commands to the Plugins menu,
  • edit the project, through the same checked edits FrameX's AI uses, each one undoable,
  • react to events: a project opened, media imported, a transcript made, a video exported,
  • draw explainer scenes, which FrameX's explainer can then plan into videos.

A plugin is an ordinary program, in any language: Python, JavaScript, C#, Rust, Go or anything else that can read and write lines of text. FrameX starts it when it's first needed and talks to it in JSON over its standard input and output. Because it runs as a program of its own, a plugin can't crash FrameX. If it crashes, stops answering or sends something FrameX can't read, FrameX stops it, turns it off and tells the user why. It stays off until the user turns it back on in Plugins > Manage plugins….

Packaging a plugin

You hand out a plugin as a zip. Users install it by dropping the zip onto FrameX's window, or with Install from a file… in the Plugins window. FrameX shows its icon, name, version, author, description and what it adds, and installs it only once the user presses Install.

hello.zip
  plugin.json       required: says what the plugin is
  icon.png          required: named by "icon" in plugin.json
  main.py           whatever your plugin needs, in any folders
  • plugin.json goes at the top of the zip. It can also be inside one folder that holds everything, which is what you get by zipping a folder.
  • author, version and icon are required, as well as the fields every plugin needs (below).
  • The icon is a square PNG, at least 64×64 and at most 1024×1024 pixels; 256×256 looks sharp everywhere. FrameX shows it when installing and in the Plugins window.
  • The program in run, when it's a file ("hello.exe", "bin/hello.exe"), must be in the zip. A bare name such as "python" or "node" is a program the user installs, so say so in your description.
  • At most 5000 files and 1 GB once unpacked.
  • No file may point outside the plugin's folder, and there may be no links. A zip that breaks a rule isn't installed, and FrameX tells the user which rule.

FrameX unpacks a plugin into a folder named after its id. Installing a newer version replaces the old one in one step and keeps the user's settings for it. If the old version had been turned off after a crash, the new one starts turned on.

Where plugins live

Installed plugins are folders inside FrameX's plugins folder, which Plugins > Open the plugins folder opens. On Windows it's %LOCALAPPDATA%\FrameX\plugins. While you develop a plugin, you can work right there, without zipping it each time: change your files, then press Reload in the Plugins window. The same rules apply, including the icon.

plugins\
  hello\
    plugin.json
    icon.png
    main.py

Plugins run like any program on the user's PC, with the same access to their files. Tell your users what yours does. FrameX reminds them to install plugins only from people they trust.

plugin.json

{
  "id": "hello",
  "name": "Hello Plugin",
  "version": "1.0.0",
  "author": "Your name",
  "icon": "icon.png",
  "description": "What it does, in a sentence.",
  "api": 1,
  "run": ["python", "main.py"],
  "commands": [{ "id": "greet", "title": "Add a greeting at the playhead" }],
  "events": ["exported"],
  "visuals": [{
    "kind": "countdown",
    "label": "Countdown",
    "description": "When to use it and what its data means, for the model planning explainers.",
    "example": { "from": 10, "label": "Until launch" }
  }],
  "settings": [{ "key": "greeting", "label": "Greeting", "kind": "text", "default": "Hello" }]
}
Field
idUnique and stable. Lower-case letters, digits, ., - and _. FrameX remembers settings and whether the plugin is on by its id.
name, version, authorRequired. Shown when installing and in the Plugins window.
iconRequired. A square PNG in the plugin, 64 to 1024 pixels across, by its path in the plugin's folder.
descriptionWhat it does, in a sentence or two. Shown when installing.
apiThe plugin API it was written for. This FrameX speaks 1. A plugin written for another version isn't loaded, and the window says why.
runThe program and its arguments. A program in the plugin's folder ("hello.exe", "bin/hello.exe") is run from there. Otherwise it's found like a command, so ["python", "main.py"] and ["node", "index.js"] work. The plugin runs in its own folder.
commandsMenu items, each with an id sent back to you and a title shown in the menu.
eventsAny of project_opened, imported, transcribed, exported.
visualsExplainer scenes you draw. The description and example are given to the model that plans explainers, so it can use your scene where it fits.
settingsShown in the Plugins window and sent with every request. kind is text, number, toggle or choice (with choices).

The conversation

Every message is one JSON object on one line. FrameX sends requests on your standard input:

{"id": 7, "method": "command", "params": {...}}

You answer each on your standard output, with the same id and either a result:

{"id": 7, "result": {"ops": [...], "message": "Done."}}

or an error, when the request didn't succeed but your plugin is fine. FrameX shows the message and leaves the plugin on:

{"id": 7, "error": {"message": "Select a clip first."}}

Requests come one at a time: FrameX waits for your answer before sending the next. Write only messages to standard output, and flush after each. Anything else, such as debugging output, goes to standard error: FrameX shows the last few hundred lines in the Plugins window and keeps them in plugin.log in your folder. A line on standard output that isn't JSON turns the plugin off.

Every request's params includes your settings, as an object of their current values.

initialize

Sent once, when FrameX starts your plugin, before anything else. Answer with any result, such as {}. An error here stops the plugin starting.

{"api": 1, "framex": "0.3.0", "settings": {...}}

command

A user picked one of your commands.

{
  "command": "greet",
  "project": { ... },
  "playhead": 12.5,
  "selection": ["clip_ab12"],
  "range": {"start": 10.0, "end": 20.0},
  "settings": { ... }
}

project is the whole project as FrameX saves it: its settings, its assets (media, with their transcripts), and its timeline (clips on the main track, overlays, texts (titles and captions), audio, markers). range is the In and Out range if one is marked, or null. Times are in seconds.

Answer with the edits to make and a line for the user, both optional:

{"ops": [{"op": "add_text", "text": "Hello", "start": 12.5, "duration": 3}], "message": "Added a greeting."}

FrameX checks every edit and applies them together as one step, named after your plugin and command, so one Ctrl+Z takes them all back. An edit it can't read or can't make is left out, and the user is told. The edits you can send are listed in Edits plugins can make.

event

Something happened that your plugin asked to hear about. The params are a command's, without command, plus:

eventdetails
project_opened{}
imported{"asset": {...}}: the media, once it's ready to use
transcribed{"asset": {...}}, with its transcript
exported{"path": "C:\\...\\video.mp4"}

Answer like a command: edits and a message, both optional. {} is fine.

paint

One frame of one of your explainer scenes. FrameX draws the shapes you send with its own fonts and colours. It then adds the video's light, grain and camera moves, so your scene matches the rest of the video.

{"kind": "countdown", "data": {"from": 10, "label": "Until launch"}, "time": 1.25, "duration": 6.0, "width": 1920, "height": 1080, "words": [{"text": "Ten", "start": 0.4, "end": 0.7}], "settings": {...}}
  • time is seconds into the scene; duration is how long it lasts.
  • The picture is always 1080 tall and width wide (1920 for 16:9, 608 for 9:16). 0,0 is the top left.
  • words are the words spoken over the scene, timed from its start, so you can show things as they're said.
  • Draw it the same way every time for the same time. FrameX may ask for frames in any order, and several times for motion blur.

Answer with up to 4000 shapes, drawn in order:

{"shapes": [
  {"shape": "rect", "x": 100, "y": 200, "w": 400, "h": 80, "color": "accent", "opacity": 1},
  {"shape": "circle", "x": 960, "y": 540, "r": 120, "color": "faint"},
  {"shape": "line", "points": [[100, 900], [1800, 900]], "width": 3, "color": "dim"},
  {"shape": "polygon", "points": [[0, 0], [200, 0], [100, 150]], "color": "#d4443a"},
  {"shape": "text", "text": "Until launch", "x": 960, "y": 820, "size": 46, "font": "title", "align": "center", "color": "ink"}
]}
  • Colours: ink (text), dim, faint, accent and background follow the video's look. #rrggbb is a fixed colour.
  • Fonts: body, bold, serif, mono, light, or title, which sets narrow, evenly spaced capitals like FrameX's own titles.
  • Text: y is the top of the line, and align is left, center or right of x.
  • A shape FrameX can't read is left out.

In an explainer's plan, your scene appears as:

{"type": "plugin", "plugin": "hello", "kind": "countdown", "data": {"from": 10, "label": "Until launch"}}

You can also write it into a scene by hand, in the Generated scene box of the Inspector. If your plugin is missing or off, the scene says it isn't available instead of being drawn.

Messages from you

While working, you can send FrameX messages of your own, without an id:

{"method": "notify", "params": {"text": "Found 12 titles", "problem": false}}
{"method": "progress", "params": {"fraction": 0.5}}
{"method": "log", "params": {"text": "for the Plugins window"}}

notify shows a line to the user. Set problem to show it as one.

When FrameX turns a plugin off

What happened
Its program couldn't be startedWrong run, or a missing interpreter
It ended while FrameX waited for an answerA crash, or it quit
It took too long to answer15 s to start, 60 s for a command, 30 s for an event, 5 s for a frame
It wrote a line that isn't JSONUsually a stray print

FrameX stops its program, turns it off, remembers why, and shows the user. The plugin stays off, even after FrameX restarts, until the user turns it back on. It then starts afresh. An error answer is not one of these: it's a plugin saying no, and it stays on.

When FrameX closes, it ends its plugins' programs too.

Your plugin and FrameX's licence

FrameX's licence lets you write plugins that work through this API and share or sell them on your own terms. They're your own work, as long as they contain no part of FrameX.

← Your files stay yours Edits plugins can make →