FxFactory 9.0.4 introduces a new type of parameter called Analyzer that lets your plugin analyze video frames in sequential order to produce one or more output movies as a result. The goal is to perform expensive computations, such as evaluating CoreML models, during a separate analysis pass so as to avoid repeating this work when rendering.
This marks the first time FxFactory plug-ins can use more than one composition. The Main composition is still tasked with producing output frames requested by the host app, whereas any compositions attached to the analysis phase produce output movies stored externally to the project.
The new parameter also enables image processing algorithms or models that work on a sequence of consecutive video frames, such as optical flow analysis. This is a scenario that could not be handled by evaluating models at render-time, since there is no guarantee that frames are processed in the correct order (the user may play and pause the clip, or move the playhead back and forth at will) and there is no provision in FxFactory for accessing neighboring frames during the render pass.
Adding an Analyzer to an Existing Plug-In
Since this feature has been introduced with FxFactory 9.0.4, make sure your product’s Requirements are set to require at least FxFactory version 9.0.4.
Next, Edit your composition and add the new Analyzer node under the FxFactory category. Publish its main input:
You will configure one or more Keys at a later step. When you save the composition and apply its changes back to the plug-in, FxFactory creates an Analyzer parameter mapped to the new input, but it will still be displayed as incomplete:
The parameter remains in that state until you have provided a composition for it to use. Any composition you wish to use with an Analyzer parameter must be present in the product’s Assets. The composition must have one published image input using the default inputImage key. This input is fed video frames during analysis. FxFactory can create a starter composition for you by clicking the New button:
The resulting composition is set up to extract the Person Contour for each frame, saving the result as a black and white map.
When you close the composition and apply changes back in FxFactory, the Person Contour output (published as outputPersonContour) has been recognized, giving you a chance to customize how the output movie is created:
The choice of codec will depend on the type of information your composition is producing. HEVC is currently the most space-efficient, but takes more CPU time to decode. ProRes variants take up significantly more space but provide both superior quality and faster decoding.
During analysis of the effect clip, a movie is produced for each output image published by the composition. The movie has the same frame rate as the effect clip. Its resolution is dictated by the size of the output image produced by the composition. It need not be the same size as the input video frames fed by the host app. In fact, most models are hard-wired to produce their output at a fixed resolution, usually much smaller than video frames:
Creating a video whose dimensions are only as large as they need to be saves more than just disk space: smaller frames are generally faster to decode as well. During analysis, the composition is required to produce images of identical size for any given output. It is not possible to encode a movie if the input frames vary in size from one frame to the next. On the other hand, if your composition has multiple published image outputs, hence producing multiple movies, there is no requirement that they all have the same dimensions.
Not all host applications feed video frames at the native resolution of the clip during analysis. InAfter Effects , frames are fed at the Canvas preview resolution. Because users may not be aware of this behavior, FxFactory displays a warning in the progress window to give users a chance to restart analysis after selecting a higher quality setting.
Feeding Analysis Results to the Main Composition
The composition used by the Analyzer parameter currently has a single published image output. To access video frames produced by the output during rendering (i.e. from the main composition) click the button indicated below to copy the published output key to the clipboard:
Open the main composition, locate the Analyzer node added in a previous step, and paste the contents of the clipboard (outputPersonContour) into its single Key input:
As you adopt the Analyzer parameter in your own plug-ins, you will likely customize both the name and number of outputs in the composition. Remember that for each published output, a movie is produced. To access frames from that movie, the main composition must know the corresponding Key.
The corresponding image output now lets the composition access any frames previously rendered during the analysis phase. Irrespective of the resolution of the movie, all video frames are automatically scaled by the Analyzer node to have the same size as the current video frame.
When using the plugin in any supported video app, a button available in the inspector and on-screen starts the analysis phase:
The title of the button is Analyze by default, but it can be customized to provide the right context to the user. During analysis, all video frames between the in and out points of the effect clip are fed in sequence to the composition associated with the Analyzer parameter. The output of the composition is displayed by the progress window, whereas the output images read from the outputPersonContour output are encoded into a movie:
When the analysis is complete, users can reveal the location of all movies created by the plug-in through a button in the inspector:
These movies are not stored with the project, nor are they added to the current Library (in
Should these files be deleted intentionally or accidentally, the Analyzer parameter allows the user to quickly start a new analysis phase. The user will also have to repeat this step any time the in and out points of the clip are adjusted, since this process potentially adds more frames to the analysis pass.
Fine-Tuning the User Experience
In some cases, a plug-in may be able to produce the same output whether or not Analysis has been performed. The analysis can be seen as a performance bonus: use the pre-computed output when available, fallback to doing the same work during a regular render pass otherwise. This approach does present some difficulties. It requires both the main composition and the one used during analysis to implement the same image-processing logic. And there might be significant differences in interpreting the output of certain models directly (e.g. Extract Mask) vs having it come from a movie encoded with a specific color space, compression setting and depth.
In other cases, the Analysis is a hard requirement: without the results of the analysis pass, a plug-in will simply be unable to produce its output. It will therefore make sense to alert the user to this fact. The Analyzer node can be customized to provide current status, as well as a pre-defined error message:
These outputs are designed to be used with an Alert node:
The Alert node automatically displays relevant warnings to the user:
Manual Access to Movies Created During Analysis
On its default configuration, the Analyzer node extracts the current video frame created during the analysis pass, one for each Key you specify. Video frames are automatically scaled to match the current video frame size. The fixed output of a CoreML model, e.g. a 512x512 grayscale map, is automatically scaled to full frame. This helps compensate for how most models work. It also accounts for another characteristic of video apps: they don’t process video frames at a single, fixed resolution during playback. They dynamically switch from full resolution to half- or even quarter-resolution to maintain interactivity. The Analyzer node eliminates the need to handle these problems manually.
What if you need to customize this process? The Analyzer node can be configured so as to provide the URL of each movie associated with a given Key:
This gives you to implement more complex behaviors, such as choosing a different playback range or accessing neighboring frames.
Why Does Analyzer Store Movies Externally?
None of the plug-in SDKs supported by FxFactory have excellent support for storing large amounts of data (or files). The Analyzer parameter can be configured to output movies in ProRes, which can quickly balloon in size even for short sequences. For this reason, FxFactory stores all movies that result from an Analysis pass inside the current user’s Movies folder.
FxFactory takes great care in preventing disk space from being wasted unnecessarily. When a new analysis pass is requested by the same parameter, any files generated by a previous request are automatically removed. Users can reveal the location where these files are stored, and delete them to make disk space. Doing so simply means the user has to repeat the Analysis pass at a later time, before rendering. The Analyzer node can be configured to detect and report this condition to the user, so as to provide a user-friendly experience:
Avoiding Chroma Sub-sampling Artifacts
Because Analyzer encodes the output of its analysis pass to a movie, it is best suited to storing grayscale images. Most of the supported video codecs employ chroma sub-sampling, which reduces precision for RGB content. This can be disastrous for content that is not meant to be viewed by human eyes, but rather used to store the intermediate steps of a render process.
While it is possible to select
One such example is splitting the results of 0…1 range expected for movie encoding, and creating two separate grayscale movies accessible from the main composition:
Mapping Parameters to Analyzer Composition Inputs
Every FxFactory developer is familiar with adding parameters to a plug-in that map to published inputs of the composition used for rendering.
The Analyzer parameter now allows you to map those same parameters to inputs published by a composition used for video analysis. The value of designated parameters is then fed to the main composition when rendering, and to the Analyzer composition during video analysis.
Limitations
Only the following parameter types can currently be mapped to compositions used for video analysis: sliders, angles, toggles, colors, points and rectangles. Moreover, if the value of these parameters has been keyframed, only the current value at the playhead location is fed to the composition when analysis begins. It is advisable to explicitly mark any parameters linked to the Analyzer composition as non-keyframable by turning off the Animatable property.
In previous versions of FxFactory, point parameters could only be mapped to two separate inputs of the main composition, for the X and Y coordinates, respectively. To simplify the process of mapping the same point parameter inside an Analyzer,
…and have it mapped to a point parameter:
This enables a simpler UI to map plug-in parameters to published inputs. Every input published by the Analyzer composition appears in a list, along with a popup menu to select one of the parameters compatible with the input type:
An option to the right of the popup menu controls whether a change to the parameter should invalidate any previous analysis. The option defaults to true, under the assumption that if the current parameter value influences the work done during analysis, the user should be immediately aware of the need to repeat the process.
If the FxFactory UI on your Mac appears to be missing icons that help you identify various parameter types, please make sure the SF Symbols app is installed on your system.
Examples
For examples on how to use the Analyzer parameter in a working plug-in, visit this section: