Machine Learning

​

Overview

Jumper uses four types of machine learning models to help you understand and search through your footage: All models run entirely on your local machine, ensuring your footage never leaves your computer.

​

Visual Search Models

Visual search models are AI systems that can “see” and understand images and video frames, then connect that understanding to text queries or other images. This is what makes Jumper’s visual search work. How visual search models work: These models learn to find correlations between text and visual data from their training material. When you search for something like “a person walking through a door,” the model has learned to understand:

• What “a person” looks like visually
• What “walking” means in terms of motion and pose
• What “a door” is and how it appears in different contexts
• How these elements relate to each other in a scene

When you perform a text search, the model identifies the best matching visual elements in your footage. Thanks to efficient algorithms, this makes it possible to find visual content across thousands of hours of video. The model doesn’t just match keywords. It understands the semantic meaning of your search query and finds frames that match that meaning, even if the exact words don’t appear in the footage. Search methods:

• Text search: Enter a natural language query to find matching visual content
• Image search: Use an image or frame as your search input instead of text. The model uses similar algorithms to find visually similar content across your footage

What this means for you:

• You can search using natural, conversational language
• You can search using images or frames from your footage
• The model understands context and relationships between objects
• It works across different languages (multilingual models)
• Higher resolution models can detect finer details like text on signs or small objects

Model characteristics:

• Size: Larger models generally offer better accuracy but require more memory and processing power
• Resolution: Higher resolution models (384×384, 512×512) analyze frames in more detail, useful for detecting text, small objects, or fine visual details
• Speed: Smaller models analyze faster, while larger models take more time but provide better results

For a complete list of available models and their specifications, see the List of Machine Learning Models.

​

Speech Models

Jumper uses different local speech-to-text models for different languages. Select the language spoken in your footage before analysis, and Jumper routes it to the most appropriate model for your platform.

​

How languages are routed

Languages on the same line use the same speech model:

• English
• Swedish (shoutout kb-whisper)
• Japanese
• Moroccan Arabic (Darija)
• Arabic, Cantonese, Chinese (including Mandarin), Filipino/Tagalog, Hindi, Korean, Malay, Persian, Russian, Thai, and Vietnamese
• All other supported languages

We plan to add more specialized language models in the future, improving transcription language by language as strong new models become available.

​

How speech analysis works

The selected speech model listens to your media’s audio track and converts speech into text during analysis. Once the media is analyzed, you can search for any word or phrase that was spoken. Key features:

• Supports more than 100 languages and language variants
• Selects a model based on the language you choose
• Handles different accents, background noise, and audio quality
• Transcribes dialogue with timestamps
• Works entirely offline with no cloud processing required

Speech analysis can also transcribe one or more selected audio channels and detect speakers. For details, see Speech search. The exact model available can vary by platform and hardware. For the complete language list, see Supported Languages.

​

Face Detection Models

Face detection uses specialized AI models to identify and recognize people in your footage. These models analyze faces frame by frame, extract facial features, and group similar faces together. How face detection models work: The face detection algorithm processes each frame of your media to:

• Detect faces in the frame
• Group similar faces together, even when lighting, angles, or expressions change

What this means for you:

• Automatically identify all appearances of a person across your entire footage library
• Search for specific people using the @ syntax (e.g., @John sitting on a bench)
• Organize people into Collections to keep different productions separate
• Find every scene where someone appears, even if they’re in the background

For more information about using face detection, see Face detection.

​

Summary Analysis Models

Summary Analysis creates a concise whole-clip overview followed by progressively more detailed sections connected to source time ranges. This gives editors and AI agents context before they perform targeted searches. The model is downloaded and runs locally. Choose from Small, Medium, Large, or Extra large in the Summary Analysis configuration. Visual analysis is required; optional speech and face analysis can make the result more useful.

​

Choosing the Right Visual Model

Depending on your specific workflow, usecase and hardware setup, you might want to choose a specific model other than the default. Some suggestions are listed below, or feel free to explore from all the available models. For a complete list of all available models with detailed specifications, see the List of Machine Learning Models.