Background

To provide slow motion video capabilities, camera manufacturers must include electronics (image sensors, video processing hardware, high‑speed active memory and storage writing capabilities) to record the 120 frames per second (fps) to 10,000 fps offered to the consumer. For most slow motion cameras, at least 240 fps is necessary to meet the user’s needs. There is also a significant tradeoff in terms of image quality and hardware costs for extremely high speed cameras (1000fps and higher), and many cameras which reach 10,000 fps have only low‑resolution output since the image sensors are unable to be read faster due to the limits of physics and speed of reading the sensors.

Portland State University has developed a best‑in‑class interpolation technology which produces near‑perfect results generating new video frames via artificial intelligence (AI) assisted algorithms. With this technology, camera manufacturers can double, quadruple, or even further increase the fps of their videos without costly hardware improvements. For extremely high speed cameras which have reached the limits of sensor physics, this technology may enable high speed shooting at higher resolutions with no appreciable image loss for the end user.

Technology Overview

This technology utilizes three separate processes integrated into a single algorithm: motion estimation, feature pyramid extraction, and neural network frame synthesis.

Motion Estimation analyzes the motion in the frames of the video, looking for areas of fast and slow movement to inform the Frame Synthesis.

Feature Pyramid Extraction analyzes the frames looking at detail concentration in the various parts of the image, optimizing processing by focusing the frame synthesis on areas of high detail while spending less computing time on areas of low detail.

The Frame Synthesis relies on a trained neural network, which incorporates the information gathered to generate new frames with exceptional image quality.

Each of these processes is patent pending (three patents total) and has example executable code.

Benefits

Videos of 3rd party results [utilizing our code SepConv], (which has been improved significantly since the video was posted) can be found in the following video:

The standard “Middlebury Benchmark” (current publicly-viewable results ranked #1 as PyrWarp) are listed here, and an example video showing the results of CtxSyn processing is available below:

Applications

The university see this technology has having two primary categories for camera makers:

Consumer (point and shoot or mobile devices) ‑ the ability to add frames for slow motion effects without the expense of high‑speed, high resolution sensors gives camera engineers more ability to trade cost for performance by designing products that would cost less (lower speed sensors or lower‑cost support hardware on device) but still be capable of producing slow motion effects for the users.

For camera manufacturers that focus on sports/rugged consumer cameras, the ability to slow down video is a primary selling point for the consumer. This algorithm can be applied even to video which was already capture at high fps to create even SLOWER video, giving the consumer the option of having extremely detailed, extremely slow‑motion capabilities for their personal use or blogging/vlogging.

High Speed Camera manufacturers specifically have extremely high development expenses to produce mid‑quality resolution with very fast fps. Producing high resolution video at 10,000 fps or higher is not economical and may not be possible due to the limits of electronics physics. However, incorporating this technology would allow high speed camera engineers to use slower, higher‑resolution sensors and create the high speed video through interpolation, opening up new product lines for high fps with high resolution.

A third party video using an older (inferior) version of the PSU software to speed up high fps video (notice the additional details created by the AI‑produced frames) can be viewed here:

For pro users, there may also be an opportunity to increase fps using cloud‑based processing, and therefore an opportunity to engage with users or provide subscription service benefits of increasing fps in post processing.

Opportunity

Portland State University are looking for licensing partners that would be interested to incorporate this algorithm into their devices or post‑device processing software (PC or mobile applications). This software is demonstrated and ready to incorporate.

For further information about the technology and other applications please see this related technology: https://app.in-part.com/articles/9y5gxxMEgBLK

Slow Motion Video From Any Source

Patents

IP Status

Seeking