Background

Consumers are increasingly demanding high-frame rate (60fps and higher) video content, whether from digitally-derived sources, such as video games and virtual reality (VR) experiences, or for passive video viewing (Movies, TV, and ‘New Media’ content). Higher frame rates improve the user experience and in some cases (such as VR) reduce motion sickness or eye strain associated with close screen proximity/immersive viewing.

However, the generation of high-frame rate video comes at a cost:

For standard video files, a high-frame rate demands increased storage capacity and/or streaming bandwidth to deliver the file to the consumer as a pre-packaged high-frame rate version.
For media companies with large film and television catalogues, most existing videos were produced at 24 or 30fps. Interpolation is the only way to convert them to high frame rates, and copyright owners demand the highest-quality interpolation to update their media catalogue to maintain original film quality as high-frame rate becomes standard.
For video games, an increased frame rate places high demands on CPU/GPU processors, increasing the hardware capital cost to the consumer (good for some hardware manufacturers but bad for game developers who want to deliver high quality products to a large audience).
For Virtual Reality, the processing demands are even higher than video games due to the need for a seamless wide-angle image. In addition, to “untether” the user through use of a mobile device requires a mobile processing unit (usually a phone) which introduces challenges for battery life and processor overheating constraints with mobile devices.
In video sources where there are fast-moving objects or complex lighting patterns, non-context-aware interpolation introduces new undesirable artifacts into the final output.
In applications where the original video source is also being “extrapolated” (converted to 1080p or 4k from a lower-resolution source) artifacts introduced by inferior interpolation are amplified even further.

Technology Overview

Using a software-based approach, this innovation solves all these problems using a single algorithm. Specific benefits under various use cases:

For pre-recorded video viewing at high-frame rates on-device (upconversion from source of 30fps or less), provides a superior user experience without increased bandwidth or storage.
For video games, interpolation of frames between traditional (mathematically)-drawn frames allows increase of frame rate [2-5x, depending on scene complexity] with minimal impact to CPU/GPU processing.
With further development by integrated manufacturers, may provide power savings on mobile devices or efficiency-focused workflows by with similar or better interpolation results compared to the current standard. For hardware manufacturers, a SoC processing solution is possible with this solution.
Enables real-time processing for on-demand video applications (such as high-frame rate live broadcasting) with less up-front capital investment in hardware.

In addition, for VR and other gaming applications, due to the machine learning-nature of this innovation, once fully trained within a data set specific to the digital environment, the efficiency will improve over time, leading to higher frame rates or decreased power usage as the algorithm self-optimizes to the video source.

Further Details

Videos of 3rd party results [utilizing our code SepConv, which have been improved significantly since the video was posted] can be found in the following video:

The standard “Middlebury Benchmark” (our current publicly-viewable results ranked #1 as CtxSyn) are listed here, and an example video showing the results of CtxSyn processing is available below:

.

Description of the two sections of the video (1) higher framerate and (2) stereoscopic interpolation:

The source (faster-looking output) is followed by our processed result, which runs “slower” because it is running at the SAME frame rate of 30fps, but there are many more frames, which results in a “slow motion” effect. For most commercial uses, the final product would appear to be playing at the same speed as the source but with the additional frames per second increasing the quality of the video.
From the source is created a “stereoscopic” frame set (still potentially at higher frame rates than the source) which allows the media publisher to produce multi-lens light fielding content from a stereoscopic input. This also enables the adjustment of the baseline between stereoscopic views and thus provides optimal stereoscopic 3D viewing experience.

The latest optimized version of the project (codename CtxSyn++) is known to be more efficient/effective (faster processing or power savings compared to the Middlebury results above); this code has been implemented internally at PSU and is not available for public use.

‑ Comparison of quality output for various common interpolation algorithms against PSU output.

‑ Comparison of high-speed interpolation results. State of the art “voxel flow” vs. PSU technology.

‑ Example of interpolated frames generated using the algorithm. Middle 4 frames are derived using PSU software to create more information between the true input of the first and last frames.

Opportunity

This suite of innovations includes the source code and filed patent application. Each is available for immediate licensing on exclusive or nonexclusive basis. For parties interested in assessing this innovation for licensing or partnering, the research group will process submitted video files under confidential agreement as proof of efficacy.

The executable for SepConv [which was utilized by a 3rd party “Two Minute Papers” to create the video comparison at the link above] is available at https://github.com/sniklaus/pytorch-sepconv for non-commercial use.

Superior Video Interpolation and 3D Stereoscopic Processing

Patents

IP Status

Seeking