"Super-resolution" is not a marketing buzzword, it's a mathematical term used by scientists. First work on this topic was published in 1984 [1] and the term "Super-resolution" itself appeared at around 1990 [2].

Methods usually discussed in scientific literature try to reproduce process of losing quality when shooting video with low-res cameras and then solve inverse problem of finding video which being downsized with that process gives us known low-res material. This is an ill-posed inverse problem which doesn't have straightforward solution and usually requires some additional regularization (applying some artificial constraints) and huge CPU time to check an awfull lot of variants. Modern practical methods are usually simpler but still effective.

SR works when several low resolution images contain slightly different views of the same objects. In this case total information about the object is much higher than information in one frame. Best case is when an object moves in the video. If it doesn't move at all and it's identical in all frames, no extra information can be extracted. If it moves or changes too fast then it looks very different in different frames and it's too hard to use information from one frame in reconstructing the other.

Super-resolution is used now in two tasks: extracting single frames from video (or set of photos) with high quality, or upsizing whole video. SR methods are usually based on two important algorithms: high quality spatial (in-frame) upsizing, and motion compensation for finding corresponding areas in neighbor frames. Many practical implementations upsize material 2 times. If we need to upsize it 4 times, we usually apply SR twice (this can be done internally in implementation).

Let's look at real example of upsizing video 4 times with super-resolution. This example was made using Video Enhancer. First, we have original low-resolution sequential frames of video:

Here are the same frames zoomed 4 times:

Step 1: upsize all frames 2 times using good spatial interpolation, in this case Lanczos3 method:

Step 2: using sub-pixel accurate motion compensation find similar areas in neighbor frames and intelligently merge frames to combine information (and in some cases reduce noise):

Now we have our video upsized 2 times using super-resolution. We repeat this process to get 4x enlargement. Step 3: spatial interpolation again:

And finally step 4: combine neighbor frames to get more details using motion compensation:

Now let's compare it with what we originally had:

One may ask: was that motion compensated merging really required? Weren't spatial interpolated frames good enough? The answer will become obvious if you look at comparison of different upsize methods. Shortly, spatial interpolation methods don't even come closer to a good super-resolution in terms of details and objective metrics.

Note that super-resolution cannot always provide very good results. Read when super-resolution doesn't work.

Papers:

1. R. Y. Tsai and T. S. Huang, "Multiframe image restoration and registration," in Advances in Computer Vision and Image Processing, vol. 1, chapter 7, pp. 317-339, JAI Press, Greenwich, Conn, USA, 1984.

2. M. Irani and S. Peleg. 1991, "Super Resolution From Image Sequences" ICPR, 2:115--120, June 1990.

Try this yourself with Video Enhancer

Get Super Resolution plugin for After Effects

Other articles...