video capture

Hyperlapse, time lapse, and video stabilization: Different problems, different solutions

Instagram Hyperlapse app

Instagram introduced its Hyperlapse app on the iOS App Store not long after Microsoft showed results from its own Hyperlapse research project in August 2014. Online reactions suggest that a lot of people are confused about what Instagram and Microsoft are actually doing. Are these companies copying each other, or is hyperlapse a trend they both want to ride? Is hyperlapse just a fancy repackaging of time lapse, which many apps already do? Or is hyperlapse stabilization just another form of the video image stabilization that’s already been available in video editing applications for years?

The short answer is that time lapse, hyperlapse, and conventional video stabilization are distinct techniques with different challenges. The recent efforts by Instagram and Microsoft specifically address the instability of hyperlapse video. But they aren’t copying each other, because they use contrasting strategies.

Time lapse versus hyperlapse

First, let’s compare time lapse and hyperlapse. In time lapse photography you record sequential frames at a much lower rate than a normal video or film frame rate. For time lapse you might record one frame every 5 seconds. After recording, you play back the frames at a normal frame rate such as 30 frames per second to produce the effect of compressed time. In the following time lapse, I compressed about 45 real-time minutes into less than one video minute:

In most time lapse photography, the camera stays in one place. The only way the camera gets to rotate or move a short distance if it’s on a motion-control rig. (In the time lapse above, the camera was locked down on a tripod; the movement was simulated in software by panning and zooming a 1920 x 1080 pixel HD video frame across a sequence of 5184 x 3456 pixel still frames from a digital SLR camera.)

In a hyperlapse, the camera can physically change position over a long distance. For example, the camera might be mounted on a car recording a 200-mile road trip, or it might be a helmet camera as you climb a mountain, or you might hold a camera as it records while you walk down the street. Hyperlapses are often recorded with a first-person point of view, especially as wearable action cameras have become affordable and popular like the GoPro. Many hyperlapse videos are recorded manually using frame-by-frame methods that are labor-intensive, as shown in the video below by DigitalRev:

Because a typical hyperlapse recording makes the camera cover a significant distance, it’s just about impossible to maintain consistent framing as you move the camera again and again. During playback, this results in much more shakiness and instability than you’d see in a traditional time lapse, making it difficult to watch. This inherent instability is the hyperlapse challenge that Instagram and Microsoft have tried to overcome.

Comparing how Instagram and Microsoft approach hyperlapse instability

One answer to the problem of hyperlapse instability comes from Microsoft, which published the results of a research project where they found a better way to analyze first-person hyperlapse footage and remove the instability. To achieve this, their solution tries to figure out the original 3D scene and motion path from the 2D video recorded by the camera, and then it uses that synthesized 3D data to reconstruct each frame so that you see much smoother playback. Here’s the demonstration video from Microsoft Research:

The Instagram solution takes advantage of both iPhone hardware and iOS APIs to acquire additional data while recording video. The Instagram Hyperlapse app takes 3D positioning data from the iPhone gyroscope and camera so that it can immediately apply accurate alterations to each frame as it renders the final video. (Instagram says Android APIs currently don’t provide the needed access to an Android phone’s gyroscope and camera.) This is a short demonstration video of the Hyperlapse app by Instagram:

Both approaches are useful in different ways. The Instagram approach is potentially more accurate because it records 3D orientation data directly from the camera at the time each frame is recorded. Having actual orientation data can greatly reduce the amount of processing needed; there’s no need to guess the original 3D motion path because it already recorded that data along with the video. The lower processing load also means it’s much easier to run it on a smartphone, where both processing power and battery power are limited. The Microsoft approach is better when the original video was recorded by a camera that couldn’t provide the necessary gyroscope and camera data, but because it doesn’t have original motion data it needs much more processing power to figure out how the camera moved during the shoot.

The Instagram Hyperlapse app currently has some additional advantages: Instagram paid a lot of attention to user experience, so using the Hyperlapse app is easier, simpler, and faster than creating and stabilizing hyperlapse videos the manual way. And it’s available to millions of people now, while Microsoft is still in the labs and its final ease of use is unknown.

Both Instagram and Microsoft are trying to solve a problem that’s increasingly common now that there’s so much more footage from action cameras like the GoPro, but their approaches are so different that they are clearly not copying each other.

[Update: Microsoft published their own response to questions about the differences between the Instagram and Hyperlapse stabilization techniques. In it they point out another advantage of the Microsoft technique, which is the ability to reconstruct missing pixels by sampling them from adjacent frames. This greatly helps the stabilization results from video taken when your hand or head jumps around too much from frame to frame.]

Hyperlapse stabilization versus software video stabilization

Some have asked: Are these hyperlapse solutions the same as the image stabilization that you find in video editing software? Mostly not. Video image stabilization in software is usually designed to address high frequency camera movement during real time recording, like when a clip looks shaky because you handheld the camera.

Advanced video stabilizing software can go beyond basic software or digital stabilization. Some, such as Adobe Warp Stabilizer VFX, try to work out the camera’s 3D motion path instead of analyzing just 2D shifts in position. Like Warp Stabilizer, the Microsoft hyperlapse solution does a 3D analysis of 2D footage, but Microsoft does additional processing to adapt and extend the 3D analysis for time scales as long as those in a hyperlapse.

The Microsoft approach can also be considered a form of digital image stabilization in that each frame is processed after a frame is recorded. In contrast, you can think of the Instagram solution as a variation on optical image stabilization where a camera or lens includes stabilizing hardware such as a gyroscope, so that an image is already stabilized before it’s recorded.

Each solution has a purpose

This overview should make it clear that these different approaches to stabilization aren’t redundant. They all exist because each of them solves a different problem.

Optical, digital, and software-based image stabilization are options for stabilizing footage that’s both recorded and played back in real time. The Instagram and Microsoft methods are ways to stabilize long-duration footage that’s recorded for a hyperlapse playback speed.

Optical stabilization and the Instagram hyperlapse approach use recording hardware that helps produce cleaner source footage. By stabilizing the image that’s originally recorded, there’s less need for additional stabilization processing.

Digital image stabilization, image stabilization in video editing software, and the Microsoft hyperlapse approach are for post-processing footage that was recorded without physical orientation data from the hardware. They require more processing power, but they work with recordings from any camera.

[Update, May 2015: Microsoft has now made its Hyperlapse technology available in desktop and mobile apps. For details, see the Microsoft Hyperlapse web page.]

Canon SLR video: Technicolor picture style optimizes dynamic range

Canon EOS 7D

In an earlier article, I talked about how customized Canon picture styles (rendering profiles) can optimize the dynamic range of video captured on Canon digital SLRs for better image quality during post-processing and color grading. Now you have one more option: The Cinestyle picture style by Technicolor.

Unlike the picture styles available so far, Cinestyle was developed by Technicolor color scientists together with Canon USA and the ASC Directors of Photography…it’s hard to beat a team like that. And the Cinestyle picture style is free.

Cinestyle is optimized for the Technicolor post-processing workflow, but Technicolor says you can use Cinestyle together with any non-linear editing system (NLE) as well. Also, while Cinestyle is optimized for the Canon EOS 5D MkII, they say it should work fine with other Canon SLRs that capture video, and based on the online comments so far that seems to be true. I personally haven’t tried it out yet. In reading through the information available online, it works best if you also apply an S-curve lookup table (LUT) that’s available as a free download. For more information and to download Cinestyle tools and documentation, go to the following links:

Technicolor Cinestyle and LUT download and documentation
Technicolor FAQ for Cinestyle picture style
Vincent LaForet’s blog post about Cinestyle with examples and tips

Canon 7D: Optimizing DSLR video dynamic range

There’s a great tutorial over at Vimeo on controlling the dynamic range of digital SLRs when capturing video. It turns out that the video mode of digital SLRs is tuned like a typical JPEG mode: To get a contrasty, “finished” look right away. But like JPEGs, this means a lot of tonal and color information is tossed out before the capture is saved, which can be limiting if you need a look that’s different than what the camera gives you. If highlights are blown or shadows are plugged, you may be left with nothing to work with at the high or low end when you try to adjust the image quality.

The technique covered in the video linked below involves using Canon Picture Styles (presets for how the camera processes images) to dial down the contrast and color in an attempt to squeeze as many of the original scene’s tones as possible into the range the sensor can capture. A file captured this way looks flat and lacks contrast, and isn’t something you would show as finished. But that’s because, as with raw capture or like Ansel Adams shooting film, we are at a step in the process where we’re not trying to create a perfect picture at the moment of capture, we’re trying to capture enough of the right data from which we can produce a perfect picture in post-processing. That’s an important difference.

It’s also important to understand that this capture technique isn’t quite as good as actually capturing video in raw format, but video cameras that do are pretty rare and I think most of them are expensive and called RED. This technique is about doing the best you can with the non-raw capture you have.

This capture technique is often called increasing or maximizing the dynamic range, but I prefer to call it optimizing. You’re not making the sensor capture more tones, you’re rearranging the tones you’ve captured before they’re recorded. I suppose you could call it maximizing the available dynamic range.

Here’s the URL to the video:
http://vimeo.com/7256322

While the video talks about the technique in terms of the Canon 7D, Canon Picture Styles can be used with the 5DMkII and others. And the principle can be applied to other brands of cameras that give you control over the image quality of the video.

(Update: [May 2011] You might want to try the new Cinestyle picture style by Technicolor and Canon, a more “official” version of the picture styles described in this post.)