“Does HFR look ‘better’ than traditional 24 fps, or does it just look ‘different’?”

“Will audiences eventually get used to HFR and then prefer it over traditional 24 fps, as with many other advances in motion imagery?”

Clearly HFR looks different. Anyone who saw “The Hobbit” in HFR can agree with that. And I think it’s fair to say that Jackson and his team succeeded in their goal to produce a more lifelike visual experience. But many viewers and critics found this new hyper-real experience off-putting, reducing the so-called “suspension of disbelief” that is considered essential to storytelling. Critics widely panned Jackson’s use of HFR, comparing the look of HFR to television and video games, deriding it as “non-cinematic,” “plasticine,” and “fake.”

If HFR didn’t work well for a cinematic adventure film like “The Hobbit,” there must be a scientific explanation. Or, as it turns out, maybe there isn’t. Yet…

At today’s Hollywood Post Alliance (HPA) Tech Retreat, Charles Poynton presented a fairly comprehensive overview of the challenges with motion imagery. His conclusion is that we don’t know enough about how motion is perceived by humans to explain with scientific certainty why HFR looks different. Poynton provided a thoughtful and detailed overview of how image capture and display technologies interact with the human perceptual system. Without going into detail about pulse width modulation or bit splitting (!) here are some of Poynton’s main points from today’s session:

• We are still trying to fully understand how motion is decoded in the brain. Some scientists believe the brain processes chunks of input (20 to 50 milliseconds each) while others believe perception is more like a continuous stream.
• Despite advances in our learning about human perception, along with advances in algorithms that use these learnings to automatically assess visual quality, the only way to conclusively assess visual quality remains subjective testing by humans.
• The way that humans track motion, by moving their eyes several times per second, presents a challenge for motion imaging systems.
• Different display technologies (LCD, Plasma, DLP, CRT) emit light in different ways over time, creating an additional factor that must be carefully optimized.
• Along with wide gamma and other advances in motion image technology, HFR needs further study and refinement before it can be successfully adopted.

As a fitting postscript to Poynton’s presentation, Mark Schubin suggested that not only do we not yet know why HFR looks so different from traditional 24 fps, we also don’t know where this will all land. Many other advances in cinema technology were initially rejected by viewers, but eventually became standard fare.

In my opinion, HFR will eventually be incorporated as an additional, valuable tool for storytellers.  Higher frame rates are not inherently superior to 24 fps for all content genres, but they may meaningfully enhance the audience experience when used for scenes that require fast motion or hyper-reality. It may turn out that some television genres like news, live entertainment and sports may be better suited to high frame rates, because the goal of these productions is to approximate live attendance of the event. Dramatic programs with high production values, such as The Hobbit, may ultimately benefit from the suspension of disbelief induced by the cinematic look of 24 fps, incorporating HFR more sparingly as a visual effect.

So maybe HFR is not the problem – maybe we are. Notwithstanding the creativity and skill of Peter Jackson and his team, perhaps we don’t know how to do HFR well yet. Maybe we don’t yet know when to use HFR and when not to. But at least Peter Jackson has forced the discussion by boldly pushing into this new frontier, for which we can all be grateful.

These changes in Avid’s leadership follow several years of declining results. Here are some of the numbers:

• When Greenfield took the helm on December 19, 2007, Avid’s stock price was $25.40. At close on Monday, it was $7.87, a decline of 69%.
• Avid’s revenue for 2007 was $989.6 million. Although Avid has not announced official results for the fourth quarter of 2012 as of yet, some analysts are predicting roughly $614 million in total revenue for 2012. If this estimate is accurate, Avid’s top line will have declined by more than 33% during the last five years. (Note that these numbers are not fully adjusted for acquisitions and divestments.)
• Avid’s cash position has declined from $224.5 million at the end of 2007, to roughly $71 million, according to Avid’s most recent results announcement (Q3 2012), a decline of more than 68%.

It’s important to note that other large vendors such as Grass Valley and Harris Broadcast have also struggled during this interval. And despite poor financial performance, Avid achieved several important accomplishments over the last five years:

• Significantly reduced price points for many products, achieving better alignment to the market.
• Decoupled hardware and software for Pro Tools and Media Composer, enabling customer choice.
• Implemented “voice of the customer” programs, to increase customer influence.
• Expanded product categories through strategic acquisitions, including Euphonix (audio consoles and control surfaces) and Blue Order (media asset management).
• Refreshed several key product lines, while improving overall product quality and customer support.
• Unified multiple brands under a single Avid brand.
• Streamlined costs by transitioning to shared services and increasing offshore contract engineering.
• Narrowed market focus on professional customers by divesting its consumer product lines, while expanding Avid’s professional services capabilities.

Although these accomplishments are significant, they did not result in sustained growth or profitability. To rebound, Avid will need to take bold steps. The good news is that the media industry’s value chain continues to evolve, creating new opportunities for technology vendors. Here are ten of the top macro trends, as I see them, in no particular order:

1. Transition to HD and Digital. Although largely complete in North America, the transition to digital distribution and HD television is still in full swing globally, creating upgrade opportunities for technology vendors.
2. Beyond HD. Film studios and broadcasters continue to advance beyond HD for high-end content production. As content creators adopt higher spatial resolution (e.g. 4K), higher frame rates (e.g. 48 fps and 60 fps), stereoscopic, and high dynamic range, as well as other advances in imaging, they will need to upgrade their equipment and expand their storage infrastructure.
3. Growth in the MI Market. As the global economy has rebounded, so has demand for musical instruments, as well as music creation tools and audio recording products.
4. More Broadcast Channels. Broadcasters continue to add new channels, especially outside of North America, driving increased demand for content creation and distribution technologies.
5. New Distribution Outlets and Multi-Screen. Broadcasters and media companies are monetizing content across more distribution outlets, including on demand, web, mobile, and social media, fueling the market for content transformation and distribution technologies. Additionally, new viewing paradigms are emerging including interactive, multi-screen consumption, creating new opportunities for targeted advertising.
6. Corporate Video. The use of video in corporate environments continues to explode, creating opportunities for vendors of simple content creation tools, collaboration infrastructure, centralized storage and content management systems, as well as transformation and distribution services.
7. Expanded Audio Deliverables. More content production means more audio production. Broader international distribution is driving more language versioning than ever before. New audio accessibility and loudness regulations, as well as new surround formats like Dolby Atmos, require more audio processing. All of these trends are fueling increased demand for audio production and processing tools.
8. Transition to File-Based Workflows. As the entire media value chain continues its transition to file-based workflows, demand will increase for media asset management, media storage, and workflow orchestration solutions.
9. Operational Integration and Automation. Media companies are looking for ways to increase their productivity without increasing operating expenses. To this end, they are adapting concepts and technologies from the IT industry to achieve new levels of process automation and operational integration, expanding the market for professional services and integration technologies.
10. Cloud. As media companies streamline their operations, cloud-based infrastructure will become increasingly appealing – providing elastic capacity and cost, while enabling new workflows for distributed teams. This trend will unfold over the next several years, creating opportunities for media-savvy cloud-based service providers.

These opportunities are not new, and they are not accessible only to Avid. But none of these opportunities is out of reach for Avid either. The question is whether Avid will participate meaningfully in these opportunities, or continue to retrench in its existing core business.

Earlier this week, one of my former colleagues at Avid told me that many Avid employees do not see this week’s announcement as a regime change, but rather a modest change in roles for the existing management team. No board members have come or gone, an existing board member chosen by Greenfield is now CEO, and Greenfield himself is still in the picture as a board member. It’s hard to argue that this week’s announcement represents a radical departure from the status quo.

Still, one could argue that Hernandez is well positioned to effect change. Although Hernandez has been on Avid’s board for several years, most Avid employees and customers are not familiar with him. This affords Hernandez the opportunity to approach his appointment as a newcomer would. Yet his tenure on Avid’s board has exposed Hernandez to Avid’s business, customers and markets. The question is whether Hernandez will take this opportunity to effect change, or continue the course set by his predecessor.

Predictably, Monday’s announcement has sparked increased interest in Hernandez’s track record. According to publicly available documents, Hernandez was previously Chairman of the Board and CEO of Open Solutions, Inc., a technology company that was acquired in January 2013 by Fiserv, Inc. for $55 million. As part of the acquisition, Fiserv also assumed approximately $960 million of debt. Open Solutions had been taken private roughly six years earlier by the Carlyle Group and Providence Equity Partners for $1.3 billion. In the six years since Open Solutions was taken private, the company’s debt more than doubled from roughly $448 million to $960 million. Because Open Solutions has had a history of losses, Fiserv will realize tax breaks to the tune of $165 million, lowering the total cost of the acquisition to around $865 million.

This means that the market value of Open Solutions has declined by roughly 33% over the last six years, from $1.3 billion to $865 million. According to articles and posts on public websites like Glass Door, Open Solutions has also implemented layoffs and has reduced employee benefits in order to manage costs, affecting employee morale. While the decline of Open Solutions has not been as precipitous as Avid’s, these similarities are worth noting.

In Avid’s Press Release Monday, Hernandez described his appointment as “…an exciting opportunity to lead Avid at this very important juncture…” adding that “[Avid] is well positioned for growth and global expansion in this fast-moving marketplace.” It certainly is an exciting time of change for the media industry, and a potential inflection point for Avid. Along with many other fans of this great company who continue to root for an Avid turnaround, here’s hoping that Hernandez is ready and able to chart a new course forward for Avid.

HEVC is now fully baked and we will soon be seeing wider industry adoption!

It is important to note that the current HEVC specification only describes profiles for single-view 8-bit and 10-bit video with 4:2:0 subsampling. Extensions are in progress for 4:2:2 and 4:4:4 color sampling as well as multi-view (3D). Until these extensions are ratified next year, H.265 will not achieve full functional parity with H.264.

Because HEVC is more complex computationally, adoption will depend on continued increases in processing power and battery power. Although some current mobile devices may be able to decode HEVC in the future simply by updating their software, HEVC will put a huge strain on processors and battery life. For many applications, hardware implementations will be optimal, and many hardware vendors have announced chips that will support H.265. For example, at CES 2013 Broadcom showed the BCM7445, an HEVC decoder chip that will be incorporated into next generation home gateway devices in mid-2014.

But as with any codec transition, consumer adoption will depend on support across the value chain:

• Consumer viewing devices will require updated software or hardware to decode HEVC
• Content owners and distributors will need to create and offer HEVC-encoded content for distribution
• Content distribution networks, such as cable operators, will need to update their delivery infrastructure to support HEVC

Unfortunately, these barriers to adoption are significant. That’s why most of us are still watching MPEG-2 on TV, even though H.264 was ratified several years ago. Still, finalizing the specification is an important first step towards adoption. And HEVC offers many benefits to broadcasters and content owners. As with 4K and Ultra High-Definition Television (UHDTV), HEVC is an important building-block in the next wave of high resolution imaging.

Not surprisingly, Ultra High Definition Television (UHDTV for short) was the big buzz at CES last week. All of the major TV manufacturers were demonstrating UHDTV sets running at 4K resolution, with Sharp showing an updated prototype 85” TV running at 8K resolution. But even as the TV sets themselves are approaching availability, the picture is becoming increasingly clear (pun intended) that uptake will be slow.

• Although UHDTV sets will be available as soon as March of this year, initial pricing will be prohibitively high (>$10,000). According to CEA analysts, UHDTV units will account for only 5% of TV’s sold in 2016.
• No 4K or 8K content is available currently to view on UHDTV sets and likely won’t be available in quantity for years to come.
• There is currently no distribution method for UHDTV content. Data rates for UHDTV can be at least four to sixteen times higher than for 1080p HD.
• Broadcasters are generally unconvinced that a viable business model will emerge for UHDTV.
• Most industry analysts and TV vendors now acknowledge that UHDTV has little (if any) value for displays smaller than 50 inches. For my thoughts on the relative value of higher spatial resolution, please read my earlier post on this topic.

In short, UHDTV is a really impressive technology that is desperately in search of a market opportunity. It’s hard to dispute that UHDTV represents a future we can all get excited about, but (like HD before it) the market for UHDTV will likely take several years to develop. Read on if you want more of the gory details. If you’re a UHDTV novice, feel free to check out the primer on UHDTV I posted last September.

UHDTV’s Will Be Available Soon

Virtually all of the major TV vendors demonstrated UHDTV sets at CES last week. Note that none of the sets is currently available, but some will begin shipping as soon as March 2013. Early units will include 94-inch displays from Sony and LG that will be priced at $20,000-$25,000. Smaller models will cost less – the Consumer Electronics Association (CEA) estimates that the average wholesale cost of 4K televisions will drop to $7,000 by late 2013, then to $2,800 in 2014. Even with this steep decline in price, only 1.4 million unit sales are projected for the U.S. in 2016, equivalent to roughly 5% of the market.

“It’s a very, very limited opportunity,” said Steve Koenig, director of industry analysis at the CEA. “The price points here are in the five digits (in U.S. dollars) and very few manufacturers, at least at this stage, have products ready.”

Content is (Lac)King

No UHDTV content is available to consumers currently. But the technology is available to create 4K content. Increasingly, feature films are shot with 4K cameras and are mastered at 4K for theatrical distribution. 4K consumer camcorders are already on the market. Classic films archived on 35mm film could be retransferred at 4K for distribution. And gaming engines could fairly easily render images in 4K or 8K. So the technology is available, but until a compelling market opportunity exists, content owners may not jump on board.

Upscaling and Other Sources

Early UHDTV sets from vendors like Samsung, Sony and Toshiba, will support upscaling of HD content to 4K. For example, Sony recently announced plans for a “Mastered in 4K” Blu-ray library that will offer content mastered in 4K that is downscaled to 1080p HD for Blu-ray release.

“When upscaled via the Sony 4K Ultra HD TVs, these discs serve as an ideal way for consumers to experience near-4K picture quality,” according to Sony.

While this may seem like a sketchy marketing ploy (because the discs are no different from today’s Blu-ray titles) I have to admit that standard def DVD’s look much better when upscaled to HD, so there is a possibility that Blu-ray discs will look even better when upscaled to UHDTV. Still, upscaling of 1080p content is at best a stopgap until UHDTV-native content becomes available.

One source of UHDTV-native content could be your computer. Graphics cards have long since surpassed the 1920×1080 resolution of HD. Home movies and photo montages will look great on a huge UHDTV. I can also imagine UHDTV’s becoming commercially viable as high-end displays for retail and corporate applications. But for television viewing, the lack of native UHDTV content will definitely be a barrier to adoption.

Bandwidth and Distribution

At 4K, UHDTV content has more than four times as many pixels as HD. At 8K, that multiplier jumps to >16x. Carrying that much data will require significant advances in compression technology as well as increases in network bandwidth. Ironically, at a time when sales of physical media are plummeting, these challenges may make physical media the most practical method for distributing UHDTV for the next few years. Recently, the Blu-Ray Disk Association has formed a task force to study the viability of extending the Blu-ray format to support 4k.

As for television, some broadcasters seem interested in exploring UHDTV. Recently, satellite operator Eutelsat Communications launched a demonstration 4K channel in Europe. And an experimental Ultra HD channel is also being planned in Korea. NHK in Japan hopes to begin experimental 8K broadcasts in 2020. But these technology experiments are primarily focused on exploring the technical viability of UHDTV broadcast.

Business Model

Market viability is perhaps the biggest question surrounding UHDTV broadcast. As with the 3D hype two years ago, some industry analysts expect sports programming to drive consumer interest in UHDTV.

The Hollywood Reporter recently stated that “BSkyB in the UK, Sky Deutschland in Germany, Japan’s Sky Perfect Jsat, and Brazil’s TV Globo have all started to explore the potential of 4K, which would include coverage of events such as sports…with an eye toward offering the 2014 FIFA World Cup and 2016 Olympics (both of which will be held in Brazil) in [UHDTV].”

But other broadcasters have gone on record as much more jaundiced about the market opportunity. During the Broadband Unlimited conference at CES last Monday, Sheau Ng, VP of research and development for NBC Universal says there’s no business model for UHD TV yet.

“Therein lies the rub,” Ng said. “It’s not the technology, it’s the business model. Where is the money? Unlike the previous revolution of HD, we have the device manufacturers selling the device when people are still scratching their head and saying ‘What do I do?’ That’s something we’re wrestling with every day. For us to say ‘We’re going to do this,’ we need somebody to say ‘here’s the business model, here’s the number of devices in the market, here’s how we’re going to make money.’”

And Bryan Burns, VP of strategic business planning and development at ESPN, hinted that some broadcasters will wait for 8k.

“By the time we get [to 4K] we will be on to 8K or whatever. I don’t want to make the capital investment [in 4K]. There might be a gradual evolution…but I don’t see us heading to 4K production or an ESPN 4K channel.”

In Summary…

With the proliferation of high resolution cameras and displays, the question surrounding UHDTV is no longer “how,” so much as “why” and “when.” From my perspective, until there is critical mass of UHDTV sets installed in homes, content owners won’t spend the extra dollars to make UHDTV content available. Until the content is available, broadcasters won’t create UHDTV channels. And until a lot of compelling content is both available and affordable, consumers won’t pay extra for UHDTV sets or service.

If this circular “chicken and egg” dynamic sounds to you a lot like the uphill battle faced by HD technology fifteen years ago, or 3D TV two years ago, I think you’ve gotten my point: UHDTV is definitely coming, but not quickly.

1. Greater Efficiency. H.265 should require only half the data rate of H.264 to deliver equivalent perceived quality.
2. Increased Spatial Resolution. Supported image sizes span from as small as QVGA (320×240) all the way up to 8K (7680×8320) for UHDTV.
3. Improved noise level, color gamut, and dynamic range as compared to H.264.
4. Improved methods for parallel processing.

In subjective testing, HEVC has delivered equivalent perceived image quality at half the data rate, when compared to H.264 (High Profile). This means that HEVC could enable web and mobile devices to consume one half their current network bandwidth when streaming video. Or this increase in efficiency could be used to significantly improve image quality at the same data rate. Either way, H.265 will be a boon to web and mobile delivery.

How does HEVC accomplish this drastic increase in efficiency? Although based on H.264, HEVC offers many small tweaks that add up, including the following:

• HEVC replaces macroblocks with a more efficient (but also complex) hierarchical system for partitioning frames.
• HEVC provides larger block sizes for higher coding efficiency.
• HEVC supports tiling, allows multiple encoder instances to work on the same frame simultaneously.
• HEVC supports wavefront parallel processing, so multiple threads can process different slices of frames more efficiently.
• HEVC is progressive-scan only, simplifying decoder implementations.
• HEVC includes entropy coding algorithmic enhancements that enable hardware decoders to run more efficiently.
• HEVC includes higher precision filtering for improved motion compensation.

HEVC is expected to be fully ratified and published by early 2013. And, although chip vendors may not finalize hardware implementations until the standard is fully baked, software implementations will likely see commercialization in 2013. At IBC earlier this month, several vendors publicly demonstrated HEVC prototypes and announced HEVC support in future products:

• Ericsson has announced and demonstrated the SVP 5500 HEVC encoder, aimed at real time encoding for mobile delivery.
• Allegro DVT has announced two HEVC broadcast encoders, the AL1200 (HD-SDI input) and AL2200 (IP input).
• Vanguard Software Solutions (VSS) has announced an x86-based software encoder that can encode 1080p/30 in real time on a single Intel Xeon processor.
• Rovi has announced that the MainConcept SDK will be updated to include HEVC encode and decode in early 2013, after the specification is finalized.
• ATEME has demonstrated a prototype of an HEVC encoder that can encode UHDTV (2160p) at 60 fps with an average rate of 15 Mbit/s. The commercial product is expected to release in October 2013.

What’s the catch? Licensing is still an unknown at this point. MPEG-LA has recently issued a call for patents that read on HEVC, so there is a possibility that H.265 will require a licensing fee. While MPEG-LA has not charged a license for H.264 when used to distribute video for free on the web, other uses of H.264 have incurred licensing costs. In any case, it seems likely that most H.265 implementations will require licensing fees.

There’s never a good time to introduce a new video codec, but it’s easy to argue that HEVC comes at just the right time. Video distribution now consumes the vast majority of all network traffic. Video-capable smart phones and tablets are ubiquitous. Unlimited mobile data plans are a thing of the past. Here’s hoping that licensing hurdles do not prevent the adoption of a new technology whose time has come!

But, like many people, I confess that I’ve been treating Super Hi-Vision like a science experiment – something cool and futuristic, but impractical and over-the-top (no pun intended). Imagine my surprise to see that last month Super Hi-Vision has spawned a bona fide ITU standard (Ultra-High Definition, ITU-R BT.2020). What’s more, at the IBC convention in Amsterdam last weekend, the NHK was showcasing a Super High-Vision camera that is actually a lot smaller than a washing machine! Some broadcasters and vendors were even speaking up in support of UHDTV as the “next big thing,” although not without lively debate from the doubters. I found myself wondering if maybe UHDTV might be on the brink of becoming more than just a science experiment.

I also wondered what UHDTV actually is, what variants it supports, and whether it is equivalent to Super Hi-Vision. This post provides a quick summary of UHDTV, what it is and what it isn’t. In a future post I’ll opine about who (if anybody) really needs UHDTV, but first it would help just to understand it.

UHDTV is a specification for a new motion image format derived from Super Hi-Vision, originally developed by NHK’s Science & Technology Research Lab. UHDTV specifies two image sizes: UHDTV1 has 4K resolution, while UHDTV2 has 8K resolution, that can run at a variety of frame rates. Super Hi-Vision is not exactly equivalent to UHDTV2, although it incorporates one variant of UHDTV2 as its image format. Here is a chart I put together to illustrate the areas of equivalence (marked in green) and variance between UHDTV and Super Hi-Vision:

Basically, Super Hi-Vision is equivalent to one of the variants of UHDTV (4320p running at 120 fps) plus 22.2 channels of audio. From my reading, ITU-R Rec. 2020 doesn’t specify anything about audio. Sadly, it appears that the ITU has decided that UHDTV should specify 18 permutations of image size and frame rate. That number increases when you consider color sampling and coding options. It seems to me the ITU should have taken this opportunity to simplify things, rather than repeating the unfortunate approach taken by the ATSC twenty years ago that resulted in fifteen HD variants.

To illustrate the spatial resolution of UHDTV, I have created a colorful graphic (see below) that accurately depicts the relative sizes of SD, HD and UHDTV images. By comparison, whereas an individual frame of 1080p HD content contains roughly 2.1 megapixels, a single frame of UHDTV1 contains roughly 8.3 megapixels, and UHDTV2 contains 33.2 megapixels. How’s that for spatial resolution?

I have always found it interesting that motion imagery can get away with much lower spatial resolution than still imagery. Although scientists have described two phenomena that explain the human perception of motion (the phi phenomenon and beta movement) recent studies indicate that human interpretation of motion video is more complex than had been previously thought. It turns out the brain is not a camera capturing a fixed number of frames per second. The brain has much more work to do to interpret motion video than a still image, and therefore doesn’t have the cycles to ascertain spatial resolution in each passing frame.

This explains why a Bluray disc looks great on my TV when 24 frames per second are flying by. But an individual 1080p frame, at only 2 megapixels, can look blocky. And at only 24 frames per second, fast motion looks jerky. Clearly UHDTV does not have either of these limitations, given its massive spatial resolution and a temporal resolution that ranges up to 120 frames per second.

Not surprisingly, all this resolution comes at a cost. In case you’re wondering, at 120 frames per second, a UHDTV2 sensor is spitting out roughly 4 billion pixels per second at data rates that can exceed 40 Gbit/s. That’s a lot of data! Imagine having to gang together fourteen 3 Gbps dual link SDI signals to carry one uncompressed stream of UHDTV2! This explains why HVEC encoding is seen as necessary to efficiently store UHDTV streams, even though UHDTV is compression agnostic.

So, that’s the quick story on UHDTV. Wikipedia has a good article on UHDTV that I recommend you read, especially if you’re interested in the history of the format. In a future post, I’ll opine on the more difficult questions: is UHDTV practical? And who really needs UHDTV? Stay tuned…