By Mary Teresa Bitti

Detective William Murdoch is always ahead of his time. For 13 seasons of Murdoch Mysteries, technology has been key to his quest to solve Toronto’s most complex Victorian-era crimes. The serial inventor developed a scope to see into the darkest night, a vest to deflect flying bullets and even a fanciful device called a “tele-vision” to transmit moving images.

Now Toronto-based producer Shaftesbury is flipping the script of its flagship series, giving fans the opportunity to work to solve a mystery with an augmented reality (AR) version of one of TV’s favourite characters.

The award-winning producer will be filming 360-degree photo-realistic video of the CBC juggernaut at Metastage, a mixed reality capture studio in Los Angeles, in April. The footage will appear in an AR Shaftesbury game set to launch globally in September. Designed for mobile phones and tablets, some content will be available for free, while a nominal fee (at the time of writing, a price had not been finalized) will unlock the full AR experience. It promises personal interaction with a realistic 360-degree video version of the lead character. (So close you can see Murdoch’s virtual shadow.)

The goal is to both attract a new audience and give existing viewers another way to engage, says Ted Biggs, the company’s VP of convergent technology. Biggs joined Shaftesbury in 2014 and has been working on digital convergent projects in support of television for about 18 years.  “We’re pushing the boundaries of how to use AR as a narrative storytelling device and using the technology in ways that haven’t been used before,” he says. For that reason, he expects AR aficionados will also give it a go. “They’ll have cool new content to play with.”

Is AR just cool or is it an important new tool going forward? While the experts say that’s not yet clear, AR and to some extent virtual reality (VR) present an opportunity for TV broadcasters and streamers to engage viewers and, more specifically, younger viewers who use mobile devices to consume entertainment.

The latest report on the audiovisual industry by the Canada Media Fund (CMF) shows the urgency for linear TV broadcasters to stay relevant. Consider the numbers: In G8 countries, linear TV viewing decreased 7.72% between 2016 and 2019 while online streaming jumped 28.39%, with smartphones the overwhelming device of choice (89.8% versus 80.5% for PCs/laptops and 36.2% for smart TVs), according to the report Closer, Wider, Faster.

“There have to be new ways to connect with audiences,” says Jill Rosenberg, co-founder of Idea Central, a digital market and content strategy company based in Toronto. “The opportunity with AR and VR is to connect with the right audience in a way that will get them interested in what you have: your IP. Those who get into the space earlier will be quicker to monetize.”

According to B2B research firm MarketsandMarkets, the global AR market is projected to grow from US$10.7 billion in 2019 to US$72.7 billion by 2024. That’s across all kinds of different applications across industries. “If you have an ability to tell stories, AR is another tool to get those stories out,” says Biggs. “With the advent of mobile AR and webAR [which can be accessed directly from a mobile browser – no need to download an app or filter], you can bring the world of your television show into people’s homes in a way you couldn’t before.”

The CMF credits the promise of 5G connectivity. With 20 times faster download speeds and almost zero latency, 5G is expected to facilitate the ability to stream content onto head-mounted displays and have a game engine driven by artificial intelligence responding in real time to adapt the game to individual users. The feeling that the time is now for AR was certainly prevalent at CES 2020.

“In some ways, AR has leapfrogged VR from a consumer standpoint because it’s more accessible,” says Rosenberg. This is thanks to the 1.5 billion mobile phones sold each year and to tech giants Apple, Google, Facebook and Snapchat releasing platforms and toolkits for developers to create their own AR. “Many brands and entertainment companies have used AR to allow people to turn themselves into Batman or a Marvel character,” she says.

Thinkingbox, an interactive production studio based in Vancouver, has done several AR apps for clients, including an AR Facebook filter for the Warner Bros. movie The Meg. The filter lets fans “capture” the 75-foot shark as it stalks local water. The promotion resulted in 100,000-plus Facebook impressions, 20,000-plus captures and 15,000-plus shares. “One of the biggest challenges of marketing a new show is getting reach organically,” says Christine Clark, managing director at Thinkingbox. “Unless you have a really good reason for someone to want to download an app with AR functionality, they won’t do it. Social platforms already have a user base that can be targeted. If you have something interesting, people will share it.”

The cost of creating something interesting will vary greatly depending on the scope of the project and what you put inside. For example, very basic animation AR experiences for game apps can be created for tens of thousands of dollars. The higher the quality of the graphics and design of the game-play, and the more sophisticated the technology, the greater the cost. Once the expense of marketing, creative, actors, props, etc. are factored in, the total cost can easily reach into the hundreds of thousands of dollars and more.

For its AR game project, Shaftesbury received $485,608 from the CMF to help finance the boundary-pushing game, which uses leading-edge technology from Microsoft. CBC is also a collaborator.

“We have a history of award-winning collaboration with Shaftesbury on innovative Murdoch Mysteries projects such as Nightmare on Queen Street and The Infernal Device,” says Chris Harris, CBC’s director, digital content, programming. “We’re excited to see how fans will participate in this new experience – while also allowing us to learn more about how our audiences will interact in AR environments.”

To this point, AR in the TV and film entertainment space has been a novelty, used to drive publicity and learning. “For marketing, it makes sense. It’s a quick interaction and fun,” says Clark. Plus research suggests AR enhances engagement as well as memory of the experience as compared to viewing a flat advertisement. “The question is how much interaction do viewers want, especially when that interaction involves holding a phone or tablet while watching a TV program. We have to think about how people want to interact with content. Is AR a longform storytelling platform? Maybe, but I haven’t seen it yet.”

The audience is already moving between different platforms; why wouldn’t we do the same? – Ted Biggs, Shaftesbury’s VP of convergent technology (pictured)

The audience has not yet been defined, says Paul Gray, research director with IHS Markit in the U.K. “Up until now people have just been trying to master the technology. Beyond that, it’s, ‘how do you create something compelling,’ because it isn’t like normal TV viewing. Broadcasters are experimenting with what adds to the enjoyment of the story and what is just noise,” he says.

Getting into the game

For Mrugesh Desai, regional VP, North America, at Accedo, perhaps the most natural audience for AR-enhanced programming is sports fans. “They tend to care about all facets of what the broadcaster can provide. It’s not just video, it’s also the stats, schedule, scores, the players, the stadiums. They tend to be younger, and are already adopting technology at a faster rate than the average audience. If they’re gamers, they’re used to wearing headsets,” he says.

Not surprisingly, sports brands are experimenting with AR and VR to create better in-stadium experiences. “These experiences aren’t directly monetized but they make the brand seem more innovative and prime the audience for adopting the technology in the future,” says Desai.

To see what that augmented future might look like, Accedo partnered with France Télévisions to create an AR experience for tennis. The project used holographic video capture to allow the viewer to move freely over the tennis court and view the game from any distance and any angle. As part of a cycling AR concept, Accedo rendered the topography of the mountains where the race was being staged.

“You could follow the different pelatons and see the standings and the stats but you could also see the mountains, the incline,” says Desai. “All it requires is a 3D modelling skillset. Lots of studios provide that and create it for their own content production needs, so they don’t have to reinvent the wheel. We can take those 3D assets and render it into an AR app experience.”

For properties that haven’t been created with AR or VR technology in mind, repurposing existing content is more challenging. “You need to plan the shoot to be 360 degrees,” says Clark. That’s exactly what Thinkingbox did when it created a VR promotion to launch season five of Corus Entertainment’s Vikings.

Thinkingbox shot five scenes, some adapted, some new, in 360-degree video and developed a VR app to engage super fans, who had to use their phones and Google Cardboard or an Oculus headset to see them.

“The link is always on storytelling,” says Biggs. “We approach the story slightly differently for every medium but it’s still a story. Shaftesbury is  working on several VR projects and one in particular that has the potential to become a television show. Wrest, a sci-fi/horror VR game, is scheduled for release next month. “It sets up a chilling, dystopian future with really great character throughlines that are not typical for a lot of VR projects.”

Like its AR game app, Wrest is part of Shaftesbury’s strategy to use new formats/media to find and engage new audiences. In this case, primarily 18- to 35-year-old gamers skewing male. If it’s successful, the story will make its way from VR headsets to television screens. “The audience is already moving between different platforms; why wouldn’t we do the same?” says Biggs.

Certainly the precedent has been set. Many fans of the Netflix series The Witcher were first introduced to this fantasy world via the video-game series. And Sonic the Hedgehog is just the latest video-game adaptation to become a major motion picture. Despite production delays and mixed reviews it brought in US$58.3 million domestically during its opening weekend and more than US$265 million globally as of March 2.

One of the biggest knocks on AR and VR is that the user experience is not seamless. According to Desai, that’s about to change. The expectation is that within the next five years, AR and VR will converge and the types of headsets will be much less invasive, which will create a more compelling reason for viewers to give AR and VR content a try.

Solution looking for a problem

But is that enough to garner mass appeal? Duncan Stewart, director of research for technology, media and telecommunications at Deloitte, isn’t convinced. “AR has been around for years now. Tens of billions of dollars have been invested in extended reality technologies, however, almost all of that money has not translated into considerable amounts of usage at the consumer or enterprise level,” he says.

“Pokémon Go was genuinely a phenomenon [since its launch in 2016, it has been downloaded more than 1 billion times and as of April 2019, generated about $200,000 a day in revenue]. The only problem is most of the people who played it shut off the AR features in the first hour or so, since they found it consumed battery life and detracted from game play.  I keep reading about other applications, such as people trying on clothes using AR or Ikea’s AR app that lets you see how furniture might look in your home, but all the data I’ve seen suggests consumers are not using it.”

He says the same is true of enterprise AR and VR applications in industries such as health care, manufacturing and the military, where the efficiencies seem obvious and people are used to wearing goggles, visors or some sort of mask. “There have been a few trials, lots of excitement, but few people actually using it. In general, AR is a solution looking for a problem to solve,” says Stewart.

Still, there is no shortage of potential use cases. For example, when it comes to producing content for TV, Jon Finkelstein, executive creative director at PwC Canada, sees opportunities for cost savings. “Imagine a set decorator having the same opportunity that Ikea’s Place AR enables and being able to lay out a set ahead of time to avoid any overruns or mistakes.”

“When I think about the proliferation of streaming services, I wonder whether the smart studios will be creating AR companion experiences that drive preference to one over another, leading to a premium on the subscription,” he says. “But the payoff has to be awesome. Otherwise, why would I do it?”

Biggs and Shaftesbury are committed to 360-degree engagement and ensuring their content lives where their audience does. “People are going to be looking at different types of media and devices every day. We might as well be available to them when they do that with a particular TV franchise.”

What’s the advantage of AR?

Both augmented reality (AR) and virtual reality (VR) have a bright future, but experts say AR represents a more dynamic world of opportunity:

1. It’s easier to develop than VR, which requires a great deal of programming and photo-realistic graphics in order to create a fully immersive virtual world.
2. It takes less time and money to develop since AR adds a digital layer on top of a real environment and users can continue to interact with their surroundings.
3. There’s a bigger opportunity on mobile devices than VR, which favours consoles and PCs.

Images courtesy of Shaftesbury