THE WORLD OF AR

Hello, this is Les Goldberg, and welcome to the Road Ahead. The Road Ahead podcast is dedicated to the future of the live events business, bringing together industry experts. Hello, production world, this is Les Goldberg and The Road Ahead. Today I have three amazing guests. I have Jeff Moore, who's the executive vice president and chief marketing officer for Ross Video. I have Berto Mora, who is a partner and head of research for All Of It Now, a content production company. And I also have Sean Borowski, who's the director of research and development for Brainbox at ETP company. Hey guys, welcome to the show. I'm glad you're here today. Our subject is augmented reality. We're going to kind of focus on one version of all the realities we live in. And so today, all of our discussion will be dedicated to augmented reality. So I'm going to throw these questions directly to Berto. And the question is what is augmented reality and how does it differ from the other realities that we have. Yeah. So augmented reality, also known as AR, you know, is a way of, you know, superimposing visual graphics over, you know, our reality. So our realities are physical space. You know, with the technology today, we actually have, you know, headsets that like glasses, you see in a variety of these different technologies, you know, in the past where we can actually wear them and, you know, composite graphics over them. So that's kind of the most basic form of augmented reality. And, you know, it's it starts to get a little bit more complicated once you start specializing the augmented reality and the virtual space or the physical space together and merging them together. So one of the biggest things with augmented reality that you can do as well is interact with the graphics and the real world. So, you know, we can, for example, you know, walk down the street and start seeing virtual billboards. This is something that we're going to see in the future. So in short, augmented reality is just, you know, compositing 3D graphics or graphics within our physical world. You know, I am so excited about seeing billboards. I'm going to have to have some special glasses, obviously, if I'm going to see these cool billboards. But I have a sign, I think I'll look at a business and it'll have a sign like 99 cent Starbucks coffee or something, and with my special glasses. So I'm going to throw this question to Sean. Sean, what is the ideal use case for AR in live events? You know, most individuals are familiar with augmented reality in the form of a mobile device. You know, popular applications like Pokèmon Go or the filters over your Instagram or Snapchat pictures are how people are familiar with a kind of first use of this technology. But as we enter into live event space, as as we are getting our message across to a an audience, whether the audience is in front of us or that audience is remotely across the internet or broadcast, this is an opportunity to use more impactful graphics to have a presenter really illustrate their message visually in a new and unique way. If you can imagine announcing a new pharmaceutical product and having the DNA strand kind of spinning in space where a presenter can interact with it, or to have a new physical product like a automotive part, where we can deconstruct it in the space in front of you, not just kind of on a PowerPoint slide, but but in space in front of you as the presenter kind of points at this widget and that gear. These are some really, really awesome opportunities to utilize augmented reality in a, in a performance and in a presentational venue. You know, that sounds super cool. Sean. The DNA strands and the deconstructing cars. You know, I'm going to throw this to Jeff because, Jeff, you know, a lot of us are there talking about very creative things from the perspective of the manufacturer who has to build machines that actually help create augmented reality. How do you go about doing it knowing as they've described these use cases, and obviously we're a partner with Ross and the Voyager solutions for augmented reality, and we use them regularly. And so could you give us maybe from the manufacturers perspective, when you have creative folks and technical folks, how do you go about building things that allow us to do this right? I think as part art and part science in all of this and making it all work. And one of the big things for us is, is getting that feedback right about from the users, from, you know, guys like Berto and Sean, they're actually where the rubber meets the road in this technology and feeding that back into our products to make them more usable, easier to set up, more realistic looking. So we spend a lot of our time and energy on that and then also providing a backstop. We have expertise in the House to, to bring, to bear, to help solve problems in as we as we roll these things out. The technology is has gotten a lot easier over the years and a lot better in terms of what you can do with it. But there are obviously it's it's still, you know, somewhat exotic technology and you need the right expertise to, to make it work. Whenever you're dealing with new technology, it's always great to have the manufacturer behind you so you can overcome those obstacles. And I can say you've done that with us on a few occasions already, but I'm going to throw this to Berto. What are some of the misconceptions about using augmented reality in live events? You know, maybe things that people don't understand. Yeah. So, you know, we've done augmented reality for a few projects. I think one of the most successful ones was the BTS Korean K-pop group, and we did an AR tour for that project. So, you know, they went around the world, they would set up their system, and from there, you know, we would show that on the Imax screen. A lot of people think that when they when they it's really interesting because a lot of people think augmented reality is just going to appear in front of their eyes. But as Sean mentioned, the most common way of seeing this is through a mobile device. So a lot of the things in, you know, we're used to seeing content today on Imax screens, large video wall screens in your concerts. But how do you showcase augmented reality in that way? So with us, a lot of our clients are like, well, how are people are going to see it? Are they going to see it through their phone, or are they going to see it on stage? Obviously they can't see it on stage, but they will see it on the iMac screen composited for for the entire audience. So I think that's the biggest misconception is people think augmented reality is really and I don't know how they think this, but they think it's just going to appear on stage. And that's something that just doesn't happen with the technology just yet. Abracadabra, I want it to appear poof. I guess it doesn't work that way. So. So throw this question to Sean. Does augmented reality in a virtual set experience require any special camera equipment? You know, if you're trying to create augmented reality, Sean, do you require a special cameras or do regular cameras work? Boy, does it ever, you know, so it's a combination normal broadcast or film. City style cameras can be used. Gosh, even mobile phone cameras can be used. However, there's a need for 3D spatial knowledge that's required. And if you think about an object floating in front of you. We need to know where you are. We need to know where the camera is, and we need to know where the object is going to be. And so we have to supplement the cameras that we use for traditional tools like broadcast or filmmaking or advertisement with special tracking sensors in order to know where the cameras are. And then if we want to, to to absolutely track the objects with accuracy, we have to add additional sensors in order to know where those objects are going to be. That sounds like a lot of knowledge and a lot of data and zeros and ones knowing you know exactly where things are for it really to work. Right. So I'm going to throw this to Jeff. Jeff, what would you say in your opinion is the most important technical challenge and deploying augmented reality for live events that you've experienced from the manufacturer of side? Well, there's a number of them. So one of them, Sean was just mentioning was tracking getting the tracking rights really important. So we need to understand, you know, the camera image that's coming from the real world, the optical, we'll call it the optical world versus the digital world. The digital world has to be rendered to match the optical world. So tracking super important. If we get it wrong then it doesn't look correct or objects don't show up where they're supposed to be. So that's one thing I think getting the creative elements like like getting the creative elements figured out so that they make sense, that the composited image is going to look the way you expect it to is is a big part of it as well. And that's partly the artistic part that comes in as well. Right? And so it's it's using the technology in a way that's going to achieve the result that you're going going for. Cool. Well, well that's sounds it sounds super technical that you have to kind of know where your, your, your optical view is and where your computer generated objects are and, and keep everything in the right place and track. And then there is lighting. Oh my goodness. And then there's lighting. So I'm going to throw this to Berto. Are there any creative limits to what can be displayed in an augmented reality or a virtual set experience? I mean, I don't I mean, today I think we're all we're really limited by technology. And a big limitation also is occlusion is. So explain it. Occlusion. Just so everybody knows, occlusion I think it's so occlusion is a way of kind of I mean there's different versions of this, but there's occlusions of being able to have your graphic and your physical world match together. As Jeff said, lighting is very important. So whether you have just let's talk about a sphere here in our 3D scene and, you know, we have physical light from our world. As you can see, when you have a sphere or an image, it casts a light from one direction. So if we look at our virtual, at our physical world, the lighting is, you know, casted from one direction, we should be able to match that. But not only that, occlusion also should is a way of occluding the 3D graphic inside your physical space. So for example, one of the hardest things to do today with AR is having a person walk behind and in front of an object. There's different ways of achieving that, but today we are working more in a 2D pixel space than a 3D pixel space. So I think that occlusion is going to get better once we have better AI depth occlusion and depth sensors and volumetric cameras. Maybe some light field cameras that can help that. But not only that, occlusion is also reflection. How do we get the reflect, the reflection of the real world onto a physical object or virtual object and vice versa? Wow, that is a very interesting answer. And I don't know how I feel like I just geeked out a little bit on my occlusion, but I asked the question and and it was a very constructive answer. So I'm going to throw this to Sean. Sean, how can presenters and performers interact with AR objects on stage? So even though I'm putting my hands out to magically hold a bucket of something, maybe it's a bucket of beer or money. Whatever. How does that really work? Absolutely. So as Berto just mentioned, occlusion becomes a big part of it. We need to know that the object is in front of the real person or behind the real person or within the virtual set. So kind of understanding that layering of where things are is very important to interacting with virtual elements. A trick that we're able to do is we are able to use trackers, GPS locators or IR locators where we can know where your hand is by the dot. And anyone that's watching any like behind the scenes footage of making films and those motion capture suits that they wear, you know, knowing where the parts of your body are in 3D space is very important to being able to interact with virtual objects that really aren't there in front of you. They're going to show up after camera, they're going to show up on screen. But but they're not there. So as we work with presenters, it's very important to provide them some confidence of what they're actually doing. And so with the, you know, very, very, very close to real time technology that's available right now, we can provide confidence monitors for a presenter or an entertainer or a performer on stage to be able to see how they're interacting with a virtual element. Again, to go back to my metaphor and an example of a DNA strand, you know that the presenter knows that they're spinning and turning the DNA strand because they're looking at it on a monitor in front of them as as part of their performance or presentation. That sounds super cool. I wonder how difficult it is for the performers to learn to adapt to this. How do I say this imaginary world of objects? You know, it seems like that that seems like a little leap of faith. They have to just kind of get over the hump. So, Jeff, how long have you guys been making augmented reality equipment and how long has have you guys been in that business? And, and what have you seen how the businesses evolved so far? Yeah. So we've been added for, I don't know, probably 7 or 8 years now, I guess. You know, one of the big things that's changed is rendering. So the capabilities of the render engines and the sophistication of the objects that we can generate, the virtual environments, the augmented reality elements, it's night and day from, you know, from years ago, it's and it's gotten so much easier. Some of the things that we used to have to bake into the models now render in real time. So the capabilities have really changed as a result of game engine render technology. So the same stuff, the exact same stuff that is being used for these hyper realistic gaming systems. We're using that to generate virtual and augmented reality now. And so you're saying you're playing a big video game in real time in a live presentation or a live production? Absolutely. And we can do we can push the boundaries. We can do, you know, raindrops, we can do like we've got access to physic, physical elements. The example Sean mentioned earlier of product launches and things like that. Well, you could have a model of a car on stage next to you without actually physically having to bring the car in, and it's going to look like the real car. You just saved a lot of money not bringing a car in and and driving it in and having to take the battery and unplug it and the gas and all that stuff. You know, this imaginary world might have some cost savings. And one of the big things now, you know, with, with LED screens is we're getting into driving the backgrounds and the LED screens with these environments and having them track the cameras in front. So now we can generate these, these kind of immersive environments and do Mandalorian style using air quotes. Productions of people have heard about how that's how that work is being done. And we can bring that to other, other types of live events and productions. And it's really cool what's possible right now. Everybody's remote. Right? And so the in-person, physical live events aren't happening so much these days, but you can generate a really cool. Remote live event using this technology in a, in a super compelling way and have amazing production values on it. I like the term amazing production values that like totally jazz me Jeff you just those the hash. That's a hashtag. So I'm going to throw this to Berto. Berto. What would a successful approach for a creative team and a technical equipment provider work together to deploy a live AR event? What would you design that approach or what's the best way to make it work? Yeah, I think, you know, with our experience, one of the biggest things that I think is very important is, you know, gathering all the teams involved, whether it's a camera director, stage managers, content creation team, even the people that are building the actual stage, you know, having all these people get together and really have a discussion we've actually done as we know, this technology is, you know, very pricey. So it's it, you know, you want it to go as well as possible. So I think the biggest thing is, yeah, you know, educating everyone that's a part of the production to be able to, you know, create the ultimate experience. We were on shows before where, you know, the director didn't even know that we were doing augmented reality. Another thing, too, is that they don't even know how to cut between these cameras. Once you add a layer of AR, everything just seems that to become very complicated and directors don't understand how that works. So I think educating everyone that is a part of that, that specific show or implementation of AR is one of the most important things I would say to make it successful. Okay, guys. Well, this this is my last question, and I'm going to throw it out for all of you to opine. I'm going to start with Sean. You know, what developments can we look forward for AR into the future. And do you think AR will become a bigger part of our world? So, Sean, why don't you opine? Absolutely. I believe that AR is going to be a part of most all productions moving forward. And I use this example of when these new technologies are done well, they just become a part of our how we understand the workflow. So if you look at a professional football, for years we have been looking at tracking markers and lines and putting dots underneath players and extruding titles in a broadcast workflow. And the audience doesn't even think about that anymore. It's just a natural part of watching the sport. And so I truly feel that as we move forward in entertainment, as we move forward in more corporate productions, as we move forward in creative productions, these elements that that accentuate and augment our experience are just going to become a more and more regular part of our process. Okay, Jeff, your thoughts add to what Sean had to say. And then Berto, you can bring us home. Well, it's hard to top that. I concur with Sean. I think, you know, one of the key things is tracking from cameras and a variety of ways of doing that. And the more that becomes available and ubiquitous, the easier it is to implement AR in any type of production. And why not? I mean, if you can enhance the the look of the production, sell stories more easily and better than you know, I think we'll see more and more of this just embedded. And I agree that that we'll stop thinking about it that way. You just it'll just be something that you'll come to expect. We are going to have high expectations in the future of really cool stuff. So, Berto, take me into the future. Take me into your crystal ball, and let's just go 2 to 5 years from now. Oh, your crystal ball is round. Okay, so take me into the future of augmented reality. And what cool things will people expect? Yeah. So, you know, I think, you know, I think a lot I don't know if it's right, but yeah, I think the future of augmented reality is that it's going to be everywhere. You know, I'm not sure if in five years we're going to get to the point where we actually where contacts and that is our AR device. But there is companies I believe there's a company called Mojo that's working on that. And yeah, I think that in the future augmented reality, you know, it's great for entertainment, but also it's going to aid humanity, I hope in a way where, you know, if someone can't see, they can actually have these virtual glasses that are connected to the Neuralink, that can actually speak to your brain so that you can actually see the world, right? Because, yeah, I mean, I think that's going to be a really crazy future that we're going to live in, and it's a lot closer than we think. You know, a lot of AI implementations in the technology are going to aid it as well. And the biggest thing here is, you know, making it more affordable for everyone at home to actually experience the technology and not just larger productions that we're a part of. And we're fortunate of that. But I would love for everyone in the world to experience augmented reality in one shape or form. Well, guys, today I got totally jazzed about learning and hearing your thoughts on augmented reality in the future. And I think it's coming. Ready or not, whether we like it or not. And it sounds like it's going to make our worlds cooler, better, and we're going to have greater experiences. I want to thank Jeff Moore from Ross, Berto Mora from All Of It Now, and Sean Borowski from Brainbox. You all are amazing people and keep all this cool technology rolling as we continue to develop productions into the future, and the world gets back to some form of normal. Thank you all again. This is Les Goldberg and The Road Ahead. Thank you very much. Les. Nice. Okay, thanks.