THE FUTURE OF LIVE WITH AUGMENTED REALITY

Hello, this is Les Goldberg, and welcome to The Road Ahead. The Road Ahead podcast is dedicated to the future of the live events business, bringing together industry experts. Hello, production world. This is Les Goldberg and The Road Ahead. Today, I have four amazing guests. I have Berto Mora. Berto is the CTO and partner at All of it Now. I have Danny Firpo. Danny is the CEO and co-founder of All of it Now. I have Sean Borowski, who's the Director of Engineering for Systems Innovation, an ETP company. I have Craig Mitchell. He's the Director of Touring for LMG Touring, another ETP company. Guys, welcome to the show. Great to be here. Hello, good morning. Thanks for having us. Guys, you're very welcome. I'm super excited about this podcast today. Today, we're going to be talking about something near and dear to my heart, augmented reality, AR, and how we're introducing AR into corporate touring, corporate events, touring events, installations, all kinds of shows, exhibitions. And I want to talk today about AR and what it is and how it works. So I'm going to throw the first question to Danny. Danny, can you give me the layman's version of how do you define AR, augmented reality? What is it in English, simple English for the audience? Yeah, I think augmented reality in its kind of format that we know now is essentially a ethereal, invisible layer of content of technology and creativity that can be kind of expressed in a camera feed that's sent to the audience. And what's historically known as AR is essentially a composited effect, something that goes over the top of the camera. But I think there's definitely some impressive developments that we've been working on, that other teams have been working on as well. To kind of better integrate those augmented effects into the environment. And what's kind of really special here is that we have, you know, an incredible piece of entertainment technology, the 3D model, the vector works, the drafting, the planning, all of those tools can really be used now in concert to create a really special augmented reality effect where we're kind of keeping the stage pieces in mind, keeping screens, keeping trussing, keeping all of that in context with the augmented reality experience and being able to create something, you know, pretty, pretty magical. I like that when we do magical things, very magical. So I'm going to throw the same question to Sean, but Sean, I want you to take it on a spin of from the Systems Innovation group, where we do an installation, you know, how is it maybe different? Danny is describing we're creating this environment where people see things. Is it becoming popular in the installation world where people are doing this permanently, or is this something that's very new? Indeed, it is less. AR from kind of a holistic perspective, I loved Danny's explanation, especially with context to event and broadcast presentation. But all of the major tech companies are chasing after the goal of providing an extension of your reality in front of you, overlaid on top of what you visibly see in your day-to-day interactions. That can be done with headsets, that can be done with holograms, that can be done with other screen surfaces that you see as you come into contact with retail and with commercial and with educational locations. And so in the integration market, we're very much experiencing this drive to try and expand what we're able to show the customer, show the student how we're able to allow them to engage in new environments, especially as Danny spoke about, utilizing generative and real-time 3D modeling techniques to either utilize virtual environments or virtual elements in front of the consumer, in front of the student, in front of the viewer, in front of the customer. Got it. Well, I like the term engagement. It sounds like when you see things visually and you see all kinds of elements that it probably increases engagement. I'm throwing this question to Craig. Craig Mitchell, is this something that is very new for touring? Are people starting to use it? Is it like in the very early phases of this? Is this a touring product? Yeah, absolutely. You know, it's interesting. Snap just signed a deal with Live Nation, a multi-year deal to bring augmented reality experiences to events, especially festivals. We've seen in the past, the Coachella Dance Tent had an AR experience that was very popular on the live stream. We've seen some music TV award shows that used AR very effectively for expanding the set pieces in the backdrop. The U2 tour featured AR experiences, both before the show and during the show. And our own Danny and Berto on this call did some amazing work with BTS on their stadium tour. Maybe they can elaborate on that. Well, okay. We'll get back to that in one second. I just want to ask this question to Berto. Where do you see this technology deployed? What do you see as the most popular use of AR from what you're experiencing? And I have to preface this that Danny and Berto, all of it now, like they do this all the time. These are industry experts. They know their stuff. So Berto, where are you finding the most amount of applications in the broadcast side, in the touring side, in corporate side? Maybe you can enlighten us. Yeah, I think currently today, you know, we're seeing most of this AR work in the broadcast world. And we've seen a little bit of this in the past with like the NFL, yellow line and, you know, all these like stats and those types of components for augmented reality. In the past has been really, really expensive. But now what's going on, obviously it's still expensive, but it's gotten more affordable for prosumer companies. And yeah, I think, you know, broadcast world is really where we're seeing it the most. We're seeing more and more real-time graphics and render engines onsite with, you know, the camera also you, I think that broadcast also lends itself to have a little bit more time for setup where, you know, when we have concerts and, you know, events like that, you're very limited on the timing that you have. So, you know, augmented realities it's a little harder to do on a, on a smaller timeframe than a larger timeframe. And I also think, you know, right now we're seeing a lot of mobile applications. That's also existed for a while, but we're starting to see more and more mobile augmented reality. You know, and the goal today is, you know, to synchronize all the augmented realities from broadcast to mobile phone, to your computer, to your web browser. So I think that that's kind of where it's going and we're probably going to see it more and more in our pocket, right? Like everyone is going into AR today. It's more accessible. Cell phones are a lot more powerful. They can actually render the graphics today. Whereas in the past, it's been a little bit harder. So we're getting better sensors in our phones that allow us to actually be able to experience these experiences. At the moment, as I mentioned, sports is probably where we see it the most. I think touring is going to be next and eventually it's just going to be on your phone everywhere you go. So, so back to this, back to Craig on this touring question, Craig, are the touring people that are using applications, are they doing it for the live audience or the stream version? What can you maybe opine on that? Yeah, I would say, you know, to Berto's point right now in the touring scenario, it's very, you know, the timeframes to set up in the morning and get it working mainly for a live audience, you know, that's going to be a small group of people using that right at the moment until the technology matures a little bit. Where I think we're seeing it the most right now is in festivals that have maybe a live streaming component or a broadcast component, you know, a live event that also has a broadcast component so they can, you know, make it a bit more interesting for the viewers. You know, I believe some of the European festivals, Tomorrow World and stuff that really went into virtual during COVID or, you know, taking what they learned during COVID in virtual worlds and combining the two nowadays, and they're doing some really fantastic stuff. So I would see more of this by watching the stream and then maybe shooting it with my phone to see some additional experience, like a second screen. But I wouldn't, if I was in the live audience, would I see the same kind of experience or not? In the live audience, I think you're, you know, if you, if you're talking about a stadium show or arena show where a lot of the audience is watching the iMac screens, I think AR can be a very compelling experience you know, on the iMac screens because so much of the audience is watching that. And I'd also say in the terms of festivals and whatnot, AR is not going to be limited to the stage. AR is going to be how people find out where the bathrooms are. AR is going to find how people find out about special merch deals or special food to go to, or special VIP meet and greets. That's all going to shift to AR as we move forward, I believe. Got it. Okay. I'm shifting this question to Danny. Danny, you and your team and us, we did a really big technology event, maybe two or three weeks ago. It was a monster event. And well, I don't want to go into all the details. What I really want you to do is kind of talk to me about how someone starts the discussion about how I can add AR in my show. I mean, our show had, I don't know, 20, 30,000 people and massive screens and, and AR was being shot with these special red spy cameras and tracking and stuff. But, but taking me to the very first conversation when a client says, Danny, I want to add an AR element. How does that work? And maybe you could kind of talk to me about the approach of where you start and, and what, what the workflow path is and your discussions. So if someone were in the audience, we're a producer and they said, I want to do something really cool. Besides calling Danny Firpo, then, then like maybe you could give us a very layman's version of how do you start this discussion? Yeah. I think, you know starting the question off kind of conceptually, creatively with, you know, what are those, what are those event concepts that you wanted to do that, you know, that were just kind of too expensive or, or too impractical, right? If you wanted to hang, you know, a hundred foot parrot above the stage, but it just, you couldn't get together to, to build, fabricate or rig you know, something that massive you know, that's a great opportunity for AR, something to create that kind of weightless, you know, kind of ethereal object that costs nothing to ship, costs nothing to to, to store, you know, and can suddenly be pulled out on a whim to be able to deliver that as an augmented reality experience. You know, those kinds of impossible or impractical objects, I think are really great starting points for AR. And, you know, I think creative directors often have those kind of thousand, you know, mile high ideas, thousand ideas that they, that they want to include in a show, but you know, have, or have struggled to include in a show previously, but either the technology wasn't ready, their budget wasn't, you know, honestly fit a round peg into a square hole. And so AR is a great kind of opportunity for that because it doesn't carry the same practical and physical weight of, you know, kind of other set pieces like screens and lights. And so that is definitely a kind of a key consideration for starting that conversation of how, how to, how to kind of conceptualize what AR elements can do to elevate the show and do something that a practical element can't necessarily do. So, so Berto, my question is just to follow up from this one with Danny was if, if we have some creative treatment that we've agreed upon, how many days, weeks, months does it take for you to plan and, and, and work the workflow to get some, we'll call it a product that could hit the screens or the, or the devices? What's the, the, the, how do I say, what's the lead time? If you were planning a show, that would be the smartest timeframe you could advise. Definitely. This is more like a Danny question, but I'll definitely answer this. I mean, obviously the right question, the right answer is the most time that you can get because you never know what are the elements, what are the you know, ideas that the client wants to do. And a lot of that time you know, we start with like a previous session to be able to figure out what the graphics are going to be, where we're going to place these cameras. And then from there figure out kind of like the rendering process of like, do we have to optimize these products? Right. So to be honest, the most time that we, that we have, the better, the, the better the results going to be. Usually, you know, we tend to do, you know, based on the ideas at minimum a month. Cause that allows, allows us to kind of develop the idea come up with different creative concepts, figure out how we're going to be able to create those elements to render them and to actually produce them, whether they need to be produced and other, you know, 3D graphic engines that are not real time, like Cinema 40 or Maya, and then bring those into Unreal Engine. But, you know, I think the project we just did, it was a couple months of ideation and coming up with those ideas and in order to be able to put them on the screen. And not only that, once you develop them, you have to pre-vis them. Once you pre-vis them, then you have to test them. And it's not just like, Oh, these graphics are going to work. You actually have to test the materials. Translucency is one of the biggest problems with augmented reality. So you have to figure out how to come up with new ways to add translucency materials so that in itself can take weeks to develop based on. How about if we're on site, how much time on site? So we're loading in a show, we've got audio, video, lighting, LED screens, or whatever kind of projection. How much time do you allot for the AR segment of the program? Yeah. I mean, right now, you know, a calibration per camera, we're talking about four to five hours, you know, from the lens to be able to calibrate the actual mapping. You know, so it really depends on how many cameras you have. But you also not just have the cameras, you actually have your video village as well. I think for this last project, we did it in like four days for three cameras set up along with testing and being able to audition and add the concept. So just so I'm clear, when you have a camera and you're doing alignment, you're trying to align the camera to the environment to make sure the background is in sync with the camera. So when the camera moves, the background moves with it. Correct. So pretty much what we do is we take a 3D version of the world or a 3D object that we know the exact dimensions, where it is in space, and then from there we calibrate to that. So instead of having like a 2D calibration card, we have a 3D calibration card. And what we're able to do is we're able to place that card with a known object. In this instance, it was the stage. We knew how high it was off the floor. We knew the dimensions of the of the screen that was the floor. And we produce that same element in augmented reality. We now place that in our origin world of our 3D space of our graphics engine. And then from there did our calibration based on that. So what we would do is we remove our camera until actually aligned. Not only that, we have latency as well that we have to account for. So that's kind of a little bit of the process of aligning the virtual world to the physical world with the 3D camera data. Got it. I'm throwing this question to Sean. Sean, if someone wants to do installation of sophisticated something cool, how does the conversation start about AR? I know Danny gave us some of this stuff that was probably more towards live events, but in a permanent installation, how you begin the conversation? We certainly we start with the goal in mind. We talk about what they can't achieve with practical effects or practical elements. And so being able to lean into what we can generate in a 3D world, how they can experience something beyond what's tangible. That's where the conversation starts, but quickly it moves into what's technologically feasible. That's where Berto and I spend a lot of time in figuring out what the current state of technology right now can accomplish. And a lot of these techniques are on the bleeding edge. But I'll go back and reinforce that they're becoming as sticky as the half line mark in most sporting events. We can't imagine watching a football or a soccer event without those overlays. So as we move into retail spaces and to higher ed, into all types of other corporate signage or exterior signage, the conversation is very much around what can't we accomplish with normal tools that we could add that you could experience. And that then leads into the same type of techniques that we're using in broadcast. And that's pinned in space elements, right? Like something that is locked to your perspective in a way that as you move your head or you move your eyes, it feels congruous with the real world. And so trying to deliver that experience so that it either feels photorealistic or it has a style that just grabs you. Some things don't need to be photorealistic. They can be hyperrealistic. And so they engage your senses so that they encourage you to purchase or visit or learn. Those are the types of heightened experiences that we want to reach into. How that practically ends up often is in this moment due to a lot of technological innovation, either a single user experience, right? Because we have to create that environment so that you can experience it. But maybe it's not aimed at the person next to you. We're excited about as technology moves and we're able to do more multi-user experiences in specific spaces for retail and for higher ed. But what we are also chasing after is providing the overlay related to your mobile phone or your personal device, right? So there's a lot of headsets that are going to start hitting the market here real soon. We're trying to be ahead of that in all of our experiences that we're motivating. How do we tie in and lean in so that you get the headset experience or the glasses experience as well as the practical physical experience? So that tends to be the conversation. The conversations track in the same type of way of how do you build a website? Well, you start with a framework, you iterate, you move through, you creatively design, you give feedback, and you make changes. So that process is very similar to the way that you work through a keynote deck in a live event presentation, right? We walk through that process and what we end up with is something that could not have been done two, three, five years ago. Something that adds ad revenue or adds excitement about stepping into a space. It adds the ability to learn more about dinosaurs because you can spin them around in front of your eyes or you're able to gamify more engagements. So those are the types of experiences that we're going for. So you're telling me that your kids, when they go to high school, are going to be wearing goggles and the teacher instructor is going to be talking about dinosaurs and they're going to be able to just push the Tyrannosaurus rex and then there'll be a 3D model that's spinning right in front of them. They can do that right now, you know, they can do that with goggles. What they can do in the future is think more about a pair of reading glasses, right? And remove that awkwardness of having something heavy on your forehead. Or to look through a pane, you know, right now almost all my kids with their teachers, you know, they've got technological tools that I didn't have, you know, growing up. They've got whiteboards that they can digitally erase and take the notes with them. And so that next step absolutely is very much about them experiencing that learning effort in a way that's more 3D and more tangible to them. So, Sean, I have one follow-up question and then I want Danny to opine on this next. Is AR going to exist in the Metaverse? I like to say AR is the gateway to the Metaverse, right? As what we, the concept, you know, reading Snow Crash and looking through the Ready Player One, right? Like that reality that, you know, certain companies are pointing at tends to be a little bit more virtual reality, fully immersive personal experience. You know, what we're really excited about is that cooperative, collaborative experience that you get to participate in with the person next to you. And I think that's why it's so relevant for live events. That it's something that doesn't take you out of the experience and you're talking to your friend in Singapore. Instead, it's something that is additive to your current experience. And so, I think that while right now it's the best way to display the Metaverse and something that we are delivering on a daily basis in retail spaces like how do we or in corporate meeting experiences, you know, what's that next step in trying to connect to the metaverse? I think AR is a huge part of that. I think a lot of what we talk about in the metaverse is a little bit more virtual reality and single user. Got it. So, Danny, what is your thoughts on what Sean said about the Metaverse and AR versus, I mean, when you always see it advertised as someone's wearing glasses, they're full goggles, they're in this other world. Do you think AR plays a part or do you feel like it's really to enter the metaverse, you're kind of operating in the virtual reality world? I absolutely agree with Sean. I think, you know, kind of AR is inherently inclusive, is that rather than rely on a closed world, a closed kind of world that is created entirely in the virtual system, that ultimately we're not kind of ready for that as people, right? We deal with the real world and we want to kind of add our own experience on top of that. And I think that that's why AR is so successful here is that it can be more social, it can be more inclusive, you know, and instead of trying to create, you know, the entirety of a new world, it kind of adds something on top of it. And by being additive, you know, kind of helps ease us into that, right, that we're kind of in this really interesting place generationally with AR, right, that the yard line marker on football was a technological breakthrough when it was first implemented. And now it's something that we assume to exist at all times. And I think that, you know, as audiences become more discerning, right, we're already seeing, you know, sports and broadcasting take that augmented reality experience to the next level by adding 3D mascots and interaction with the environment and all of this. And that will be, you know, that same kind of 10 yard line marker that we now know to be ubiquitous. We're going to see that same level of, you know, ubiquity moving forward as those audiences mature and as that becomes kind of de rigueur and standard. And I think the same goes for the Metaverse, where, you know, it's a great gateway. It's a great way to start into this concept of this persistent game world that we're around all the time. And it's a great way to visualize that in our everyday human context and physical context, and that as people get more comfortable with the idea, then the virtual reality kind of context can get up, you know, and become more standardized. But I think, you know, especially we're bucking against the kind of limitations of real-time rendering technology and multi-user experiences and all that, and that AR is a great way to kind of hint at the future of that without necessarily trying to do everything in that instance. So my question to Craig is, Craig, are people doing tours in the Metaverse where you aren't going anywhere, you're just putting on the goggles and you look around and you're having this very immersive, I want to say virtual experience, are they doing that now? Is that the future? Are they monetizing it? Yeah, I mean, there was certainly some successful experiments during COVID. You had large festivals like Tomorrowland. There was one down in Australia that we supported one of our artists playing a live show there all virtually. Yeah, I think it's coming. I mean, we're in the early days. And I think, you know, specifically about AR, one of the most powerful things is it's persistent. It'll be there. And whether you scan across it with the camera tracking, whether you hold up your phone and see it, whether in the future you wear a headset and see it, you know, it'll be persistent, but it won't be static. This is stuff you'll be able to interact with, almost like a real object. You know, with cloud data streaming, you know, it can do updates, depending on the environment. So I think it's a very powerful layer of information, entertainment, knowledge that you'll be able to access, you know, whether in a live event, you know, retail store or any other area. So I think it's going to open up whole new forms of entertainment. And I think it's going to open up whole new forms of education and knowledge base. I know all of us have cell phones. That's the one thing I'm 100% sure of. So if they can get something to work with your cell phone, and you point at it, someone said, I take a picture of the coffee shop, and then all of a sudden, it's going to know what brand it is. And then it's going to automatically put up the promotional things that are happening as I'm driving by, whether I'm wearing glasses or a cell phone. I think the sky is the limit for this technology. And incorporating it is that we're kind of I still think in the infancy of incorporating it to all of our daily life. And kind of the example, Danny, you used of the football line, it's been out for years, but now we just kind of expected, who knows what AR will be in five or 10 years. So this is my last question. I'm gonna start with Berto. If you were trying to pitch someone about why they should be doing some work in AR on their meeting event, on their broadcast, on their exhibition, whatever the tools, what's your best pitch for this technology? Why should I be thinking and using AR? Why is that going to help me in the long run? Oh, this is hard. I personally think because, you know, augmented reality is a spatializing technology, you're able to use the space that you're in, you're able to use your surroundings in order to add a layer of information, as Craig was saying, whether it's a virtual wallpaper on your wall, or you know, some advertisement that comes out of your screen, or that you're as you're driving, you see advertisement in the air. I think it's just another way of using the space with this new medium in order to, you know, create a new experience. That's really it. It sounds like it's going to be in my car windshield, not too long from now. Yeah, it's, it's going to be everywhere, wherever there's glass and a display and wherever you can put a projector, whether it's a tiny projector that fits on your glasses or a large projector, you know, I think superimposing images on, you know, a visual that we see it, that's where it's going to be, you know, it's going to be everywhere. And one day, it's just going to be, you know, in, it's literally going to be in our contacts, and you're not even going to worry about it, like, you're going to walk out of your house, like you're, you know, you forgot your keys, without this technology, if you don't have it. That is awesome. I'm looking forward to the future. Now, Craig, give me your best shot at why I should add AR to my next concert tour. Oh, absolutely. You know, I think, as Danny mentioned... Sell more merchandise? Yeah, absolutely. You know, I, again, I think, you know, in the touring world, we create short term, fantastical places, immersive environments, and AR is, is essentially built for that, you know, there's no better tool in the toolbox to do that. The only challenge is for the users to display. Right now, really, the only realistic option they have is to hold up their phones and watch through their phones. But I would say within the next two to three years, there'll be some opportunities to change that with headsets of varying sizes. And, you know, we'll see that technology shrink down to glass size, probably over the next decade. But, you know, the technology is there, it's in the lab, it's starting to get out into the wild, it's being used commercially. And it's, I believe it's just going to accelerate as we go. And it will lead to just about unlimited imagination for the creative team. Unlimited imagination for a creative team. That sounds like a hashtag, Craig. That is just, that's so much fun, that one. Okay, Danny, I know we did this monster corporate event. I hope we do more monster corporate events using this cool technology. Give me your best pitch for why AR should be in the next corporate event for next week, who we're working with. Yeah, I think, you know, partially because it's cool, but also, you know, really because it, you know, it allows another layer of communication with your audience. It gives you more visual real estate, more narrative real estate. You know, it's embracing the very, you know, edge of what's possible. And I think that as a corporate company, as a company that is trying to signal that you're embracing the latest, greatest technology, I think it's absolutely a factor, right? In the same way that, you know, in the 90s, right, you weren't a company unless you had a website. I think that we're going to be approaching that same thing here with if you don't have an augmented reality layer to your company, that way to communicate your brand in that augmented reality space, that you're not going to have the same layer of, the same way to communicate with your users and audience and communicate that brand as other brands will. You know, it's interesting. A lot of people have spaces in the metaverse where they have their store or something, but it seems like the gap between the metaverse and our real live reality is this augmented reality. So it's the glue between the two. Sean, bring us home and tell us why for the next big installation, a venue, a retail store or something should have an augmented reality component. Absolutely. Les, as usual, you've got some great guests and there've been some great answers. You know, I lean into that, you know, being able for a client to unleash their creativity, being able to really deliver on something that is new and sticky and being able to grab eyeballs, right? When we're talking about events and broadcast and telling stories, when we're talking about retail and corporate and installation and signage, what we're talking about is personalization. We're able to deliver data directly to you. There can be different advertisements presented to your eyeballs than the person next to you. You can be looking at the same thing. So, you know, that's where there's a lot of ability to grow. There's a limitless ceiling when it comes to how we can, as you said, driving down the street and your mobile read something and you get personally sold to, you know, so advertisement dollars, promotional materials, opportunities, you know, as we get into spaces where we want to experience, it comes out of that personalization and it comes to being able to experience things like the metaverse and art installations and entertainment together, right? So we move from personalization to a group opportunity to experience something more, something beyond the tangible. So that's my pitch for you. You know what? I like all of your answers. I think you all are rock stars and super smart in what you're talking about. I got to tell you, I am so excited that we made it through a whole podcast and no one said the word COVID. God bless you all. Because I think that we got to go back to doing what we do and the world is going back to whatever our new normal is. And I'm super excited about AR, especially because I sat in the audience in that show two weeks ago and I was like, I knew exactly when we were using it and I knew when we weren't using it. So it was super cool. But I want to thank all of you. Y'all are amazing. Berto, Danny, Sean and Craig, thank you for your insight. And we look forward to making sure to enter the AR world and make sure we embrace it and we don't talk people out of it, but we talk them into it. And this is Les Goldberg and The Road Ahead. Thank you, Les. Thank you.