Magician Albert Cadabra

DEEP LEARNING creator Will Brierly on his AI bot, Merrill Grambell

The REFUGE Interview

Bracey Smith talks about his VR creation ARMONIA

This interview has been edited and condensed.

How did Armonia get conceptualized?

In late spring of 2018, Google Daydream and the Tribeca Film Institute started this Immersive Lab program where they wanted five teams to work on a narrative VR project and they were going to give a $40,000 grant for each of the teams to produce something. I had wanted to do this interactive idea, it was a narrative story, and initially the idea I had come up with was interactive. I pulled together a team of people I knew might be interested in doing this, we applied, and we got accepted.

Who was on the team?

It was Neil Dvorak, Sara K. White, and Josh Bernhard. Initially, we thought we were going to make an interactive piece but once we got chosen we found out very quickly that it was actually more geared toward 360 film. Google had given us access to this very expensive 360 camera. We really wanted to use it but, ultimately, when they told us that it wasn’t necessary to fit what we needed to deliver, we ended up not using it. I made the decision to shift more toward something that we could build on. I felt like if we just made a movie, and filmed something with a camera, then we wouldn’t be able to build on that. So instead, a good portion of the budget went to build this computer so we could actually make this an interactive experience. It basically renders the whole experience in 3D. I wanted to actually spend the money in a way that after we delivered this thing, we can make more, and build on the brand.

What kind of computer is it?

I actually built it. It was about $5,000 in components. I wanted to save some money and just build it myself, which I wouldn’t recommend making your first computer that you ever build so expensive [laughs]. The reason is I knew it was going to need to be 360 video with a 360 narrative. Ultimately, I wanted to do something that was going to be more like an animation, and to be able to do that I needed some serious computing power. Instead of it just being a traditional animated narrative, I wanted to lean a little more on music. So that brought in the whole piano concerto element to it.

So you built the computer to basically address your specific needs?

Yeah. The idea was approaching it not necessarily as a traditional narrative but like it was a performance. If you go to hear an orchestra play, you get this fantastic sound experience but almost always the visual element to that is lacking. I thought it would be really interesting to free up the audio localization of an orchestra, where you’re used to them being pinned to the stage in front of you, and be able to utilize this 360 audio and visual environment.

So it’s all around you?

Yeah, so [the images] can fly and move all around you and, ultimately, you can use the music to tell a narrative the way you would expect a piano concerto to be able to tell a narrative. [My idea was] to apply [the music] to visual storytelling with virtual reality and free up the audio from being just a stereo experience to give it a spatial audio experience. I like the idea of being able to be in the center of a performance and you can tell without seeing the character, just by the music alone, what the mood is. All these things that you can pick up on just musically, you don’t have to visually see. I wanted to explore how you can play with how music tells a story inside of a 360 space.

So once you assembled the team, got the grant, and you built the computer, who did what?

When things shifted from being a film project to being more of an animation piece, their skill sets didn’t really fit the structure of the project that much anymore. So once that happened, it basically became me. Everything but designing the 3D models fell on me. While they were really key in helping shape the project overall, they kind of fell into more producer roles. At the same time, Anja became a major producer and helped organize things on the ground. She came in and became a key part of the project organizing and making sure we were hitting the deadlines. That basically became the team. We got a guy named Jacques Lalo of Nova Media, and he designed the characters as well as the environment.

So the music and the animation was all done by you?

I wrote the piano concerto and recorded all the pieces except for the flute. I had done everything on a Yamaha keyboard in MIDI and recorded everything on my iPhone, and then had my friend (Daniel Coletta) take the MIDI files and actually mix them and then apply all those MIDI files to the proper instruments. But one of the things that is especially hard to get right in MIDI is the flute and wind instruments. The wind instrument is a very key component of the whole piece, so for that part we brought in a flute player (Michael Carleo).

How long did the whole thing take to put together?

Conceptually, I started rearranging what we were going to build in August [2018]. It fully solidified by early October. In that time, I had started writing the piano concerto. I wrote the music portion and then had to go back to the visuals and write the visual narrative, so it was kind of this back and forth. When that whole writing process was done, then we started the animation production in November, and it was due in December. So we basically did everything else in the span of two months.

Wow, that’s a pretty tight timeline!

Yeah. The experience is not as refined as I want it to be but also I wasn’t expecting it to be a completely polished piece by the end of a four-month period. I think once you understand the context in which it was made you realize that with just a little more money and a little more time you could actually make some seriously amazing stuff. With this new type of production, which is what I’m calling XR (Extended Reality) Production, it’s kind of a blanket term for virtual reality, augmented reality, and mixed reality. Basically, it’s using cheaper virtual reality tech and leveraging that to make a video production or 360 animation.

When you got this grant from Tribeca and Google, was the product always going to appear at the Tribeca Film Festival?

No, we had to submit it. We got into the Film Institute Lab but not only did we have to finish the product but then we had to submit. I know two of the five teams aren’t in Tribeca but I don’t know if it’s because they didn’t finish, or they didn’t submit, or they didn’t make it in. There were five teams altogether who won the grant and only three are in the festival. I don’t know how many applied for the grant in total, the number 1,000 is sticking out to me but I’m not sure. The thing I really liked is it was very diverse. All the teams were led by either a woman or a minority. They were all chosen on the theme of the elements. Everybody had a different approach to the elements and our idea was to do an experience that incorporated all the elements.

How is the piece going to be presented? How are audiences going to experience it?

At Tribeca, I believe they are going to be using this headset or one just like it; it’s called Gear VR and it will have the experience loaded on a Samsung phone. They’ll insert that phone into the headset. It gives you what’s called three degrees of freedom, which means that you’ll be able to turn left to right, look up and down and turn your head, and roll your head around as well.

They’re curating all the experiences together, so you’ll be able to get a ticket for this showing and be able to see all three pieces back to back. Afterward, we’ll do a Q&A where you can meet the creators and talk about it. I believe that’s the format. What I like about it is that it’s curated. There’s going to be the experience, and it’s going to be a room of people who are going to watch it all at the same time, they can form their opinions about it but there’s also the space to talk about it. I may or may not like that after we get to talk about it [laughs].

What’s cool is what we did to make this animation. Instead of using traditional animation software, we ended up using the Unity game engine, which is what people use to make video games. This game engine has now gotten to the level where it can do some things in real time as opposed to being pre-rendered like tape. That’s why I’m really excited about this and part of the reason why I’m excited about building a computer that can handle this. If I just took it a little further, you could even render to the quality of Pixar or something like that, which was unheard of even a year ago.

It’s a spatial experience. The full experience is created to make you feel like you’re actually there, so a noise that is happening to your left is going to always be on your left and if you turn to it, you’ll see it in front of you. Your audio experience becomes where you’re looking and it adds to your immersion of what you’re experiencing. You attach sound to an object and then that object produces sound, and so if it’s visually over there, it’s also audibly over there.

It’s almost as if the music is animating the images.

Yeah, it’s kind of playing with that back and forth, what’s animating what. Is the music motivating this or is the visual motivating the music? The idea is it’s opening up a new frontier of play, a new area and a new way to play with telling a story that just kind of takes the shackles off your imagination. It lets you just do things you couldn’t do before. For some people that’s overwhelming. For me it’s just crazy exciting.

This isn’t exactly theater per se but it is experiential–you’re moving your head, you’re seeing different things, and you’re extracting what you want to extract from it.

Yeah, and a cool element about this is the way we did it, it can be a live experience. And that’s why I’m really excited about it. All those characters, all the movement, all the animation, that was me with motion trackers on. But because this is in a game engine, when I say real time, we can generate all that stuff in real time. We could take a live musical performance–say someone was recording the guitar in one room, and then we had someone inside the motion capture equipment, inside the game engine I could run real time to that VR headset and the person could be attached to the music so wherever he went that guitar would be playing through him. And not only can we make it live but we could broadcast it so people anywhere in the world are experiencing this immersive live experience where music and physical performance become one and not this separate thing. We’re interested in pushing the bounds of what an experience is, what live performance is, what storytelling is, and what people think is possible. It’s also about changing expectations, expecting more out of what you’re getting.

Do you want to do anything more with this?

I’m working on something I’m calling “Armonia Festa.” It’s a feature film that’s composed of 10 vignettes. Each one is a different group of artists from around the world that composes a 7-minute experience that’s just a celebration of performance, visual art, and music. It’s trying to tell a narrative with those elements, taking out language, and just being able to tell a story with those elements. They’re all tied together with this “Armonia” brand but each one would be its own unique story, own unique visual, and own unique characters. But it would be utilizing this brand of “Armonia” as a lens to jump into this new dimension. And the stories are about issues in the world, like climate change, that try to tell a narrative without language.

Is the idea to combine artists from different visual media?

Yeah, exactly. Interpretive artists, and combining artists like this with a production team that understands how the technology works and is also interested in writing and telling a story. So taking these two groups and combining them to make a project like I just made, and from that project we tie together an entire feature film. But then we also have an immersive VR experience, creating immersive content in a way that was never possible. I’m really excited about creating content in a way that is also universal, the idea of being able to create a movie that anyone in the world can enjoy. Then, if they were already a fan of any of the visual artists, performance artists, or musicians, they would find this and be exposed to so many more artists they didn’t even know existed. So my excitement comes from the idea of being a vessel to get these artists together from around the world. Then it becomes kind of like the World Cup of culture, this moment where you can really relish in what humans are really capable of.