Beyond the Screen: The Future of Virtual & Augmented Reality in Design

The rise of developments in virtual and augmented reality presents vast new possibilities for designers and creatives across a broad array of businesses and disciplines.

I’m old enough to remember a time when the closest thing to virtual reality was flipping through 3D images on circular reels, and augmented reality meant covering your bedroom window with vinyl superheroes.

As a child during the 1970s, if I wanted to enter an imaginative science fiction universe, I pulled out my View-Master. I inserted one of my many reels, put the toy up to my eyes and saw layered imagery: Foreground and background came to life right before my eyes. After the View-Master reels, I’d pull out my Shogun Warriors and Amazing Spider-Man Colorforms. The sets each included their own vinyl character designs that would cling—and uncling and recling—to smooth surfaces like the included cityscape backdrop or my bedroom window.

Today’s virtual reality (VR) and augmented reality (AR) is a far cry from the analog toys I tinkered with decades ago, but in many ways, and for me at least, they were a precursor. Using Oculus Rift, you can put on a headset to see what others have seen, be where others have been. You can build robots, monsters and worlds in 3D. The creative potential is as extensive as the breadth of your imagination.

[Related: Close Encounters of the 3D Kind: The Rise & Future of 3D Printed Design | 3D Printing: BOLTgroup Product Innovation and Brand Experience]

When it comes to AR, most of us think of Pokémon GO, where characters appear layered on top of the real environment seen through your smartphone and displayed on screen. VR and AR experiences are intimate, putting us into fully immersive worlds where we can look, learn, play and create. If the prior eight years were all about smartphones and mobile optimization, the next eight years in the design world—and any creative discipline, for that matter—may revolve around virtual and augmented reality.


It can’t compare to Colorforms, but today we plaster small and large, static and animated graphics everywhere we can using our smartphones. Facebook, Twitter, Snapchat, Instagram, iMessage—you name it: If it’s on our phones, we’re interacting with it. And if it’s not, we’ll download an app to do so ASAP. It’s not only social media and messaging, but also creative apps, games, news and educational tools. It seems like everybody has a smartphone, young or old. The ubiquity of the smartphone may be why Apple’s Tim Cook sees AR as the next big realm of development for design and technology. Compared to VR, Cook called AR “the larger of the two” in an interview with ABC News in September 2016. Cook suggested that AR allows us to be more “present” because users are more connected to the real world in AR.

VR does have its disadvantages. In some cases, VR has steep hardware requirements with headsets or gloves required for a truly immersive experience. But VR gives you the feeling of being right there and in the moment. You can go anywhere you want in VR, to faraway National Parks, distant lands or even outer space. You move and feel as if you’re moving within that space. If you’re wearing gloves with VR integration, you can see your hands and any tools the software enables you to hold, poke, prod, sculpt, scrape and engage with. Sounds, and even vibrations in the form of haptic feedback, are all part of the experience. For example, Oculus Touch uses rumble for feedback in the controllers, and spatial audio gives the user a truly enhanced, “in the real” experience that adds to one’s awareness of what’s happening in the VR moment.


The Oculus Touch allows you to work with your hands in a virtual environment.

But for the most part, VR is synthetic compared to AR, and VR is somewhat closed. AR lets us live in the moment with the world surrounding us, and as we look through a smartphone screen we see the real world and the people in it. Perhaps this is one of the reasons Cook sees it as a “larger” technology. Then again, maybe it’s because Apple already has millions of iPhone devices in users’ hands, and all those devices need is an AR-enabled app to let you experience that next-level interaction.

In terms of socialization, AR is more socially engaged since you can see those around you as you experience what you see on screen. You couldn’t play Pokémon GO outdoors, walking through the streets of New York City with a VR headset completely covering your eyes. VR requires a headset that masks your entire field of view, closing you off from the real world by putting you squarely within the digital world.

In this way, VR has a lot in common with the View- Master, which is on the verge of a comeback. In 2015 View-Master partnered with Google to bring VR to the masses through Google’s Cardboard VR platform. Mattel’s View-Master VR Starter Pack retails for $29.99 and is intended for educational uses where an inserted smartphone lets you run “apps, games, videos and View-Master Experience Packs to create stunning virtual reality worlds for kids to explore.” Google’s Cardboard, VR on YouTube, and Cardboard Camera are all part of its equation.


Games, advertising, field work, research, medical applications, education. If you can imagine it, there’s a way you can use VR and AR. But children are the future, which is why educators are leaping at the opportunity to introduce AR to their students. Today’s digital natives grew up with smartphones, tablets and iPods. They’re neither intimidated nor reluctant to use technology. In fact, if it’s on a screen, they want to use it and they will likely adapt to it much quicker than an older user.

At the Valhalla Union Free School District, Cayne Letizia, reading teacher for grades 6 through 8, used VR in the classroom to explore “the role VR has in storytelling and empathy.” Letizia used the New York Times VR experience to connect students to content in a fresh new way. His upper elementary and middle school students used NYT VR with the Google Cardboard VR goggles to give them access to content that they would not ordinarily be able to engage with. How do you see a National Park that’s thousands of miles away? Use VR.



Students at the Valhalla Union Free School District used NYT VR to go beyond the classroom in an immersive learning experience with cardboard VR headsets.

“As we celebrated 100 years of National Parks it was impossible to visit many of the parks, but Google 360 allowed us to travel all over the U.S. and experience them firsthand,” Letizia says.

Letizia stresses how important it is for students to connect—really connect—to content during a day and age when it’s all digitized ephemera, for better and worse (but mostly for worse). “So much of the technology my students use daily removes any sort of empathy—Snapchats that exist temporarily, the ease with how a student’s life can be changed for the worse with a tweet or Instagram comment. VR can place the students in the moment and break down that anonymity and teach them that empathy is an important trait to have as a human.”

The course material available for exploration is as wide and deep as the internet itself, and Letizia has plenty of his own ideas for taking advantage of the technology for classroom use. “Art teachers could visit European museums, history classes can take 360-degree tours of famous battle sights or other historically significant landmarks, P.E. teachers could have students experience a VR fi lm about an Ironman! The applications to enhance what we do are endless.”

Educational applications are one of the many ways that AR and VR will get into the hands of a new audience, especially as more schools adopt the technologies.

The other inroad? Games. Lots of games. If you thought Pokémon GO excitement was contagious, that was just the beginning. Wait until Minecraft comes to AR and VR—specifically HoloLens, Microsoft’s holographic computer—because when it does, children all over the world are going to want one. HoloLens has only been released as its Development Edition so far, and my kids have been asking for one since Microsoft teased the platform in 2015.

hololens2 hololens

But is HoloLens AR or VR? Maybe it’s neither. A Microsoft spokesperson replied to an email where I
asked about how to classify it: “Microsoft HoloLens is the world’s first fully untethered holographic computer, running Windows 10. And it’s a great example of mixed reality—the ability to blend holograms into your real world. With AR, the user sees a layer or screen of data that overlays the real world. While this data can be contextual to the user’s location, or where the device’s camera is pointed, it is not the same as being able to see holographic objects pinned, or anchored, to specifi c physical locations or objects in the real world.”

HoloLens will enter the mainstream soon, making for another platform in an already busy ecosystem, and it promises to change everything. The Microsoft representative touted how it will improve on not only “business processes” but also a wide range of “mixed-reality applications.” Want to develop for it? Sorry Mac users. You’ll need Windows 10 and HoloLens. Then again, if you dual boot your Mac, you should be good to go.


Some designers are going to create apps, games and entertainment for these platforms. Others want to create with these platforms. If you’re eager to create using VR, rather than developing for it, then you want to get your hands on the right tools, and Oculus is the current front-runner in that respect.

In early September, I had the pleasure of watching a Facebook livestream with artist Goro Fujita discussing his VR creative process—while he sculpted in virtual reality using Oculus Medium.


The livestream intercut what Fujita saw through his Rift headset with images of him in his gear, moving around an office, and moving his hands to create a 3D robot. Considering that he had only been using the tools for two months, he handled it all masterfully. Fujita built up one layer after another, and livestream viewers watched him pick, choose and use tools for sculpting. Joints, armature, color, lighting, textures, cables—you name it, he built it into his robot, or carved elements away to make space or insert objects. Fujita often referred to Oculus’ virtual reality creation platform Medium as a “sculpting tool,” which is appropriate given what I saw him do.

In 50 minutes, the sculpt was “pretty much done,” according to Fujita. He added realistic effects such as scratches, weathering, oil stains, wear and tear, and added a nice final touch: hotspots in the eyes. With Oculus Medium you can “Sculpt, model, paint and play in virtual reality,” according their website. If the technology and software catch on, it’s going to give Photoshop a run for its money. After all, who wouldn’t want to get up from their chair and walk around their designs day in and day out?

In the not-too-distant future, perhaps Adobe software will come to such a platform. Even Steven Spielberg’s Minority Report showed us one possibility for moving and managing data, images, information and windows. And even that was based on something in the works as far back as the 1990s at the MIT Media Lab, courtesy of John Underkoffler’s scientific and interactive research that became the foundation for Oblong Industries Inc. It’s neither AR nor VR, but “g-speak,” the technology behind Oblong’s Mezzanine, is a spatial operating environment.



If there’s one place where AR and VR will make a breakthrough in the near future, it’s with advertising, promotion and marketing for travel and hospitality. Looking back to the early 2000s, the internet changed the way that everybody shopped for a home or apartment. You could do it all in the comfort of your own home, provided you had a high-speed internet connection, a computer and a web browser. Today, most of us have that, and the entire experience can take place on our smartphone.

In late 2016, Oasis, the innovative hospitality company that pioneered the Home Meets Hotel concept, debuted a series of “immersive 360 video city guides” intended to give travelers a closer look at destinations. Oasis’ founder and CEO, Parker Stanberry, replied to an email questionnaire about the project’s creative process and why a 360-degree virtual reality experience was the way to go. “The 360 video platform allows us to tell the brand’s story in a creative and immersive
way,” Stanberry said. “The videos present an exciting opportunity for consumers as they have the opportunity to be in the driver’s seat and explore these fascinating cities without having to hop on a plane.”

Oasis worked with SPECTACLE VR and director Peter Martin to produce the 360-degree tour. “We chose to work with Peter and his team as his extensive portfolio of videos and expertise in storytelling aligned with the ultimate goal of our campaign,” Stanberry said. You can watch the tour on YouTube, provided you have the proper browser. Or, you can watch it in a VR headset or Google Cardboard VR. Visually it’s a stunning experience, and the only things missing are the sensual, delicate aromas that the real places can offer you.


Plenty of tools are available for developing in AR and VR spaces, and open-source platforms may be the way to go. In many cases, they’re free, easy to learn and updated routinely to offer the latest advancements, encouraging users to go deeper, explore the tools and solve problems. One such platform for AR is ARToolKit, developed by professors Mark Billinghurst and Hiro Kato with Ph.D. students at the University of Washington’s Human Interface Technology Laboratory.


As chief open source evangelist at DAQRI, a tech company that focuses on developments in augmented reality, Ben Vaughan is “responsible for encouraging the adoption of ARToolKit” and building their opensource community. Since ARToolKit joined DAQRI, Vaughan says there have been “more than 100,000 downloads of the software.”


Vaughan cites the open-source platform as one of the reasons ARToolKit has advanced the way it has. “By releasing the software open source, the founders of the company wanted to encourage innovation in the space.” When it comes to giving credit where it’s due, Vaughan suggests that “Mark Billinghurst and Hiro Kato should be remembered as two of the earliest pioneers—and also Jun Rekimoto, who developed virtually the same technology at the same time while working at Sony Computer Science Labs.”


Open-source tools are just one of the many ways you can begin creating for yourself. You could also grab a book, such as Learning Virtual Reality: Developing Immersive Experiences and Applications for Desktop, Web and Mobile by Tony Parisi. It covers “the three go-to platforms—OculusVR, Gear VR and Cardboard VR—as well as several VR development environments, programming tools and techniques.”

Programmers and creatives who have a knack for picking things up, running with them and innovating will have relative ease when it comes to working within the platforms featured in Learning Virtual Reality.

On the other hand, if you’re somebody who’s driven more by image and less by code, then consider something as simple as Google’s Jump camera rig, an array of 16 cameras assembled in a circle used for capturing 3D space. GoPro VR also lets you “Capture, Stitch, Share” with relative ease, as promised on its website.


At the moment, today’s youth appear to be the ones consuming the most AR and VR, and everything in between that can be found on personal electronic devices, gaming platforms and—if/when they watch it—television.

But will today’s youth be the ones taking AR and VR tools and making something new? Vaughan sees the promise in sharing development platforms such as ARToolKit with anyone and everyone, including students, but he sees something beyond what students can do. “While I fully support the idea of students becoming more familiar with programming and software development tools, I believe that it is creatives and visionaries that are needed to identify the ways in which the technology can be used.”

According to Vaughan, they will be the ones who will identify problems that can be solved using AR technology.

Then again, why can’t students—or children, for that matter—be creative and visionary, as well as problem-solvers? AR and VR tools should enable students of all ages, especially children, to not only experience the world they live in, but also to play around and create new experiences. After all, play is serious business. And if there’s one thing children know how to do, it’s play.

Earn a Certificate in Character Animation | HOW Design University

mkuh7gr7fqllw0grpayg (1)

Design News, Featured, HOW Interactive: Web Design & Interactive Web Design Tips

About Jason Tselentis

Jason Tselentis is an educator, writer, and designer. As Associate Professor at Winthrop University, he teaches visual communication design, brand strategy and development, web design, and typography, among other design courses. His writings about design and visual culture have appeared in Arcade, Eye, Mental Floss, Open Manifesto, HOW, and Print. He was a contributing editor to Print magazine and a regular contributor to HOW magazine. Mr. Tselentis also has four books to his credit on design and typography principles, and design history.