Instructor: Jon Zmikly

July 11-15

Zoom July 15, 1-5pm CT

Description:
In this first part of the module, we will discuss basic terms, uses, considerations and strategies for teaching a course that integrates augmented, virtual and mixed reality. We will also discuss some of the new opportunities and challenges with teaching immersive media, and you will learn various software programs for creating immersive projects. Finally, we will discuss artificial intelligence and machine learning and some tools used for storytelling today.


Outline

I. Hello in VR! What is AR/MR/XR?

II. Four Project Options (360º Photo Story, 360º Video, Augmented Reality Experience, 3D Website)

III. Course Resources

IV. Artificial Intelligence (AI) and Machine Learning for Storytellers

V. Discussion / Slack

Online Exercises & Materials

Pt. I: Hello in VR!

Before we get started with this week’s content, I wanted to say a quick “Hello in VR” from the Media Innovation Lab at Texas State University! Watch the 360º video below to take a look at some of the equipment and hear about some of the pedagogy behind Jon Zmikly’s VR and 360º Video course!

Virtual Reality

As shown in the video above, virtual reality (VR) refers to any computer-aided technology that allows users to feel like they are in a different environment than their actual reality. The onset of digital and mobile technology has brought with it new apps and devices that can deliver much more believable and “real” immersive experiences than ever before. VR can be created using computer-generated programs such as Unity or 360º cameras. Thanks to new technologies like the Oculus Quest 2 and Valve Index, there are a number of realistic experiences for users to enjoy, such as interactive VR films and documentaries. Other examples range from VR gaming to powerful character-driven narratives, each of which brings its own unique set of opportunities and challenges.

Experience VR:

  • From your computer, or a smartphone (YouTube app) watch one of the following videos. If you are using a browser, you can use your mouse to scrub around the environment. If you are using the app, you can move your smartphone around to explore.
    • After Solitary — Go inside the Maine State Prison to hear the harrowing story of Kenny’s time in solitary – how he coped, “fished” contraband to other inmates and fought guards during cell extractions – and what happened when he got out.
    • The Displaced – War has driven 30 million children from their homes. These are the stories of three of them. A New York Times immersive story on the Syrian refugee crisis.

Augmented Reality (AR)

While virtual reality experiences are meant to bring the user into another world (walking around on Mars or “attending” a concert halfway around the world), augmented reality changes (or augments) the world in which a user is already experiencing. One of the most common examples of AR is the face filters seen on Snapchat and Instagram. These experiences are widely popular due to their low barrier of entry (cross-platform, social media and smartphone ubiquity), and they allow people to interact with their environment in a way they never had before. Other examples include iOS’s Measure app or games like Pokemon Go, the smartphone app that allows users to find and “catch” Pokemon characters that exist in their physical space.

Thanks to software like Apple’s Reality Composer, Snapchat Lens Studio and Meta’s Spark AR Studio, augmented reality is becoming much more accessible and shareable platform for content creation and storytelling.

Experience AR:

  • Google Translate for iOS or Android – Download the app, and then use the interface to view text of any language. The app will translate it for you on-the-fly! It’s magic. And amazing.
  • If you subscribe to the New York Times, find the following articles by searching within the app. (Linked below are web versions of The New York Times Augmented Reality experiences. In a browser, they are not immersive, but they are within the app)
  • Within the Amazon app, type in “virtual try on” in the search bar. You will see an option to “Virtual Try-On Shoes”. Tap the shoe of your choice, and then tape the “Virtual Try-On” button to test the shoes out on your feet!

Mixed Reality (MR)

Many experts believe the future of immersive content lies in “mixed reality”, a combination of augmented and virtual reality. In MR, a user can interact with virtual objects within their actual space. For example, the Magic Leap is a “lightweight, wearable computer that brings the physical and digital worlds together as one.” Microsoft HoloLens is another exciting platform for immersive experiences, and this industry is seeing great potential for commercial applications. Watch the video below for a demonstration on the HoloLens 2.

Pt. II: Four VR/AR Project Options

Option 1: 360 Photo / Story Spheres Exercise (Beginner)

  • Technology Requirements: Smartphone, Google Street View app, Computer with web browser
  • Resources: PDF Instructions

Option 2: 360º Video (Intermediate – some video editing experience preferred)

Option 3a: Plane-Based AR Application (Beginner)

This was created in conjunction with a tutorial on X Code, but this method is much simpler and easier.

Option 3b: Marker-Based AR Application (Intermediate / Advanced)

This was created for my class, and you will see references to our Learning Management System. The downloadable instructions referenced are above this video.

Option 4: 3D Website (For the Code-Lovers!)

WebGL stands for Web Graphics Library and is a JavaScript API for rendering interactive 3D and 2D graphics within any compatible web browser (Firefox 4+, Google Chrome 9+, Opera 12+, Safari 5.1+, Internet Explorer 11+, and Microsoft Edge build 10240+). WebGL works with HTML5 <canvas> elements, and the Canvas API to do 2D graphics on web pages. Learn more about Web GL at Mozilla.org.

Babylon.js is a real-time 3D engine using a JavaScript library for displaying 3D graphics in a web browser via HTML5. It works with Web GL (in the browser) and considers itself to be the “most powerful, beautiful, and simple Web rendering engines in the world!” Take a look at a few demos of what Babylon JS can do:

Examples:

To learn more about Babylon JS, go to babylonjs.com

Instructions: 
Use Web GL (Web Graphics Library) and Babylon.js to create a 3D website. This will utilize a template gallery to build whatever world you wish (including, but not limited to the gallery).

  • Download Babylon JS Template
  • Open / extract the .zip file, and open the index.html file in a text editor of your choice (Sublime Text is recommended). 
  • In Safari, go to Preferences, and choose the Advanced tab. At the bottom, check “Show Develop Menu in Menu Bar”. Then, go to the Develop menu in Safari, and and check “Disable Cross-Origin Restrictions”. 
  • Open the index file in Safari. You should see the 3D site, but it will be very basic.
  • Now, go through the code and delete all the code that says //–> That’s the comment code that hides the content. As you delete the comment code, you should see the template emerge. Follow the tutorial to uncover the comment code, revealing the virtual gallery. Customize the files in some way (put your own picture on the wall, move the wall, move the camera, etc. Do something to make it your own! Be creative.

Pt. III: VR Course Resources

Equipment and 360º Video Options

Stitching Apps:

Editing in 360

VR / AR Resources

Professional Examples / Discussions:

Jon Zmikly’s Course Resources

Pt. IV: Artificial Intelligence (AI) and Machine Learning for Storytellers

We will discuss the use of AI and machine learning in digital media on our Friday Zoom call. It would be best to familiarize yourself a bit with these articles (skimming them is fine). We will do a live chatbot demonstration together during the Zoom call.

Resources:

Pt. V: Discussion

Above, you viewed four different VR/AR assignments to choose from (ranging from beginning, intermediate and advanced, and depending on your access to technology). Just complete one of the four, and post the link or screen recording of you using your project to our #immersivestorytelling Slack channel. Once you have posted your project, please answer the following questions:

  • Between AR, VR, and MR, which do you think is more exciting, and why?
  • After experiencing the VR examples provided in this module, how would you describe the experience? Did you truly feel immersed, forgetting about your actual reality, or were you still quite aware of your actual surroundings? What factors contributed to those feelings?
  • What are some potential challenges or ethical issues that might arise from these new forms of storytelling? 

Zoom Session

On Friday, July 15 we will meet via Zoom to chat about (and show off!) your projects, discuss the pedagogy of an immersive course, as well as answer any questions you may have.

Module Evaluation

After each module, you will be asked to do an evaluation. Go to the link for the PhDigital 2022 Zoom Modules Evaluation and select the module you are reviewing. Please do this as soon as you can after the Zoom session.

Final Curriculum Assignment

What is your dream course, program or degree? Create an outline for a proposed course, program or degree, including mission/description, objectives, topics outline and provide a few relevant resources. Just a page or two is fine. This can be for an undergraduate or graduate program (or even a new doctoral program!), a minor or concentration, whatever you think is necessary in your current environment, future employment home or the discipline at large. Post in the #finalcurriculum Slack channel. Turn this in by July 22.

Final Evaluation – Please take a moment to do the PhDigital Online Final Evaluation. This is an evaluation of the program as a whole. We really appreciate the time you took to evaluate each module and the program.