Week 2

Unity3D and Content Generation Tools


Week 1 we showed off a bunch of different sample VR / AR worlds, so this week we are going to talk about how to build them, in particular with respect to Project 1.



There are various libraries that have been used to create virtual reality worlds, and most of them come and go after a few years as the technology landscape changes.

Personally, I have used a variety of languages and libraries to create VR applications:


With the newer consumer level devices a couple of the free big game development engines now support VR development.

The three projects in the class we will be using Unity3D so people can share experiences and issues.

Unity3D runs on windows and OS-X so you can do much of your development on a laptop and then move over to a machine connected to a VR / AR display for testing. Under normal circumstances I would highly suggest that you test on the actual hardware regularly so you don't go too far down a development path that wont work for the final deployment. In the lab, there is one VIVE in the main lab, and another in the classroom, both connected to Windows PCs. The one in the classroom has a webcam attached for AR work. This term with social distancing in force we will try and minimize the need for testing in the classroom, but I hope people will be able to come in before the end of the term to experience VR for real.

To get started go to https://unity.com/ and Get Started, Click on the Individual Tab, Under Personal get Started. Either the First-time users or Returning users start by downloading UnityHubSetup and launching it.

Unity Hub gives you a central place for your various projects, and various installs on Unity. Once you have the Hub running you should click on Installs and then Add to add a new Unity version. The version we want is not one of the ones in the short menu so click to visit the download archive, click on the Unity 2019.x tab, and pick unity 2019.4.1f1 to install with Unity Hub. Note that there is a new release of Unity roughly every week, and things can quickly get unmanageable if everyone is using a different release, so that is why we are standardizing on one version.

You should have it install:

Normally everyone would be presenting their work on the windows 10 PC in the classroom. If you are on OS-X and forget to install the Windows Build Support, you can come back and install it later from the Unity Hub. If you want (or feel the need) to install other components that should be fine. Be prepared to lose 4-5 gigabytes of disc space and at least 10 minutes for the install. You can have multiple versions of Unity installed at the same time accessible through the Unity Hub in case you need to have different versions running for different classes or projects.

With Unity you will be using a combination of an IDE and writing C# code, so its a good idea to go through the introductory tutorials if you haven't used Unity before (e.g. in the CS Video Game course).

The tutorials at https://unity3d.com/learn are a good place to start learning the Unity3D IDE, and there are quite a few intro tutorials on YouTube such as https://www.youtube.com/watch?v=gEARZ20M-Zc

Unity can launch a variety of external editors for working with code - a good multi-platform one is Microsoft's Visual Studio Code - https://code.visualstudio.com/

Unity's preferences allow you to set an external editor under External Tools.

Note that VR and AR libraries are also changing regularly and often rapidly, so when you look for documentation online please be sure to check the version number.


Augmented Reality / Vuforia (which we will use in Projects 1 and 2)

There are currently a variety of AR platforms under development. Apple has ARKit https://developer.apple.com/arkit/, Google has ARCore https://developers.google.com/ar/. There is the venerable ARToolkit http://www.hitl.washington.edu/artoolkit/. Unity is trying to unify them with ARfoundation but cross-platform development is still tricky, so we are going to stick with Vuforia for the class, which is less powerful than the most modern libraries but works on a wider variety of hardware and is easier to develop cross-platform apps.


You should:


Create a free account at Vuforia - https://developer.vuforia.com/

Create a new 3D Project in Unity 2019.4.1f1

Add Vuforia Engine (currently 9.3) to a Unity Project using - https://developer.vuforia.com/downloads/sdk

Opening the downloaded file should automatically take you into the unity project you created above. Import everything and let it update.

Under the GameObject menu you should now see Vuforia Engine as one of the sub menu options.

In the Unity Scene Hierarchy delete the Main Camera since Vuforia will add its own

Under the GameObject menu you should now see Vuforia Engine and from there you can pick an AR Camera to add to the scene, and an Image Target, which should automatically load in the default database.

Clicking on the ImageTarget in the hierarchy you can set the type to From Database and it should automatically pick the astronaut. If not pick the VuforiaMars_Images Database and the Astronaut Image Target.

You can use GameObject to create a new 3D Object / cube (which will initially be 1m by 1m by 1m) so scale it to 0.01 by 0.01 by 0.01 and move it a little above the astronaut image.

Make the cube a child of the ImageTarget so the cube will only appear when the camera sees that specific image target.

Under Window click Vuforia Configuration and check the Camera Device. Make sure it is the correct one.

Have a copy of the astronaut image target on hand, either a print our or on a pad or phone.

Click Play and point your camera at the astronaut image and you should see a little virtual cube floating on top of the actual marker sitting on an actual desk.



Related to Project 1 we are now going to turn this into an AR widget to show the current time, so you can make the box bigger (say 0.05, 0.05. 0.01)

From Game Object / 3D Object, create a new Text - TextMeshPro and call it TimerText. Make TimerText a child of the Image Target as well. Go to Window TetMeshPro - Font Asset Creator and if this is the first time using TMP let it Import TMP Essentials and the Examples & Extras. Now if you click on the Text (TMP) item in the hierarchy you can type some text into the Text field and you should see it. By default the text will be huge so scale it down so it fits in front of the cube you just resized. Since we are going to use the text for a clock, in the Text field type a time like 12:57 PM to get the size right until we start getting the actual time.

More details on TextMeshPro at https://docs.unity3d.com/Packages/com.unity.textmeshpro@2.1/manual/index.html


If you press play, pointing the camera at the Image Target should now show a box with the static time text you added. Now to make it actually give the current time ...

With TimerText element selected in the hierarchy go over to the Inspector and Add Component and add a New Script called timeTeller. Open up the script and past in the following code which will regularly get the system time, format it, and display it on our AR clock.

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using TMPro;

public class timeTeller : MonoBehaviour
{
    public GameObject timeTextObject;

    // Start is called before the first frame update
    void Start()
    {
    InvokeRepeating("UpdateTime", 0f, 10f);   
    }

    // Update is called once per frame
    void UpdateTime()
    {
    timeTextObject.GetComponent<TextMeshPro>().text = System.DateTime.Now.ToString("h:mm tt");
       
    }
}

then in the Inspector for TimerText you will have a new Time Text Object (Script) field so click and drag TimerText from the hierarchy into it so the code knows where to put the formatted time (into the Text field of the TextMeshPro - Text of this object)


Pressing play again and pointing the camera at the astronaut marker should now show you the current time formatted US style.

But if I take the marker away the object doesn't disappear. Vuforia defaults to  Extended Tracking to deal with loss of tracking, and that can be a good thing in the real world, but for now it looks goofy. We can fix that by looking at the Default Trackable Event Handler Script for the Image Target. With this we can set the target being visible to when its Tracked, and then when its found set the GameObject of its widget (ClockWidget) to be Active (checked) and when tracking is lost set the GameObject of its widget (ClockWidget) to not be Active (unchecked)

  

and then looking through the webcam the clock will appear on the table where the marker is located, in this case with a bit more work to make it look like a traditional clock. Moving the marker moves the 3Daugmented reality clock. Moving the webcam you should be able to look around the back of the clock as though it was a real object. Note that it will always be visible in front of any real objects on your table.

What if I want another AR widget. First we need to go to the Vuforia Configuration and set the Max Simultaneous Tracked Images to something bigger than 1.


Add a new ImageTarget, and this time pick the Drone Image Target from the VuforiaMars_Images Database so we can have both of them active at once and move it away from the first one in the sceen so we can see them separately. For now we can copy the same cube we used for the time and make it a child of the new ImageTarget.

Lets go over and sign up for a free account over at OpenWeather - https://openweathermap.org/ and then verify your email address, and then I get my API key. This will basically be the same procedure for any site willing to let you have access to some of their data at some limited rate of speed without paying.


You can check that the key is working by asking for the weather in the Chicago area using your key in place of <Key>

http://api.openweathermap.org/data/2.5/weather?lat=41.88&lon=-87.6&APPID=<Key>

Giving you something like this:

Received: {"coord":{"lon":-87.6,"lat":41.88},"weather":[{"id":800,"main":"Clear","description":"clear sky","icon":"01n"}],"base":"stations","main":{"temp":298.15,"feels_like":300.07,"temp_min":298.15,"temp_max":298.15,"pressure":1019,"humidity":71},"wind":{"speed":2.11,"deg":17},"clouds":{"all":10},"dt":1593834445,"sys":{"type":3,"id":2005153,"country":"US","sunrise":1593771616,"sunset":1593826131},"timezone":-18000,"id":4887398,"name":"Chicago","cod":200}

We can specify whether we want imperial or metric results by adding &units=imperial or &units=metric

Lets create a new script component under the cube copy and call it WeatherAPIScript and add some code so it prints out the same data on the Unity console.

Unity has a nice example at https://docs.unity3d.com/ScriptReference/Networking.UnityWebRequest.Get.html

public class WeatherAPIScript : MonoBehaviour
{
    public GameObject weatherTextObject;
       string url = "http://api.openweathermap.org/data/2.5/weather?lat=41.88&lon=-87.6&APPID=<key>&units=imperial";

   
    void Start()
    {

    // wait a couple seconds to start and then refresh every 900 seconds

       InvokeRepeating("GetDataFromWeb", 2f, 900f);
   }

   void GetDataFromWeb()
   {

       StartCoroutine(GetRequest(url));
   }

    IEnumerator GetRequest(string uri)
    {
        using (UnityWebRequest webRequest = UnityWebRequest.Get(uri))
        {
            // Request and wait for the desired page.
            yield return webRequest.SendWebRequest();


            if (webRequest.isNetworkError)
            {
                Debug.Log(": Error: " + webRequest.error);
            }
            else
            {
                // print out the weather data to make sure it makes sense
                Debug.Log(":\nReceived: " + webRequest.downloadHandler.text);
            }
        }
    }
}

Then the JSON needs to get parsed into something useful. You can just use C#'s string manipulation code (IndexOf, Substring, Parse, etc.) in this simple case and then display the text as in the clock example, and then use those values to update the 3D objects (hands on a clock, direction of a wind sock, whether its raining or not, etc.).

A couple things that may trip you up in this tutorial - if you have multiple cameras attached to your computer then you may need to go to Window/VuforiaConfiguration and see which Camera Device Vuforia wants to use and make sure its the one you want it to use. When you play your application it should show the view from the camera you want. Another is a common issue in Unity that any changes you make while the game is playing don't remain after the game stops playing, so if you want to make permanent changes to your project make sure the game isn't still running (the play button is highlighted while the game is running)

For the projects you will also need to interact with these virtual objects through virtual buttons. This is a pretty nice button tutorial - https://library.vuforia.com/articles/Solution/How-To-Implement-Virtual-Buttons.html and a video on the same topic https://www.youtube.com/watch_popup?v=ElmzIq6stNI

Unity wouldn't be Unity if things didn't change constantly, though this time its a change in the way Vuforia sets up buttons. While the tutorials are still good in terms of setting up the buttons, the scripting logic has changed. This code worked when I added it to my astronaut marker.

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using Vuforia;

public class click : MonoBehaviour
{
    public GameObject vbBtnObj;

    // Start is called before the first frame update
    void Start()
    {
        vbBtnObj = GameObject.Find("vb");
        vbBtnObj.GetComponent<VirtualButtonBehaviour>().RegisterOnButtonPressed(OnButtonPressed);
        vbBtnObj.GetComponent<VirtualButtonBehaviour>().RegisterOnButtonReleased(OnButtonReleased);
 
    }

public void OnButtonPressed(VirtualButtonBehaviour vb)
{
         Debug.Log("Button pressed");
}
public void OnButtonReleased(VirtualButtonBehaviour vb)
{
         Debug.Log("Button released");
}

    // Update is called once per frame
    void Update()
    {
    }
}


In all your projects, make sure you have the rights to use any assets you find on the internet, and make sure you cite their creators. Any code should be your own.


Virtual Reality / VRTK (which we will use in Project 3)

With several different manufacturers of VR HMD there was a lack of standardization in the libraries to drive them which continues on, though pretty much everyone ties into Unity or Unreal. We are also starting to see browser-based VR solutions such as WebXR to try and create more widely distributable applications, including a Unity export Plugin, but WebXR itself is still not sufficiently cross-platform yet.

Given this current state we are going to use VRTK version 3 for our VR work. It is a nice cross platform library for Unity with a built in simulator and a decent amount of online documentation. The more recent VRTK version 4 remains in beta and has not been stable enough.

For now I am going to show what is likely to be the starter project that you will be adding onto for the third project, which is a version of the classroom that we would be having this class in if we were on campus. I will show how this looks in the simulator, and how it looks using a VR headset. Again this will make use of Unity, but VRTK will be handling the stereo visuals, and the interaction.

There are a lot of helpful YouTube VRTK tutorials, including the following that you will want to take a look at before you get to project 3, and possibly earlier if you want to get a better idea what is coming. Again, be careful that you are following the V3 tutorials:



Content Generation Tools

Along with the main engine there are a variety of tools that can be used to create models, sounds, textures, etc. Here you are free to use the tools of your choice. The following are pretty decent free ones.

There are also various websites that provide Open Source / Royalty Free resources, just be sure to cite the creators and where you downloaded the assets. Soundbible is a nice place for free sounds http://soundbible.com/. There are several lists on-line for good 3D model repositories, here is one:
https://all3dp.com/1/free-3d-models-download-best-sites-3d-archive-3d/


GitHub

We will be using GitHub for turning in the projects - https://github.com/

If you don't have a git account yet you should sign up for one

You (or your group if we are doing groups) should create a repository for each project.

Some Tutorials
GitHub has their own getting started page which is a nice intro -
https://guides.github.com/activities/hello-world/


This is a nice intro for the basics - https://rubygarage.org/blog/most-basic-git-commands-with-examples

Quick Command List - https://rogerdudler.github.io/git-guide/

GitHub and Unity - https://thoughtbot.com/blog/how-to-git-with-unity

At minimum you will be using git to turn in all your code and assets in your Unity project, but I would also recommend regularly updating your code on git so there is some external proof of when you submitted, as well as having backup copies at various checkpoints. Git can also be useful in its typical role for group projects, though that does require some tweaks for it to play nice with unity. Note that git can also be a nice place to store a copy of your website files to prove they were done on time, and you can host your website for your project on git as well if you prefer. Note that I would not rely solely on Git for backing up your projects - keep multiple backups in multiple places.

For example I could take my Vuforia project above and move it up to a new git repository:

(I updated this sequence in mid September to be more space efficient so it should take up much less space and be much faster to update as it should ignore many more files that we don't need to put on git. The previous sequence still works, but this is a better one to use going forward.)

- I already have a Git account so I can use my favorite web browser to go to github.com and sign in and use the + in the upper right corner to add a New repository
- I can give it a name like Week1, make it private for now, and add a .gitignore file of type Unity. Then I click the green button to create the repository
- Keep this tab open in your browser - we will come back to it.

- Back in my Unity project I need to set things up so the project plays nice with git so I am going to Edit the Project Settings for the Editor. Version Control Mode should be Visible Meta Files, and Asset Serialization Mode should be Force Text.

- Go to the terminal and change the current directory to the one for your Unity project, e.g. cd Documents/unity folder/428F20P1

- If you look at the files in that directory you should see an Assets folder, Packages, Logs, etc. type git init to create a new .git folder. 

- Now we need to link this local repository to GitHub so if you go back to GitHub and look at the Code tab and then click on the green Code button you can get the https link which will look something like https://github.com/YourGitAccountName/YourGitProjectName.git - back in the terminal you can use the command git remote add origin https://github.com/YourGitAccountName/YourGitProjectName.git so in my case git remote add origin https://github.com/andyevl/Week1.git

- Before we can push our project code up to GitHub we need to pull that .gitignore file we added to the repository when we created it. The .gitignore file tells git not to backup files that Unity can recreate (Library, Temp, Build, etc.) which will save a lot of space in Git, but will increase the time when you open the project for the first time. We can do that with the command git pull origin master --allow-unrelated-histories

    after October 1 2020 instead you should: git pull origin main --allow-unrelated-histories

- You can then add all the files to the list to be tracked with git add --all

- You can then commit the changes with an initial comment (feel free to modify the comment) with git commit -m 'starter project'

- You can check the status with git status. Status is helpful to tell you which files are being tracked, and which are untracked, which have been modified, etc.

   - Another helpful command with the recent switch from master to main is git branch. This gives you the name of the branch you are linked to. After October 2020 it should be 'main'. if it is 'master' then a handy command is git branch -mv master main to rename the branch to main in order to match Git's new default.

- Now we can finally push our code up onto GitHub with git push origin master

    after October 1 2020 instead you should: git push origin main

- This will be slightly unhappy since it wants a comment but you can just save out of the editor.

- If you now go back to GitHub and refresh the page you should see a lot more files there with their modification dates.

- Whenever you want to update the repository at GitHub with your latest version you would go to the top level of your Unity project directory and type this sequence to update only those files that have changed (or added to the repository):

- git add --all
-
git commit -m 'new comment for this version'
-
git push origin master

    after October 1 2020 instead you should: git push origin main

- At some point if you have files that are too big (> 100 MB) then your push will fail with an error telling you which files are too big. We can get around this by installing Git Large File Storage, so follow the instructions at this link for your platform: https://docs.github.com/en/github/managing-large-files/installing-git-large-file-storage - Once it is installed correctly you should be able to type git lfs install from your terminal and see Git LFS Initialized.

We can tell LFS to track these files with git lfs track <filename> and then make sure that the new .gitattributes file is tracked with git add .gitattributes - note that you need to tell LFS track files before you git add them or you have to git lfs migrate those big files over to LFS. If you want to stop tracking something in LFS then you can give the command git lfs untrack <filename> and then you can go back and add them to LFS if you want to make them available on GitHub, or add them to the ignore list if you don't want to store them on GitHub. There is a pretty good discussion of this here. Note that LFS is limited to 1gig of storage and 1 gig of downloads per month.

- At the alpha deadline you will add the TA as a member of your team to be able to look at your code. For the final submission you should make your project public  so everyone can look at everyone else's code. Before each of those dates you should re-download your Project from GitHub to make sure it works. Probably the easiest way to do this is to try and Download Zip from under the green Code button, put it into a new directory on your computer, and try to add it to Unity Hub and start it up in Unity to make sure everything made it to and from GitHub correctly.

- If you find you need to remove a file from LFS to get some space back then you can use the command git lfs migrate export --include <filename?>


One important thing to keep in mind with VR and AR is that, unlike other media, the size of the screen does not affect the size of the objects in the world. If you are watching a movie then Humphrey Bogart or Lego Batman may vary from an inch tall to 100 feet tall depending on the display you are watching it on. In VR or AR the display is a window into the virtual world, and changing the size of the window just gives you a bigger or smaller window, it doesn't change the size of the things the window looks out on. Scale is absolute in VR or AR, so at some point the environment you create ends up being scaled in units of feet or meters for people to move around in. Similarly there are various image based tricks that can be used in 2D or even 3D movies, that don't work in VR or AR where the space is really 3D.

In this course you will be doing a lot of work with Unity and the content generation tools so make sure you have access to a decent computer that can run these pieces of software. For the AR assignments you will need access to a webcam for that computer - built in cameras should be OK but an external one you can move around will make your life a lot easier.


Building Vuforia apps for your phone

If you don't have a webcam handy, or you just want to see this working on your actual smartphone, here are the steps you can use to get your project running on a moderately new smartphone. Note that each of these builds is a multi-step process so its not as fast as testing with a webcam, but its more realistic, and can be more satisfying.

  • In Unity Hub click on installs and select 2019.4.1f1 and click on the mini hamburger (3 vertical dot) menu and Add Modules. If you have an iOS device then check iOS Build Support. If you want to run on an Android device then select Android Build Support and click Done. Give Unity time to download all the necessary additions.
  • Open up your project in Unity 2019.4.1f1 and go to File / Build Settings. If your scene is not listed at the top in 'Scenes in Build' then add your scene. If you want to build for iOS then click on iOS in the Platform list and click on Switch Platform. Similarly if you want Android then click on Android in the Platform list and and click on Switch Platform. Unity will install a bunch of assets.


iOS

  • In Unity go to Edit / Project Settings / Player. Below 'Cursor Hotspot' there should be a horizontal menu with icons for PC / iOS / Android. Click on iOS in the middle. Midway down you should see 'Camera Usage Description'. Type something in like 'Vuforia needs access to your camera'. This will appear when the app first tries to use your camera so a user knows why the app to wants access to the camera.

  • In Unity click on Build. It will ask you for a directory to put the build. This will take about a gig of space. Note that you probably don't want to build this within your Unity project directory as that will dramatically increase the number of files being backed up to GitHub.

  • If you do not have Xcode installed you will need to install it from the App Store. This may take some time, and about 10 gig of disk space. You can find it with the other Applications.
  • Its important to have the correct version of Xcode for your version of iOS. Usually having the latest release version of both is best, otherwise this page helps with the mapping -
    https://developer.apple.com/documentation/xcode-release-notes

  • Launch Xcode and then open the directory that you did your Unity iOS build into. Xcode will do a little bit of processing and tell you there are some warnings. These are probably fine.

  • Plug in your iPhone with the appropriate cable. If this is the first time then Xcode will want to link it to an iCloud account. You should now see your phone listed at the top of the screen 'Unity-iPhone > your-phone-name'. If not, click on Generic iOS Device and you should see your phone in the drop-down list.
  • Scroll to the top of the big list on the left and click on the top item Unity-iPhone. This will bring up a bunch of settings for the project. Click on 'Signing & Capabilities'. It will probably say Team: None. You need to give it a team or you will get a build error. If you already have an account select it. If not you need to create a new one. Make sure that 'Automatically manage signing' is checked as this simplifies things dramatically with regards to certificates.
  • If you want to avoid doing this every time, you can follow the instructions here to get your team ID to give to Unity - https://steemit.com/xcode/@sham-san/how-to-link-your-xcode-signing-team-id-in-unity-when-you-have-a-free-account
  • Now you should be able to click on the '>' button at the top and the app should build, install, and run on your phone. If its the first time it should ask you for permission to use the camera and after that it should work the same as the webcam version. You also get access to the iOS debugging tools.
  • If you are using a free account (rather than a paid developer account) you will need to give permission in Settings for the phone to run this unidentified application you just created.




  • You now have a version of the app on your phone that can work independently of Unity or Xcode.


Android


  • In Unity go to Edit / Project Settings / Player / Other Settings and select Minimum API Level to Android 4.4 Kitkat (API level 19)
  • Connect your android phone to the computer with the appropriate cable. On the phone go to Settings, select Developer options, and then enable USB debugging. If you don't see developer options, go to Settings > About Phone > Build Number and tap the "Build Number" option 7 times to enable developer options. Once developer options are enabled, turn on USB debugging.

  • In Unity go to  File / Build Settings and hit Refresh' next to Run Device. You should now see the connected mobile phone in the dropdown menu. Select the phone and hit Build and Run. If you are prompted to save the apk, save it on your computer. The project will build and run automatically on your phone.



Coming Next Time

History of VR and AR



last revision

11/26/2020 - updated notes for new 2020 git nomenclature for the one command that I missed earlier

10/06/2020 - updated the code for Vuforia buttons

10/02/2020 - updated notes for new 2020 git nomenclature

9/29/2020 - tweaking the git notes to create a revised simpler git video

9/19/2020 - reordered the git notes some more to reduce the repository size and reduce the need for Git LFS in Project 1

9/17/2020 - added notes for not backing up ArtifactDB to save lfs space

9/1/2020 - clarified instructions for building on iOS

8/28/2020 - added instructions for building on iOS and Android (thanks Sai)

8/27/2020 - clarified the installation instructions for vuforia