Week 1 we
showed off a bunch of different sample VR / AR worlds, so this
week we are going to talk about how to build them, in particular
with respect to Project 1.
projects in the class we will be using Unity3D so people can
share experiences and issues.
Unity3D runs on windows and OS-X so you can do much of your development on a laptop and then move over to a machine connected to a VR / AR display for testing. Under normal circumstances I would highly suggest that you test on the actual hardware regularly so you don't go too far down a development path that wont work for the final deployment. In the lab, there is one VIVE in the main lab, and another in the classroom, both connected to Windows PCs. The one in the classroom has a webcam attached for AR work. This term with social distancing in force we will try and minimize the need for testing in the classroom, but I hope people will be able to come in before the end of the term to experience VR for real.
started go to https://unity.com/ and Get
Started, Click on the Individual Tab, Under Personal get
Started. Either the First-time users or Returning users start by
downloading UnityHubSetup and launching it.
gives you a central place for your various projects, and various
installs on Unity. Once you have the Hub running you should
click on Installs and then Add to add a new Unity version. The
version we want is not one of the ones in the short menu so
click to visit the download archive, click on the Unity 2019.x
tab, and pick unity 2019.4.1f1 to install with
Unity Hub. Note that there is a new release of Unity roughly
every week, and things can quickly get unmanageable if everyone
is using a different release, so that is why we are
standardizing on one version.
You should have it install:
Normally everyone would be
presenting their work on the windows 10 PC in the classroom. If
you are on OS-X and forget to install the Windows Build
Support, you can come back and install it
later from the Unity Hub. If you want (or feel the need) to
install other components that should be fine. Be prepared to
lose 4-5 gigabytes of disc space and at least 10 minutes for the
install. You can have multiple versions of Unity installed at
the same time accessible through the Unity Hub in case you need
to have different versions running for different classes or
you will be using a combination of an IDE and writing C# code,
so its a good idea to go through the introductory tutorials if
you haven't used Unity before (e.g. in the CS Video Game
are a good place to start learning the Unity3D IDE, and there
are quite a few intro tutorials on YouTube such as https://www.youtube.com/watch?v=gEARZ20M-Zc
can launch a variety of external editors for working with code -
a good multi-platform one is Microsoft's Visual Studio Code - https://code.visualstudio.com/
preferences allow you to set an external editor under External
that VR and AR libraries are also changing regularly and often
rapidly, so when you look for documentation online please be
sure to check the version number.
Reality / Vuforia (which we will use in Projects 1 and 2)
currently a variety of AR platforms under development. Apple has
Google has ARCore https://developers.google.com/ar/.
There is the venerable ARToolkit http://www.hitl.washington.edu/artoolkit/.
Unity is trying to unify them with ARfoundation but
cross-platform development is still tricky, so we are going to
stick with Vuforia for the class, which is less powerful than
the most modern libraries but works on a wider variety of
hardware and is easier to develop cross-platform apps.
Create a free account at Vuforia - https://developer.vuforia.com/
Create a new 3D Project in Unity 2019.4.1f1
Engine (currently 9.3) to a Unity Project using - https://developer.vuforia.com/downloads/sdk
Opening the downloaded file should automatically take you into the unity project you created above. Import everything and let it update.
GameObject menu you should now see Vuforia Engine as one of
the sub menu options.
Unity Scene Hierarchy delete the Main Camera since Vuforia
will add its own
GameObject menu you should now see Vuforia Engine and from
there you can pick an AR Camera to add to the scene, and an
Image Target, which should automatically load in the default
Clicking on the ImageTarget in the hierarchy you can set the type to From Database and it should automatically pick the astronaut. If not pick the VuforiaMars_Images Database and the Astronaut Image Target.
You can use GameObject to create a new 3D Object / cube (which will initially be 1m by 1m by 1m) so scale it to 0.01 by 0.01 by 0.01 and move it a little above the astronaut image.
Make the cube
a child of the ImageTarget so the cube will only appear when the
camera sees that specific image target.
Under Window click Vuforia Configuration and check the Camera Device. Make sure it is the correct one.
Have a copy of the astronaut image target on hand, either a print our or on a pad or phone.
and point your camera at the astronaut image and you should see
a little virtual cube floating on top of the actual marker
sitting on an actual desk.
Project 1 we are now going to turn this into an AR widget to
show the current time, so you can make the box bigger (say 0.05,
Object / 3D Object, create a new Text - TextMeshPro and call it
TimerText. Make TimerText a child of the Image Target as well. Go to Window TetMeshPro - Font
Asset Creator and if this is the first time using TMP let it
Import TMP Essentials and the Examples & Extras. Now if you
click on the Text (TMP) item in the hierarchy you can type some
text into the Text field and you should see it. By default the
text will be huge so scale it down so it fits in front of the
cube you just resized. Since we are going to use the text for a
clock, in the Text field type a time like 12:57 PM to get the
size right until we start getting the actual time.
on TextMeshPro at https://docs.unity3d.com/Packagesemail@example.com/manual/index.html
If you press
play, pointing the camera at the Image Target should now show a
box with the static time text you added. Now to make it actually
give the current time ...
TimerText element selected in the hierarchy go over to the
Inspector and Add Component and add a New Script called
timeTeller. Open up the script and past in the following code
which will regularly get the system time, format it, and display
it on our AR clock.
public class timeTeller : MonoBehaviour
public GameObject timeTextObject;
// Start is called before the first frame update
InvokeRepeating("UpdateTime", 0f, 10f);
// Update is called once per frame
timeTextObject.GetComponent<TextMeshPro>().text = System.DateTime.Now.ToString("h:mm tt");
then in the Inspector for TimerText you will have a new Time Text Object (Script) field so click and drag TimerText from the hierarchy into it so the code knows where to put the formatted time (into the Text field of the TextMeshPro - Text of this object)
again and pointing the camera at the astronaut marker should now
show you the current time formatted US style.
But if I take
the marker away the object doesn't disappear. Vuforia defaults
to Extended Tracking to deal with loss of tracking, and
that can be a good thing in the real world, but for now it looks
goofy. We can fix that by looking at the Default Trackable Event
Handler Script for the Image Target. With this we can set the
target being visible to when its Tracked, and then when its
found set the GameObject of its widget (ClockWidget) to be
Active (checked) and when tracking is lost set the GameObject of
its widget (ClockWidget) to not be Active (unchecked)
looking through the webcam the clock will appear on the table
where the marker is located, in this case with a bit more work
to make it look like a traditional clock. Moving the marker
moves the 3Daugmented reality clock. Moving the webcam you
should be able to look around the back of the clock as though it
was a real object. Note that it will always be visible in front
of any real objects on your table.
What if I
want another AR widget. First we need to go to the Vuforia
Configuration and set the Max Simultaneous Tracked Images to
something bigger than 1.
Add a new
ImageTarget, and this time pick the Drone Image Target from the
VuforiaMars_Images Database so we can have both of them active
at once and move it away from the first one in the sceen so we
can see them separately. For now we can copy the same cube we
used for the time and make it a child of the new ImageTarget.
Lets go over
and sign up for a free account over at OpenWeather - https://openweathermap.org/
and then verify your email address, and then I get my API key.
This will basically be the same procedure for any site willing
to let you have access to some of their data at some limited
rate of speed without paying.
You can check that the key is working by asking for the weather in the Chicago area using your key in place of <Key>
something like this:
We can specify whether we want imperial or metric results by adding &units=imperial or &units=metric
Lets create a
new script component under the cube copy and call it WeatherAPIScript and add some code
so it prints out the same data on the Unity console.
Unity has a nice example at https://docs.unity3d.com/ScriptReference/Networking.UnityWebRequest.Get.html
WeatherAPIScript : MonoBehaviour
public GameObject weatherTextObject;
string url = "http://api.openweathermap.org/data/2.5/weather?lat=41.88&lon=-87.6&APPID=<key>&units=imperial";
// wait a couple seconds to start and then refresh every 900 seconds
InvokeRepeating("GetDataFromWeb", 2f, 900f);
IEnumerator GetRequest(string uri)
using (UnityWebRequest webRequest = UnityWebRequest.Get(uri))
// Request and wait for the desired page.
yield return webRequest.SendWebRequest();
Debug.Log(": Error: " + webRequest.error);
// print out the weather data to make sure it makes sense
Debug.Log(":\nReceived: " + webRequest.downloadHandler.text);
Then the JSON
needs to get parsed into something useful. You can just use C#'s
string manipulation code (IndexOf, Substring, Parse, etc.) in
this simple case and then display the text as in the clock
example, and then use those values to update the 3D objects
(hands on a clock, direction of a wind sock, whether its raining
or not, etc.).
things that may trip you up in this tutorial - if you have
multiple cameras attached to your computer then you may need to
go to Window/VuforiaConfiguration and see which Camera Device
Vuforia wants to use and make sure its the one you want it to
use. When you play your application it should show the view from
the camera you want. Another is a common issue in Unity that any
changes you make while the game is playing don't remain after
the game stops playing, so if you want to make permanent changes
to your project make sure the game isn't still running (the play
button is highlighted while the game is running)
For the projects you will also need to interact with these virtual objects through virtual buttons. This is a pretty nice button tutorial - https://library.vuforia.com/articles/Solution/How-To-Implement-Virtual-Buttons.html and a video on the same topic https://www.youtube.com/watch_popup?v=ElmzIq6stNI
Unity wouldn't be Unity if
things didn't change constantly, though this time its a change
in the way Vuforia sets up buttons. While the tutorials are
still good in terms of setting up the buttons, the scripting
logic has changed. This code worked when I added it to my
public class click : MonoBehaviour
public GameObject vbBtnObj;
// Start is called before the first frame update
vbBtnObj = GameObject.Find("vb");
public void OnButtonPressed(VirtualButtonBehaviour vb)
public void OnButtonReleased(VirtualButtonBehaviour vb)
// Update is called once per frame
In all your
projects, make sure you have the rights to use any assets you
find on the internet, and make sure you cite their creators. Any
code should be your own.
Virtual Reality / VRTK (which we will use in Project 3)With several different manufacturers of VR HMD there was a lack of standardization in the libraries to drive them which continues on, though pretty much everyone ties into Unity or Unreal. We are also starting to see browser-based VR solutions such as WebXR to try and create more widely distributable applications, including a Unity export Plugin, but WebXR itself is still not sufficiently cross-platform yet.
Given this current state we are
going to use VRTK version 3 for our VR work. It is a nice cross
platform library for Unity with a built in simulator and a
decent amount of online documentation. The more recent VRTK
version 4 remains in beta and has not been stable enough.
now I am going to show what is likely to be the starter project
that you will be adding onto for the third project, which is a
version of the classroom that we would be having this class in
if we were on campus. I will show how this looks in the
simulator, and how it looks using a VR headset. Again this will
make use of Unity, but VRTK will be handling the stereo visuals,
and the interaction.
There are a lot of helpful YouTube VRTK tutorials, including the following that you will want to take a look at before you get to project 3, and possibly earlier if you want to get a better idea what is coming. Again, be careful that you are following the V3 tutorials:
Along with the main engine there are a variety of
tools that can be used to create models, sounds, textures,
etc. Here you are free to use the tools of your choice. The
following are pretty decent free ones.
There are also various
websites that provide Open Source / Royalty Free resources, just
be sure to cite the creators and where you downloaded the
assets. Soundbible is a nice place for free sounds http://soundbible.com/.
There are several lists on-line for good 3D model repositories,
here is one:
We will be using GitHub for turning in the projects - https://github.com/
If you don't have a git
account yet you should sign up for one
You (or your group if we are doing groups) should create a repository for each project.
GitHub has their own getting started page which is a nice intro - https://guides.github.com/activities/hello-world/
This is a nice intro for the
basics - https://rubygarage.org/blog/most-basic-git-commands-with-examples
Quick Command List - https://rogerdudler.github.io/git-guide/
GitHub and Unity - https://thoughtbot.com/blog/how-to-git-with-unity
minimum you will be using git to turn in all your code and
assets in your Unity project, but I would also recommend
regularly updating your code on git so there is some external
proof of when you submitted, as well as having backup copies at
various checkpoints. Git can also be useful in its typical role
for group projects, though that does require some tweaks for it
to play nice with unity. Note that git can also be a nice place
to store a copy of your website files to prove they were done on
time, and you can host your website for your project on git as
well if you prefer. Note that I would not rely solely on Git for
backing up your projects - keep multiple backups in multiple
For example I could take my Vuforia project above and move it up to a new git repository:
(I updated this sequence in mid
September to be more space efficient so it should take up much
less space and be much faster to update as it should ignore
many more files that we don't need to put on git. The previous
sequence still works, but this is a better one to use going
I already have a Git account so I can use my favorite web
browser to go to github.com and sign in and use the + in the
upper right corner to add a New repository
- I can give it a name like Week1, make it private for now, and add a .gitignore file of type Unity. Then I click the green button to create the repository
- Keep this tab open in your browser - we will come back to it.
Back in my Unity project I need to set things up so the project
plays nice with git so I am going to Edit the Project Settings
for the Editor. Version Control Mode should be Visible Meta
Files, and Asset Serialization Mode should be Force Text.
Go to the terminal and change the current directory to the one
for your Unity project, e.g. cd Documents/unity
- If you look at the files in that directory you should see an Assets folder, Packages, Logs, etc. type git init to create a new .git folder.- Now we need to link this local repository to GitHub so if you go back to GitHub and look at the Code tab and then click on the green Code button you can get the https link which will look something like https://github.com/YourGitAccountName/YourGitProjectName.git - back in the terminal you can use the command git remote add origin https://github.com/YourGitAccountName/YourGitProjectName.git so in my case git remote add origin https://github.com/andyevl/Week1.git
- Before we can push our project code up to GitHub we need to pull that .gitignore file we added to the repository when we created it. The .gitignore file tells git not to backup files that Unity can recreate (Library, Temp, Build, etc.) which will save a lot of space in Git, but will increase the time when you open the project for the first time. We can do that with the command git pull origin master --allow-unrelated-histories
after October 1 2020 instead you should: git pull origin main --allow-unrelated-histories- You can then add all the files to the list to be tracked with git add --all
You can then commit the changes with an initial comment (feel
free to modify the comment) with git commit -m 'starter
You can check the status with git status.
Status is helpful to tell you which files are being tracked, and
which are untracked, which have been modified, etc.
- Another helpful command with the
recent switch from master to main is git branch.
This gives you the name of the branch you are linked to. After
October 2020 it should be 'main'. if it is 'master' then a
handy command is git branch -mv master main
to rename the branch to main in order to match Git's new
Now we can finally push our code up onto GitHub with git
push origin master
after October 1 2020 instead you should: git push origin main
- This will be slightly unhappy since it wants a comment but you can just save out of the editor.
If you now go back to GitHub and refresh the page you should see
a lot more files there with their modification dates.
- Whenever you want to update the repository at GitHub with your latest version you would go to the top level of your Unity project directory and type this sequence to update only those files that have changed (or added to the repository):
git add --all
- git commit -m 'new comment for this version'
- git push origin master
after October 1 2020 instead you should: git push origin main
- At some point if you have files that are too big (> 100 MB) then your push will fail with an error telling you which files are too big. We can get around this by installing Git Large File Storage, so follow the instructions at this link for your platform: https://docs.github.com/en/github/managing-large-files/installing-git-large-file-storage - Once it is installed correctly you should be able to type git lfs install from your terminal and see Git LFS Initialized.
tell LFS to track these files with git lfs track
<filename> and then make
sure that the new .gitattributes file is tracked with
git add .gitattributes - note that you need to
tell LFS track files before you git add them or you have to
git lfs migrate those big files over to LFS. If you want to stop
tracking something in LFS then you can give the command
git lfs untrack <filename>
and then you can go back and add them to LFS if you want to
make them available on GitHub, or add them to the ignore list
if you don't want to store them on GitHub. There is a pretty
good discussion of this here. Note that LFS is limited to 1gig
of storage and 1 gig of downloads per month.
- At the alpha deadline you will add the TA as a member of your team to be able to look at your code. For the final submission you should make your project public so everyone can look at everyone else's code. Before each of those dates you should re-download your Project from GitHub to make sure it works. Probably the easiest way to do this is to try and Download Zip from under the green Code button, put it into a new directory on your computer, and try to add it to Unity Hub and start it up in Unity to make sure everything made it to and from GitHub correctly.
- If you find you need to remove a file from
LFS to get some space back then you can use the command git
lfs migrate export --include <filename?>
thing to keep in mind with VR and AR is that, unlike other
media, the size of the screen does not affect the size of the
objects in the world. If you are watching a movie then Humphrey
Bogart or Lego Batman may vary from an inch tall to 100 feet
tall depending on the display you are watching it on. In VR or
AR the display is a window into the virtual world, and changing
the size of the window just gives you a bigger or smaller
window, it doesn't change the size of the things the window
looks out on. Scale is absolute in VR or AR, so at some point
the environment you create ends up being scaled in units of feet
or meters for people to move around in. Similarly there are
various image based tricks that can be used in 2D or even 3D
movies, that don't work in VR or AR where the space is really
In this course you will be doing a lot of work with Unity and the content generation tools so make sure you have access to a decent computer that can run these pieces of software. For the AR assignments you will need access to a webcam for that computer - built in cameras should be OK but an external one you can move around will make your life a lot easier.
Building Vuforia apps for your phone
If you don't
have a webcam handy, or you just want to see this working on
your actual smartphone, here are the steps you can use to get
your project running on a moderately new smartphone. Note that
each of these builds is a multi-step process so its not as fast
as testing with a webcam, but its more realistic, and can be
History of VR and AR
11/26/2020 - updated notes
for new 2020 git nomenclature for the one command that I missed
10/06/2020 - updated the code
for Vuforia buttons
10/02/2020 - updated notes
for new 2020 git nomenclature
9/29/2020 - tweaking
the git notes to create a revised simpler git
reordered the git notes some more to reduce the
repository size and reduce the need for Git LFS in
9/17/2020 - added notes for
not backing up ArtifactDB to save lfs space
9/1/2020 - clarified
instructions for building on iOS
8/28/2020 - added instructions for building on iOS
and Android (thanks Sai)
8/27/2020 - clarified the installation
instructions for vuforia