Prepare For VR Development

1. Create a new folder on Desktop and give it a name, say “Lab4”.

2. Create a new project at the position of “Lab4”

3. In this lab, we start to introduce how to develop VR software/game with Unity3D. This lab will be split into Oculus and Vive, so please follow the correct guide for the device you're using.

Oculus VR Development

Note that this section is only for Oculus Rift and Oculus Rift S. If you want to develop on the HTC Vive, click here. If you want to develop on Oculus Quest, try following the instructions on this page.

4. To start using VR with Oculus, you need to download the Oculus software from the Oculus website and go through the steps to connect your Oculus device to your PC. Once that is done, go to Oculus > Setting > General > Unknown Source. You need to enable Unknown Source, or you cannot preview the project with your Oculus Device.

5. Next, in your Unity project, go to Edit > Project Settings > XR Plugin Management and click on Install XR Plugin Management.

Once that is done, you should see a screen like this.

Click on the box next to Oculus. That will install the Oculus package.

6. Next, go to the Unity Store by clicking Window > Asset Store and search for "Oculus Integration". Download and import it to your project. Make sure that all files are imported. Now, we are ready to go into development for the Oculus Rift or Oculus Rift S.

7. You will notice that there is now a "Oculus" folder under Assets. It contains a lot of useful scripts and prefabs that you can use with Touch controllers. However, in this tutorial we won't be using these high-level scripts, and will be creating our own for grabbing and throwing objects.

8. Navigate to Oculus > Sample Framework > Core > CustomHands > Models and find l_hand_skeletal_lowres and r_hand_skeletal_lowres. Drag them into the scene and rename them "Left Hand" and "Right Hand" respectively. Place them under the Player object (which is the Camera's parent). Then, set their position as (0, 0, 0).

9. Now, we need to map the hands' movements to the controllers' movements. Create a script called "TouchController" as follows:

This script is not specific to the left or right controller so you can use it for both controllers. Return to the editor and add this script to both Left Hand and Right Hand. You'll notice that there is a public field where you have to set the correct controller corresponding to each hand. For Left Hand, select "L Touch" and for Right Hand, select "R Touch".

10. Now, go test out the touch controllers. The hands in the VR space should move and rotate according to the controllers.

11. Now, we shall move on to the Grab function. First, we need to decide which button on the controller to use to grab objects. Refer to the diagram below to see the names of each button.

You can choose to use any button, but as a standard, grabbing objects is usually mapped to the 11th Axis and the 12th Axis, or the Left and Right hand triggers (Not to be confused with the Index triggers!)

12. Now we need to map the buttons to an input. Go to Edit > Project Settings and click on Input Manager. Create an input (Do this by increasing the size under Axes) and name in "LHandTrigger". Set Gravity to 0, Dead to 0.19 and Sensitivity to 1.5. For Axis, choose "11th Axis (Joysticks)". Duplicate LHandTrigger and rename the copy to "RHandTrigger" and choose the axis to be "12th Axis (Joysticks)".

13. Create a script called "Grab". First, we want to keep track of whether the hand is currently grabbing an object, so we have a private boolean for that. In the Update function, we check if the hand trigger is being pressed and call the functions GrabObject() and DropObject(), which we will define later. The script should now look like this:

Note that the string buttonName is there so that the script isn't particular to either the left or right controller, just like for the TouchController script.

14. Now we move on to the GrabObject() function. It works by using a sphere cast to check for items around the controller in a sphere, then picks up the nearest object. Add some lines to the top of the script so that it looks like this:

Controller is to provide a reference to the left or right controllers, grabRadius is for the range of the sphere cast, grabbedObject is to keep track of the object being held, and grabMask is a layer mask such that only objects in that layer can be grabbed.

Next, we define the GrabObject() function:

Note that the rigidbody in the grabbed object is set to kinematic so that gravity doesn't work on it while it is being held, and also the grabbed object is set as a child of the controller so that they will move together.

15. In the DropObject() function, we want to reverse everything that is done in the GrabObject() function.

16. Before we can use the script, we need to add a layer called "Grabbable". All objects that we want to be grabbed has to be in that layer.

17. Next, add the Grab script to both Left Hand and Right Hand. For Left Hand, set the Controller field as "L Touch", the Button Name field as "LHandTrigger", Grab Radius as 1, and Grab Mask as "Grabbable". For Right Hand, set the Controller field as "R Touch", the Button Name field as "RHandTrigger", Grab Radius as 1, and Grab Mask as "Grabbable".

18. Now, you can create any object, add a rigidbody, set the layer to "Grabbable", and test it in VR! Do remember to create a ground, if not the objects will just keep falling.

Note: If the touch controller input is not registering (due to bug) press the Oculus home button twice. More details af the fix is at: https://forum.unity.com/threads/oculus-touch-input-not-detected.546942/

Now skip to the UI in VR section.

HTC Vive VR Development

Note that this section is only for HTC Vive. If you want to develop on the Oculus Rift or Oculus Rift S, click here.

19. You can also use the HTC Vive instead of the Oculus Rift. The advantage is that you can use trackers to track movement of feet, for example.

20. If you are using Unity version 2020 and above, go here to install the latest release for SteamVR plugin (Do not install from Unity Asset Store)

If you are using Unity version 2019 and below, go to the Asset Store, search for "SteamVR" and download the SteamVR Plugin. Import it into the project.

Now that you have installed the SteamVR Plugin for Unity, you have to install SteamVR and its drivers. It is best to install the necessary software from Viveport, which will help you install Steam, SteamVR and setup the device. Click here to download Viveport. After installing, you should see this Device Setup screen. Choose the device you are using.

After you correctly set up your device, you are good to go!

You might encounter some problems in the above steps. If all else fails, you may choose to install the openXR plugins and SteamVR separately from Steam and https://github.com/ValveSoftware/unity-xr-plugin (Also, try to disable your antivirus or any applications that may require the use of VR if not SteamVR might not run properly)

21. We will use the camera provided by the SteamVR plugin, so delete the Main Camera in the scene. Then, go to the folder SteamVR > Prefabs and find the [CameraRig] and [SteamVR] prefabs. Drag them into the scene. Set the position of the [CameraRig] to [0, 0, -1.1]. Now you can put on the HTC Vive headset, and you should see the controllers in the virtual space.

22. In SteamVR 2.0, you need to define Actions for the headset and controllers. Go to Window > SteamVR Input and you'll see something like this.

23. Now we will attempt to implment our own function for grabbing objects with the controllers. Under the default Action Set, select the GrabGrip action and rename it to Grab. Then, click Save and Generate.

You will notice that there will be a new SteamVR_Input folder under Assets that includes all the scripts you need.

24. Here is a diagram so you can familiarize yourself with the Vive controller inputs.

In this tutorial, we will be using the Grip button to grab objects.

25. Now, we need to bind the action to the hardware. Make sure that SteamVR is open and click Open binding UI in the SteamVR Input window. A web page will open on the default browser. Under Current Controller, make sure that Vive Controller is selected. Then, below Current Binding, click on Edit. You will notice that there are already some default bindings. However, for the sake of the tutorial, delete all these inputs. Next, click the plus sign next to Grip. Select BUTTON in the window that pops up.

Another window will pop up with a list of possible actions. This time, select grab.

Click Apply on the bottom left to save the action set. Since the bindings are mirrored, this will apply to the right hand grip button as well. After that, click on Replace Default Binding to overwrite the default bindings.

26. Before using the grab action, we first need to link the controllers in virtual space to the physical controllers. Select Controller(left) under [CameraRig], then change the Pose Action to \actions\default\in\SkeletonLeftHand. Do the same for Controller(right), but select \actions\default\in\SkeletonLeftHand.

27. We will use collision triggers to hold onto objects. For that, the controllers and objects to be grabbed need rigidbodies and colliders. The balls in the scene already have them, so we can leave them alone. Add Rigidbody to both the left and right controllers. Check Is Kinematic and uncheck Use Gravity for both rigidbodies.

Next, add a Box Collider to both controllers and check Is Trigger. Set the center of the Box Collider to (0, -0.04, 0.02) and the size to (0.14, 0.07, 0.05). If you run the game and look at the controllers in the scene, you will notice that the box colliders are on the top of the controllers.

28. Create a script and name it "ControllerGrabScript" and make the following declarations:

The handType, controllerPose and grabAction store references to the hand type and actions, while collidingObject stores the GameObject the trigger is colliding with, and objectInHand serves as a reference to the object currently being grabbed.

29. Next, add these trigger methods.

30. Then, we need to define SetCollidingObject.

As you can probably tell, this method sets collidingObject unless there is already a collidingObject or if the other collider does not have a Rigidbody.

31. Now, we will add methods to grab and release objects.

Here, the FixedJoint connects the object to the controller.

32. Finally, we just need to handle the controller input in Update.

33. Now, go back to the editor and add ControllerGrabScript to each controller. There are some public fields we have to set. For the left controller, set Hand Type to Left Hand, Grab Action to \actions\default\in\Grab. Then, drag the Left Controller to Controller Pose. Repeat this for the Right Controller, but use the Right Hand.

34. Run the game and test it out with the HTC Vive headset and controllers. You should be able to pick up and throw the cubes and balls in the scene.

Now go to the next section to learn about UI in VR.

UI In VR

35. Now, let’s start to set some more UI for your VR game world.

36. The UI of VR is a little bit different from the common UI. The common UI is set on a 2D plane, then players can see them from there screen. However, in VR, we use our headset to view this world, so we need to set these UI objects in 3D world. Then, users can see these UI clearly.

37. To create UI for VR is similar to what we did in creating the reticle.

You need to create a canvas first in your scene. You’d better set its position with some small number and set it with some suitable width and height.

Remember to set the render mode as “World Space”!!!

38. Then you can set some UI component in this canvas. For example, you can set some text in it.

39. You can also add some UI like the reticle which can follow your camera.

Useful links

You are encouraged to learn many different VR techniques outside this tutorial, such as teleportation and other forms of movement. Here are some useful links:

• Introduction to VR in Unity Youtube playlist (The XR Toolkit introduced here may be useful for your final project, but please do not use it for the assignment below! The point is to get you to learn how to code your own interaction which may not be present in an existing framework.)
• Vive Input Utility for Unity (This can simulate the VR headset and controllers.)
• How to Teleport in VR - Oculus Unity Tutorial
• Oculus Integration Package Components
• Advanced VR Movement in Unity Tutorial
• Advanced VR Interactions in Unity Tutorial
• HTC Vive Tutorial for Unity

Assignment 2 - Shooting and Porting

In this assignment, you need to build a shooting game. You can use any assets from the Asset Store, or build your own assets, if you have the know-how.

The game should fulfill the following requirements:

• 1. In your game, one touch controller should take the form of a shooting device (this need not be a gun) in game, while the other touch controller should take the form of a hand. (This is just a guideline to make it easier to refer to each controller. If you can think of some other objects that better fit your theme, use them!)

• 2. With your shooting device, you should be able to pick up projectiles like this (once again, this is just a guideline):

  

• 3. You should be able to fire these projectiles from your shooting device, following some laws of physics (Use Rigidbody and AddForce)
• 4. There should be some targets in your scene, which are able to interact with the bullet. These targets can be static or moving.
• 5. As your ball hits the targets, there should be some response form the targets, such as disappearance or fragmentation.
• 6. Also, as the targets are hit, there should be a score recorder to record the score. Implement a UI object to show the score.
• 7. There should be movement in VR. As you should have learnt in your lectures, smooth locomotion is not ideal because it easily causes nausea. Thus, you should include some kind of teleportation. You can have free teleportation, or have some portals for the player to teleport to. Your other controller (the "hand") should be used for this teleportation. The common way of doing is holding a trigger to let a ray of light appear, with an indicator at the other end (remember raycasts?), then releasing the trigger to teleport to the location of the indicator. Something like this:



• 8. Your grabbing and teleporting script must be coded by you. i.e. No using scripts that may come with assets in the Assets Store.
• 9. Add some audio into this game to improve the sense of reality and immersion. You should at least have sound clips for picking up the ball, shooting, hitting target (target response).
• 10. Also, add immersion by making the shooting and teleporting follow some theme! Basically, think about why the player can do all this! Adding a level background that fits this theme helps a lot.

Grading Criteria

Grabbing and Shooting (30%)	Player can grab projectiles and shoot them. Projectiles follow the laws of physics.
Teleporting (20%)	Player can teleport around the map by pointing and clicking one button.
Targets (20%)	There are targets spread around the level, and they provide some response when hit.
UI (10%)	Score recorder UI works perfectly and is placed appropriately. It is not difficult for the player to view the score at any time.
Audio (10%)	Audio enhances the experience and immersion.
Creativity (10%)	Unique and engaging theme that is well thought-out and refreshing.

Should you need any further clarification, feel free to contact any TA.

Submission Guidelines

After completing your project, you need to select all your objects in Project panel and click Assets > Export Package to export what you used in your project.

This is a group project. Groups members will be same as your final project group. You only need to submit one file.

What to submit:

XXX. unitypackage (Your game package)

InputManager.asset (Found in ProjectSettings folder)

README.txt (You need to write how to play your game and what downloaded packages or objects you have used in your project)

Zip these files into Your_Group_Number.zip and upload it to Canvas->Assignments->Assignment 2 Submission. Due date: Refer to Canvas->Syllabus->Schedule .