HoloLens Tutorial – Cursor and Voice

Last Updated: Apr 30, 2017

Tutorial Accomplishments

  1. Add a cursor to the scene
  2. Add A Voice Command to the scene

 

This section of the tutorial will add user control to our application. We will add a cursor which can be used in conjunction with the input manager we saw in a previous tutorial to allow the user to click on objects. We will also add a voice command to the scene that will allow us to turn the visualization of the spatial map on and off as desired.

 

The Cursor

Open the project in Unity.  Open Assets -> HoloToolkit -> Input -> Prefabs -> Cursor.  Add the prefab Cursor under the Control folder.  Your project hierarchy should now look like this:

Build and Deploy the project. Notice how the cursor aligns to the spatial map as you are mapping the room. This is because each triangle of the spatial map has a collider attached when it is created. You will also notice that the cursor tracks the billboard, this is because of the attached Box Collider. Finish the spatial map, notice how the cursor goes through the holograms. This is because these objects don’t have colliders. Let’s fix these two issues. Edit ObjectCollectionManager and change the contents to this:


using System.Collections.Generic;
using HoloToolkit.Unity;
using UnityEngine;
public class ObjectCollectionManager : Singleton<ObjectCollectionManager>
{
[Tooltip("A collection of square building prefabs to generate in the world.")]
public List<GameObject> SquareBuildingPrefabs;
[Tooltip("The desired size of square buildings in the world.")]
public Vector3 SquareBuildingSize = new Vector3(.5f, .5f, .5f);
[Tooltip("A collection of Wide building prefabs to generate in the world.")]
public List<GameObject> WideBuildingPrefabs;
[Tooltip("The desired size of wide buildings in the world.")]
public Vector3 WideBuildingSize = new Vector3(1.0f, .5f, .5f);
[Tooltip("A collection of tall building prefabs to generate in the world.")]
public List<GameObject> TallBuildingPrefabs;
[Tooltip("The desired size of tall buildings in the world.")]
public Vector3 TallBuildingSize = new Vector3(.25f, .05f, .25f);
[Tooltip("A collection of tree prefabs to generate in the world.")]
public List<GameObject> TreePrefabs;
[Tooltip("The desired size of trees in the world.")]
public Vector3 TreeSize = new Vector3(.25f, .5f, .25f);
[Tooltip("Will be calculated at runtime if is not preset.")]
public float ScaleFactor;
public List<GameObject> ActiveHolograms = new List<GameObject>();
public void CreateSquareBuilding(int number, Vector3 positionCenter, Quaternion rotation)
{
CreateBuilding(SquareBuildingPrefabs[number], positionCenter, rotation, SquareBuildingSize);
}
public void CreateTallBuilding(int number, Vector3 positionCenter, Quaternion rotation)
{
CreateBuilding(TallBuildingPrefabs[number], positionCenter, rotation, TallBuildingSize);
}
public void CreateWideBuilding(int number, Vector3 positionCenter, Quaternion rotation)
{
CreateBuilding(WideBuildingPrefabs[number], positionCenter, rotation, WideBuildingSize);
}
private void CreateBuilding(GameObject buildingToCreate, Vector3 positionCenter, Quaternion rotation, Vector3 desiredSize)
{
// Stay center in the square but move down to the ground
var position = positionCenter – new Vector3(0, desiredSize.y * .5f, 0);
GameObject newObject = Instantiate(buildingToCreate, position, rotation);
if (newObject != null)
{
// Set the parent of the new object the GameObject it was placed on
newObject.transform.parent = gameObject.transform;
newObject.transform.localScale = RescaleToSameScaleFactor(buildingToCreate);
AddMeshColliderToAllChildren(newObject);
ActiveHolograms.Add(newObject);
}
}
public void CreateTree(int number, Vector3 positionCenter, Quaternion rotation)
{
// Stay center in the square but move down to the ground
var position = positionCenter – new Vector3(0, TreeSize.y * .5f, 0);
GameObject newObject = Instantiate(TreePrefabs[number], position, rotation);
if (newObject != null)
{
// Set the parent of the new object the GameObject it was placed on
newObject.transform.parent = gameObject.transform;
newObject.transform.localScale = RescaleToSameScaleFactor(TreePrefabs[number]);
newObject.AddComponent<MeshCollider>();
ActiveHolograms.Add(newObject);
}
}
private void AddMeshColliderToAllChildren(GameObject obj)
{
for (int i = 0; i < obj.transform.childCount; i++)
{
obj.transform.GetChild(i).gameObject.AddComponent<MeshCollider>();
}
}
private Vector3 RescaleToSameScaleFactor(GameObject objectToScale)
{
// ReSharper disable once CompareOfFloatsByEqualityOperator
if (ScaleFactor == 0f)
{
CalculateScaleFactor();
}
return objectToScale.transform.localScale * ScaleFactor;
}
private Vector3 StretchToFit(GameObject obj, Vector3 desiredSize)
{
var curBounds = GetBoundsForAllChildren(obj).size;
return new Vector3(desiredSize.x / curBounds.x / 2, desiredSize.y, desiredSize.z / curBounds.z / 2);
}
private void CalculateScaleFactor()
{
float maxScale = float.MaxValue;
var ratio = CalcScaleFactorHelper(WideBuildingPrefabs, WideBuildingSize);
if (ratio < maxScale)
{
maxScale = ratio;
}
ScaleFactor = maxScale;
}
private float CalcScaleFactorHelper(List<GameObject> objects, Vector3 desiredSize)
{
float maxScale = float.MaxValue;
foreach (var obj in objects)
{
var curBounds = GetBoundsForAllChildren(obj).size;
var difference = curBounds – desiredSize;
float ratio;
if (difference.x > difference.y && difference.x > difference.z)
{
ratio = desiredSize.x / curBounds.x;
}
else if (difference.y > difference.x && difference.y > difference.z)
{
ratio = desiredSize.y / curBounds.y;
}
else
{
ratio = desiredSize.z / curBounds.z;
}
if (ratio < maxScale)
{
maxScale = ratio;
}
}
return maxScale;
}
private Bounds GetBoundsForAllChildren(GameObject findMyBounds)
{
Bounds result = new Bounds(Vector3.zero, Vector3.zero);
foreach (var curRenderer in findMyBounds.GetComponentsInChildren<Renderer>())
{
if (result.extents == Vector3.zero)
{
result = curRenderer.bounds;
}
else
{
result.Encapsulate(curRenderer.bounds);
}
}
return result;
}
}

view raw

ObjectCollectionManager.cs

hosted with ❤ by GitHub

This is a very minor change adding a line into CreateTree that adds a collider to each object as it is being instantiated, and a new method that adds a collider to each gameobject and all of its child objects which we use to add colliders to the buildings.

Open Unity 3D.  Select the Spatial Status Billboard GameObject in the Hierarchy and disable the box collider.  The Components should look like this:

Build and Deploy your application.  After you finish deploying notice that the cursor tracks exactly to the real world objects and to the holograms and does not track the billboard.

The cursor is a fundamental component of any HoloLens application and I recommend that you add it at the beginning of your project.  I added it late in this tutorial to demonstrate that although the Cursor is a prefab that just works, you need to be aware of what objects you add colliders to and which ones you do not in order to get the desired behavior for your app.

 

Voice Commands

Open the project in Unity.  Edit the Player Settings and add the microphone permission:

 

Create a new Script called Speech.cs in your scripts folder.  Open it in Visual Studio and add the following contents:


using HoloToolkit.Unity;
using UnityEngine;
public class Speech : MonoBehaviour
{
public SpatialUnderstandingCustomMesh SpatialUnderstandingMesh;
public void ToggleMesh()
{
SpatialUnderstandingMesh.DrawProcessedMesh = !SpatialUnderstandingMesh.DrawProcessedMesh;
}
}

view raw

Speech.cs

hosted with ❤ by GitHub

This script provides one method that toggles the drawing of the spaial mapping mesh on and off.

Create a new Empty Gameobject called Speech under the GameObject Control.  Add the HoloToolkit Component Keyword Manager to the Speech GameObject. Add the Speech.cs script to the Speech GameObject.  Configure the properties to look like this:

The Keyword Manager is configured to start automatically, and will response to the keyword “Toggle”.  When Toggle is detected by the keyword manager, it calls the ToggleMesh method of the instance of the Speech script also attached to this gameobject.  The Speech Component gets a reference to the Spatial Understanding Custom Mesh so that it can manipulate the visibility of the Mesh.

Build and Deploy the application. You should be to speak the word “Toggle” at any time to show/hide the current spatial mesh.  Notice that even after you finalize the spatial mesh you can use “Toggle” to turn the mesh back on.

 

 

Tutorial Index

  Versions: Unity 2017.1.0p5 | MixedRealityToolkit-Unity v1.2017.1.0 | Visual Studio 2017 15.3.2
Unity 3D Project Creation How to create a HoloLens project in Unity 3D
Source Control Configure git for HoloLens / Unity work
Spatial Mapping How to spatial map a Room
Object Surface Observer Set up fake spatial map data for the Unity editor
TagAlongs and Billboarding   Tag along instructions to the user to force text on screen
Spatial Understanding Add spatial understanding to get play space detail
Finalizing Spatial Understanding Complete Spatial Understanding and Input Manager
Object Placement and Scaling Find valid locations for holograms in the play space
Hologram Management Manage the holograms you want to place in the world
Cursor and Voice Add a cursor and voice commands
Occlusion Add occlusion with the real world to your scene
Colliders and Rigidbodys Add Colliders and RigidBodys to your holograms
Cameron Vetter 4 Comments

4 thoughts on “HoloLens Tutorial – Cursor and Voice”

  • Cameron VetterPost author
    Posted on May 8, 2017 at 8:55 pm

    Update for this tutorial. You must add the Microphone permission for the speech recognition to work. The article has been updated to correct this mistake.

    Reply
    • Cameron VetterPost author
      Posted on October 25, 2017 at 10:14 am

      Yes that is correct, it has been obsolete for many months now. I will address that in the next update of the tutorial.

      Reply
  • mradamp
    Posted on October 25, 2017 at 2:39 pm

    One other issue that I have is that the mesh does not toggle. I only get a box with the ‘trying to map your surroundings’ and the contoured hill (the one that comes on when you first start up the HoloLens). It does toggle though. I have only tried this tutorial using the emulator until now. Could that be the reason?

    Reply

Leave a Reply