HoloLens Tutorial – Spatial Understanding

Last Updated: Apr 23, 2017

Tutorial Accomplishments

  1. Add Spatial Understanding
  2. Show Spatial Understanding Quality on Billboard

Add Spatial Understanding

This tutorial will start by adding Spatial Understanding on top of Spatial Mapping and change the visualization that the user sees to show Spatial Understanding instead of Spatial Mapping. Start by creating an empty gameobject under “Spatial” called “Spatial Understanding”. Add the component “Spatial Understanding” from the HoloToolkit. This will automatically add the “Spatial Understanding Source Mesh” and “Spatial Understanding Custom Mesh” components. Set the Mesh Material of the Custom Mesh to “Spatial Understanding Surface” which is included in the HoloToolkit. Your new GameObject should look like this:

Now select the Spatial Mapping node in the Hierarchy. Uncheck the Draw Visual Meshs check box so that the spatial mapping is no longer drawn.  This gameobject should now look this:

Spatial Understanding currently does not work in the Unity Editor. (This is supposed to be fixed, but as of the last update still did not work)  Build your application and deploy to either the device or the emulator.  You should now be able to map the room with the Spatial Understanding Mesh which is more detailed in appearance than the Spatial Mapping Mesh.  You should see something like this:

 

Show Spatial Understanding Quality on Billboard

The application needs to give the user feedback on how good their spatial map is so the user can decide when spatial mapping is complete. I recommend that the developer should understand what is required for your app to work and show the user what is needed to meet that threshold. For our tutorial we will simply output the raw quality information onto our billboard so that we can understand how the quality is changing in real time. (To give credit where credit is due, the format of the info displayed here and some of the code is borrowed from the spatial understanding example included with the HoloToolkit here)

In the project Assets folder create a new folder called Scripts. This is the location where all of our custom code will be stored. Because this project is fairly simple, we will place all of them at the root of this folder, if the project was more complicated I recommend creating a folder hierarchy to keep the custom project code organized. Create a new script named “SpatialUnderstandingState” in the Scripts folder. This script will contain the logic for displaying information on the billboard.

Double Click the SpatialUnderstandingState.cs file to open it in Visual Studio. Start by changing the class that is created to contain the following contents:


using UnityEngine;
using HoloToolkit.Unity;
using HoloToolkit.Unity.SpatialMapping;
public class SpatialUnderstandingState : Singleton<SpatialUnderstandingState>
{
public TextMesh DebugDisplay;
public TextMesh DebugSubDisplay;
private bool _triggered;
public string PrimaryText
{
get
{
return "PrimaryText";
}
}
public Color PrimaryColor
{
get
{
return Color.white;
}
}
public string DetailsText
{
get
{
return "DetailsText";
}
}
private void Update_DebugDisplay()
{
// Basic checks
if (DebugDisplay == null)
{
return;
}
// Update display text
DebugDisplay.text = PrimaryText;
DebugDisplay.color = PrimaryColor;
DebugSubDisplay.text = DetailsText;
}
// Update is called once per frame
private void Update()
{
// Updates
Update_DebugDisplay();
}
}

This creates the basic skeleton of the script that will update the billboard message with each frame.  The update method is called once per frame, it calls the Update_DebugDisplay method. This methods updates the text and color based on calls to properties that return hard coded values for now.

For this script to execute we need to add it to a GameObject.  In Unity select the Spatial Status Billboard in the Hierarchy.   In the Inspector click Add Component.  Select the Spatial Understanding State component that we just created.  Set the Debug Display property to the TextDisplay TextMesh Component.  Set the Debug Sub Display property to the TextSubDisplay TextMesh Component.  Your new component should now look like this:

Press Play on your project, you will see that the Hello World from the Billboard has changed to “PrimaryText” and “DetailsText” by the new script.

Next we will make that text show the spatial understanding status. Open SpatialUnderstandingState.cs for editing.  Update the class with the following code:


using System;
using UnityEngine;
using HoloToolkit.Unity;
using HoloToolkit.Unity.SpatialMapping;
public class SpatialUnderstandingState : Singleton<SpatialUnderstandingState>
{
public TextMesh DebugDisplay;
public TextMesh DebugSubDisplay;
private bool _triggered;
public bool HideText = false;
private string _spaceQueryDescription;
public string SpaceQueryDescription
{
get
{
return _spaceQueryDescription;
}
set
{
_spaceQueryDescription = value;
}
}
public string PrimaryText
{
get
{
if (HideText)
return string.Empty;
// Display the space and object query results (has priority)
if (!string.IsNullOrEmpty(SpaceQueryDescription))
{
return SpaceQueryDescription;
}
// Scan state
if (SpatialUnderstanding.Instance.AllowSpatialUnderstanding)
{
switch (SpatialUnderstanding.Instance.ScanState)
{
case SpatialUnderstanding.ScanStates.Scanning:
// Get the scan stats
IntPtr statsPtr = SpatialUnderstanding.Instance.UnderstandingDLL.GetStaticPlayspaceStatsPtr();
if (SpatialUnderstandingDll.Imports.QueryPlayspaceStats(statsPtr) == 0)
{
return "playspace stats query failed";
}
return "Walk around and scan in your playspace";
case SpatialUnderstanding.ScanStates.Finishing:
return "Finalizing scan (please wait)";
case SpatialUnderstanding.ScanStates.Done:
return "Scan complete";
default:
return "ScanState = " + SpatialUnderstanding.Instance.ScanState;
}
}
return string.Empty;
}
}
public Color PrimaryColor
{
get
{
return Color.white;
}
}
public string DetailsText
{
get
{
return "DetailsText";
}
}
private void Update_DebugDisplay()
{
// Basic checks
if (DebugDisplay == null)
{
return;
}
// Update display text
DebugDisplay.text = PrimaryText;
DebugDisplay.color = PrimaryColor;
DebugSubDisplay.text = DetailsText;
}
// Update is called once per frame
private void Update()
{
// Updates
Update_DebugDisplay();
}
}

The Property PrimaryText has now been given some business logic.  This property tells the user the state of the current scan and also adds the SpaceQueryDescription, which when set, is shown to the user instead of the scan state, we will use that later.  Press Play and you will now see the current state of the spatial mapping process is displayed.  Next lets set the DetailsText property. Open SpatialUnderstandingState.cs for editing.  Update the class with the following code:


using System;
using UnityEngine;
using HoloToolkit.Unity;
using HoloToolkit.Unity.SpatialMapping;
public class SpatialUnderstandingState : Singleton<SpatialUnderstandingState>
{
public float MinAreaForStats = 5.0f;
public TextMesh DebugDisplay;
public TextMesh DebugSubDisplay;
private bool _triggered;
public bool HideText = false;
private string _spaceQueryDescription;
public string SpaceQueryDescription
{
get
{
return _spaceQueryDescription;
}
set
{
_spaceQueryDescription = value;
}
}
public string PrimaryText
{
get
{
if (HideText)
return string.Empty;
// Display the space and object query results (has priority)
if (!string.IsNullOrEmpty(SpaceQueryDescription))
{
return SpaceQueryDescription;
}
// Scan state
if (SpatialUnderstanding.Instance.AllowSpatialUnderstanding)
{
switch (SpatialUnderstanding.Instance.ScanState)
{
case SpatialUnderstanding.ScanStates.Scanning:
// Get the scan stats
IntPtr statsPtr = SpatialUnderstanding.Instance.UnderstandingDLL.GetStaticPlayspaceStatsPtr();
if (SpatialUnderstandingDll.Imports.QueryPlayspaceStats(statsPtr) == 0)
{
return "playspace stats query failed";
}
return "Walk around and scan in your playspace";
case SpatialUnderstanding.ScanStates.Finishing:
return "Finalizing scan (please wait)";
case SpatialUnderstanding.ScanStates.Done:
return "Scan complete";
default:
return "ScanState = " + SpatialUnderstanding.Instance.ScanState;
}
}
return string.Empty;
}
}
public Color PrimaryColor
{
get
{
return Color.white;
}
}
public string DetailsText
{
get
{
if (SpatialUnderstanding.Instance.ScanState == SpatialUnderstanding.ScanStates.None)
{
return "";
}
// Scanning stats get second priority
if ((SpatialUnderstanding.Instance.ScanState == SpatialUnderstanding.ScanStates.Scanning) &&
(SpatialUnderstanding.Instance.AllowSpatialUnderstanding))
{
IntPtr statsPtr = SpatialUnderstanding.Instance.UnderstandingDLL.GetStaticPlayspaceStatsPtr();
if (SpatialUnderstandingDll.Imports.QueryPlayspaceStats(statsPtr) == 0)
{
return "Playspace stats query failed";
}
SpatialUnderstandingDll.Imports.PlayspaceStats stats = SpatialUnderstanding.Instance.UnderstandingDLL.GetStaticPlayspaceStats();
// Start showing the stats when they are no longer zero
if (stats.TotalSurfaceArea > MinAreaForStats)
{
SpatialMappingManager.Instance.DrawVisualMeshes = false;
string subDisplayText = string.Format("totalArea={0:0.0}, horiz={1:0.0}, wall={2:0.0}", stats.TotalSurfaceArea, stats.HorizSurfaceArea, stats.WallSurfaceArea);
subDisplayText += string.Format("\nnumFloorCells={0}, numCeilingCells={1}, numPlatformCells={2}", stats.NumFloor, stats.NumCeiling, stats.NumPlatform);
subDisplayText += string.Format("\npaintMode={0}, seenCells={1}, notSeen={2}", stats.CellCount_IsPaintMode, stats.CellCount_IsSeenQualtiy_Seen + stats.CellCount_IsSeenQualtiy_Good, stats.CellCount_IsSeenQualtiy_None);
return subDisplayText;
}
return "";
}
return "";
}
}
private void Update_DebugDisplay()
{
// Basic checks
if (DebugDisplay == null)
{
return;
}
// Update display text
DebugDisplay.text = PrimaryText;
DebugDisplay.color = PrimaryColor;
DebugSubDisplay.text = DetailsText;
}
// Update is called once per frame
private void Update()
{
// Updates
Update_DebugDisplay();
}
}

Build and Deploy to HoloLens or to an Emulator. Once the Spatial Understanding map is displayed, and we cross the minimum threshold,  stats are displays about the quality of the scan. This information will be used in the next section.

 

Tutorial Index

  Versions: Unity 2017.1.0p5 | MixedRealityToolkit-Unity v1.2017.1.0Visual Studio 2017 15.3.2
Unity 3D Project Creation How to create a HoloLens project in Unity 3D
Source Control Configure git for HoloLens / Unity work
Spatial Mapping How to spatial map a Room
Object Surface Observer Set up fake spatial map data for the Unity editor
TagAlongs and Billboarding   Tag along instructions to the user to force text on screen
Spatial Understanding Add spatial understanding to get play space detail
Finalizing Spatial Understanding Complete Spatial Understanding and Input Manager
Object Placement and Scaling Find valid locations for holograms in the play space
Hologram Management Manage the holograms you want to place in the world
Cursor and Voice Add a cursor and voice commands
Occlusion Add occlusion with the real world to your scene
Colliders and Rigidbodys Add Colliders and RigidBodys to your holograms

9 thoughts on “HoloLens Tutorial – Spatial Understanding

  • Posted on March 13, 2017 at 3:13 am

    At this point in the tutorial, when I attach the code to the 3D text, the spatial understanding mesh never appears, and accordingly, the text always shows that the scan is incomplete. When I deploy without the 3D text, the mesh shows up. Has anyone else ever had this problem? I’m currently trying to compare this code to the HoloToolkit spatial understanding code to find the differences and debug.

    Reply
    • Posted on March 27, 2017 at 7:23 pm

      I haven’t tried the tutorial with the latest version of the HoloToolkit, but it wouldn’t surprise me if something has changed, I will give it a try and update the tutorial as soon as I can.

      Reply
  • Posted on July 10, 2017 at 5:13 am

    Hi there,

    I seem to have the same problem, trying to implement spatial understanding. The scan remains incomplete and won’t progress through the app without finishing. Any updates from you guys? Thanks.

    Reply
  • Posted on July 10, 2017 at 11:51 am

    I have updated the article since the previous comment. If you would like to share your code base that isn’t working I can take a look at it to see if I can determine the problem, I’m guessing there is something in my tutorial that isn’t clear and is being missed…

    Reply
    • Posted on July 13, 2017 at 1:58 am

      Yesterday it worked somehow, so I probably missed something the first round. Thanks!

      Reply
  • Pingback: 10 Questions with Cameron Vetter | The Imaginative Universal

  • Posted on August 16, 2017 at 6:06 am

    I had the same problem and found that the placement of the camera was critical. After some tries I understood that I was only able to see the ceiling because I was beneath the floor. Changing the camera position so it was within the space of the Room defined in Spatial Mapping solved the problem.

    Reply
  • Posted on April 12, 2018 at 7:15 am

    Hello Cameron Vetter,

    I don’t know why, but i’m not able to see the spacial understanding inside the emulator. I use the latest version of the HoloToolkit and Unity 2017..0.0f1. Are you able to help me on this?

    Reply
    • Posted on April 27, 2018 at 4:59 pm

      It’s been my experience that it sometimes takes a very long time to get spatial understanding working inside the emulator. Sometimes I have to switch the simulated room to get it to work, i’d recommend trying that and waiting a few minutes to see if it starts working. The simulators ability to do spatial understanding is quirky at best, and I usually just work on the device for spatial understanding projects.

      Reply

Leave a Reply to Urban HedeåsCancel reply