Last Updated: Apr 23, 2017
Tutorial Accomplishments
- Add Spatial Understanding
- Show Spatial Understanding Quality on Billboard
Add Spatial Understanding
This tutorial will start by adding Spatial Understanding on top of Spatial Mapping and change the visualization that the user sees to show Spatial Understanding instead of Spatial Mapping. Start by creating an empty gameobject under “Spatial” called “Spatial Understanding”. Add the component “Spatial Understanding” from the HoloToolkit. This will automatically add the “Spatial Understanding Source Mesh” and “Spatial Understanding Custom Mesh” components. Set the Mesh Material of the Custom Mesh to “Spatial Understanding Surface” which is included in the HoloToolkit. Your new GameObject should look like this:
Now select the Spatial Mapping node in the Hierarchy. Uncheck the Draw Visual Meshs check box so that the spatial mapping is no longer drawn. This gameobject should now look this:
Spatial Understanding currently does not work in the Unity Editor. (This is supposed to be fixed, but as of the last update still did not work) Build your application and deploy to either the device or the emulator. You should now be able to map the room with the Spatial Understanding Mesh which is more detailed in appearance than the Spatial Mapping Mesh. You should see something like this:
Show Spatial Understanding Quality on Billboard
The application needs to give the user feedback on how good their spatial map is so the user can decide when spatial mapping is complete. I recommend that the developer should understand what is required for your app to work and show the user what is needed to meet that threshold. For our tutorial we will simply output the raw quality information onto our billboard so that we can understand how the quality is changing in real time. (To give credit where credit is due, the format of the info displayed here and some of the code is borrowed from the spatial understanding example included with the HoloToolkit here)
In the project Assets folder create a new folder called Scripts. This is the location where all of our custom code will be stored. Because this project is fairly simple, we will place all of them at the root of this folder, if the project was more complicated I recommend creating a folder hierarchy to keep the custom project code organized. Create a new script named “SpatialUnderstandingState” in the Scripts folder. This script will contain the logic for displaying information on the billboard.
Double Click the SpatialUnderstandingState.cs file to open it in Visual Studio. Start by changing the class that is created to contain the following contents:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
using UnityEngine; | |
using HoloToolkit.Unity; | |
using HoloToolkit.Unity.SpatialMapping; | |
public class SpatialUnderstandingState : Singleton<SpatialUnderstandingState> | |
{ | |
public TextMesh DebugDisplay; | |
public TextMesh DebugSubDisplay; | |
private bool _triggered; | |
public string PrimaryText | |
{ | |
get | |
{ | |
return "PrimaryText"; | |
} | |
} | |
public Color PrimaryColor | |
{ | |
get | |
{ | |
return Color.white; | |
} | |
} | |
public string DetailsText | |
{ | |
get | |
{ | |
return "DetailsText"; | |
} | |
} | |
private void Update_DebugDisplay() | |
{ | |
// Basic checks | |
if (DebugDisplay == null) | |
{ | |
return; | |
} | |
// Update display text | |
DebugDisplay.text = PrimaryText; | |
DebugDisplay.color = PrimaryColor; | |
DebugSubDisplay.text = DetailsText; | |
} | |
// Update is called once per frame | |
private void Update() | |
{ | |
// Updates | |
Update_DebugDisplay(); | |
} | |
} |
This creates the basic skeleton of the script that will update the billboard message with each frame. The update method is called once per frame, it calls the Update_DebugDisplay method. This methods updates the text and color based on calls to properties that return hard coded values for now.
For this script to execute we need to add it to a GameObject. In Unity select the Spatial Status Billboard in the Hierarchy. In the Inspector click Add Component. Select the Spatial Understanding State component that we just created. Set the Debug Display property to the TextDisplay TextMesh Component. Set the Debug Sub Display property to the TextSubDisplay TextMesh Component. Your new component should now look like this:
Press Play on your project, you will see that the Hello World from the Billboard has changed to “PrimaryText” and “DetailsText” by the new script.
Next we will make that text show the spatial understanding status. Open SpatialUnderstandingState.cs for editing. Update the class with the following code:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
using System; | |
using UnityEngine; | |
using HoloToolkit.Unity; | |
using HoloToolkit.Unity.SpatialMapping; | |
public class SpatialUnderstandingState : Singleton<SpatialUnderstandingState> | |
{ | |
public TextMesh DebugDisplay; | |
public TextMesh DebugSubDisplay; | |
private bool _triggered; | |
public bool HideText = false; | |
private string _spaceQueryDescription; | |
public string SpaceQueryDescription | |
{ | |
get | |
{ | |
return _spaceQueryDescription; | |
} | |
set | |
{ | |
_spaceQueryDescription = value; | |
} | |
} | |
public string PrimaryText | |
{ | |
get | |
{ | |
if (HideText) | |
return string.Empty; | |
// Display the space and object query results (has priority) | |
if (!string.IsNullOrEmpty(SpaceQueryDescription)) | |
{ | |
return SpaceQueryDescription; | |
} | |
// Scan state | |
if (SpatialUnderstanding.Instance.AllowSpatialUnderstanding) | |
{ | |
switch (SpatialUnderstanding.Instance.ScanState) | |
{ | |
case SpatialUnderstanding.ScanStates.Scanning: | |
// Get the scan stats | |
IntPtr statsPtr = SpatialUnderstanding.Instance.UnderstandingDLL.GetStaticPlayspaceStatsPtr(); | |
if (SpatialUnderstandingDll.Imports.QueryPlayspaceStats(statsPtr) == 0) | |
{ | |
return "playspace stats query failed"; | |
} | |
return "Walk around and scan in your playspace"; | |
case SpatialUnderstanding.ScanStates.Finishing: | |
return "Finalizing scan (please wait)"; | |
case SpatialUnderstanding.ScanStates.Done: | |
return "Scan complete"; | |
default: | |
return "ScanState = " + SpatialUnderstanding.Instance.ScanState; | |
} | |
} | |
return string.Empty; | |
} | |
} | |
public Color PrimaryColor | |
{ | |
get | |
{ | |
return Color.white; | |
} | |
} | |
public string DetailsText | |
{ | |
get | |
{ | |
return "DetailsText"; | |
} | |
} | |
private void Update_DebugDisplay() | |
{ | |
// Basic checks | |
if (DebugDisplay == null) | |
{ | |
return; | |
} | |
// Update display text | |
DebugDisplay.text = PrimaryText; | |
DebugDisplay.color = PrimaryColor; | |
DebugSubDisplay.text = DetailsText; | |
} | |
// Update is called once per frame | |
private void Update() | |
{ | |
// Updates | |
Update_DebugDisplay(); | |
} | |
} |
The Property PrimaryText has now been given some business logic. This property tells the user the state of the current scan and also adds the SpaceQueryDescription, which when set, is shown to the user instead of the scan state, we will use that later. Press Play and you will now see the current state of the spatial mapping process is displayed. Next lets set the DetailsText property. Open SpatialUnderstandingState.cs for editing. Update the class with the following code:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
using System; | |
using UnityEngine; | |
using HoloToolkit.Unity; | |
using HoloToolkit.Unity.SpatialMapping; | |
public class SpatialUnderstandingState : Singleton<SpatialUnderstandingState> | |
{ | |
public float MinAreaForStats = 5.0f; | |
public TextMesh DebugDisplay; | |
public TextMesh DebugSubDisplay; | |
private bool _triggered; | |
public bool HideText = false; | |
private string _spaceQueryDescription; | |
public string SpaceQueryDescription | |
{ | |
get | |
{ | |
return _spaceQueryDescription; | |
} | |
set | |
{ | |
_spaceQueryDescription = value; | |
} | |
} | |
public string PrimaryText | |
{ | |
get | |
{ | |
if (HideText) | |
return string.Empty; | |
// Display the space and object query results (has priority) | |
if (!string.IsNullOrEmpty(SpaceQueryDescription)) | |
{ | |
return SpaceQueryDescription; | |
} | |
// Scan state | |
if (SpatialUnderstanding.Instance.AllowSpatialUnderstanding) | |
{ | |
switch (SpatialUnderstanding.Instance.ScanState) | |
{ | |
case SpatialUnderstanding.ScanStates.Scanning: | |
// Get the scan stats | |
IntPtr statsPtr = SpatialUnderstanding.Instance.UnderstandingDLL.GetStaticPlayspaceStatsPtr(); | |
if (SpatialUnderstandingDll.Imports.QueryPlayspaceStats(statsPtr) == 0) | |
{ | |
return "playspace stats query failed"; | |
} | |
return "Walk around and scan in your playspace"; | |
case SpatialUnderstanding.ScanStates.Finishing: | |
return "Finalizing scan (please wait)"; | |
case SpatialUnderstanding.ScanStates.Done: | |
return "Scan complete"; | |
default: | |
return "ScanState = " + SpatialUnderstanding.Instance.ScanState; | |
} | |
} | |
return string.Empty; | |
} | |
} | |
public Color PrimaryColor | |
{ | |
get | |
{ | |
return Color.white; | |
} | |
} | |
public string DetailsText | |
{ | |
get | |
{ | |
if (SpatialUnderstanding.Instance.ScanState == SpatialUnderstanding.ScanStates.None) | |
{ | |
return ""; | |
} | |
// Scanning stats get second priority | |
if ((SpatialUnderstanding.Instance.ScanState == SpatialUnderstanding.ScanStates.Scanning) && | |
(SpatialUnderstanding.Instance.AllowSpatialUnderstanding)) | |
{ | |
IntPtr statsPtr = SpatialUnderstanding.Instance.UnderstandingDLL.GetStaticPlayspaceStatsPtr(); | |
if (SpatialUnderstandingDll.Imports.QueryPlayspaceStats(statsPtr) == 0) | |
{ | |
return "Playspace stats query failed"; | |
} | |
SpatialUnderstandingDll.Imports.PlayspaceStats stats = SpatialUnderstanding.Instance.UnderstandingDLL.GetStaticPlayspaceStats(); | |
// Start showing the stats when they are no longer zero | |
if (stats.TotalSurfaceArea > MinAreaForStats) | |
{ | |
SpatialMappingManager.Instance.DrawVisualMeshes = false; | |
string subDisplayText = string.Format("totalArea={0:0.0}, horiz={1:0.0}, wall={2:0.0}", stats.TotalSurfaceArea, stats.HorizSurfaceArea, stats.WallSurfaceArea); | |
subDisplayText += string.Format("\nnumFloorCells={0}, numCeilingCells={1}, numPlatformCells={2}", stats.NumFloor, stats.NumCeiling, stats.NumPlatform); | |
subDisplayText += string.Format("\npaintMode={0}, seenCells={1}, notSeen={2}", stats.CellCount_IsPaintMode, stats.CellCount_IsSeenQualtiy_Seen + stats.CellCount_IsSeenQualtiy_Good, stats.CellCount_IsSeenQualtiy_None); | |
return subDisplayText; | |
} | |
return ""; | |
} | |
return ""; | |
} | |
} | |
private void Update_DebugDisplay() | |
{ | |
// Basic checks | |
if (DebugDisplay == null) | |
{ | |
return; | |
} | |
// Update display text | |
DebugDisplay.text = PrimaryText; | |
DebugDisplay.color = PrimaryColor; | |
DebugSubDisplay.text = DetailsText; | |
} | |
// Update is called once per frame | |
private void Update() | |
{ | |
// Updates | |
Update_DebugDisplay(); | |
} | |
} |
Build and Deploy to HoloLens or to an Emulator. Once the Spatial Understanding map is displayed, and we cross the minimum threshold, stats are displays about the quality of the scan. This information will be used in the next section.
Tutorial Index
Versions: Unity 2017.1.0p5 | MixedRealityToolkit-Unity v1.2017.1.0 | Visual Studio 2017 15.3.2Unity 3D Project Creation | How to create a HoloLens project in Unity 3D |
Source Control | Configure git for HoloLens / Unity work |
Spatial Mapping | How to spatial map a Room |
Object Surface Observer | Set up fake spatial map data for the Unity editor |
TagAlongs and Billboarding | Tag along instructions to the user to force text on screen |
Spatial Understanding | Add spatial understanding to get play space detail |
Finalizing Spatial Understanding | Complete Spatial Understanding and Input Manager |
Object Placement and Scaling | Find valid locations for holograms in the play space |
Hologram Management | Manage the holograms you want to place in the world |
Cursor and Voice | Add a cursor and voice commands |
Occlusion | Add occlusion with the real world to your scene |
Colliders and Rigidbodys | Add Colliders and RigidBodys to your holograms |
- Download the completed app Western Town in the Windows Store!
- Completed Source code from the entire tutorial available on GitHub.
At this point in the tutorial, when I attach the code to the 3D text, the spatial understanding mesh never appears, and accordingly, the text always shows that the scan is incomplete. When I deploy without the 3D text, the mesh shows up. Has anyone else ever had this problem? I’m currently trying to compare this code to the HoloToolkit spatial understanding code to find the differences and debug.
I haven’t tried the tutorial with the latest version of the HoloToolkit, but it wouldn’t surprise me if something has changed, I will give it a try and update the tutorial as soon as I can.
Hi there,
I seem to have the same problem, trying to implement spatial understanding. The scan remains incomplete and won’t progress through the app without finishing. Any updates from you guys? Thanks.
I have updated the article since the previous comment. If you would like to share your code base that isn’t working I can take a look at it to see if I can determine the problem, I’m guessing there is something in my tutorial that isn’t clear and is being missed…
Yesterday it worked somehow, so I probably missed something the first round. Thanks!
Pingback: 10 Questions with Cameron Vetter | The Imaginative Universal
I had the same problem and found that the placement of the camera was critical. After some tries I understood that I was only able to see the ceiling because I was beneath the floor. Changing the camera position so it was within the space of the Room defined in Spatial Mapping solved the problem.
Hello Cameron Vetter,
I don’t know why, but i’m not able to see the spacial understanding inside the emulator. I use the latest version of the HoloToolkit and Unity 2017..0.0f1. Are you able to help me on this?
It’s been my experience that it sometimes takes a very long time to get spatial understanding working inside the emulator. Sometimes I have to switch the simulated room to get it to work, i’d recommend trying that and waiting a few minutes to see if it starts working. The simulators ability to do spatial understanding is quirky at best, and I usually just work on the device for spatial understanding projects.