Last Updated: Apr 23, 2017
Tutorial Accomplishments
- Use Spatial Understanding to find the location for a hologram
- Instantiate the hologram in the world
- Scale the hologram to the size of the available space
This section of the tutorial will focus on object placement and scaling of Holograms. For this section I am using a pack of 3D objects from the Unity Store, but you should be able to use any objects that have been optimized for Unity and aren’t too complex for the 3D processing power of the HoloLens.
Add 3D Objects to the Project
Start by adding a 3D Objects pack form the Unity Asset Store. I’m using the Pro Wild West Pack, there is also a free version called the Pro Western Starter Pack that should get the job done as well.
After purchasing, use the download manager in unity to download and import the pack. Unity will display an import screen that looks like this:
With everything selected click the Import button. A new folder is created in your Assets folder called “Pro Wild West Pack”. Browse through the folders and you will find all the parts that make up the 3D objects. For this project we will use the contents of the Prefabs folder.
Disable the Spatial map and the Billboard
Before we display a hologram, we need to disable the spatial map visualization and the billboard containing text. Although this is only tangentially related to placement we will do this as the beginning of the script for object placement. Create an empty GameObject under Holograms and name it “Placement”. In the project Pane, select Scripts. Create a new script called “ObjectPlacer.cs”. Add the component Object Placer to the Placement node in the Hierarchy. Your project should now look like this:
Add the following contents to ObjectPlacer.cs:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
using System.Collections.Generic; | |
using HoloToolkit.Unity; | |
using UnityEngine; | |
public class ObjectPlacer : MonoBehaviour | |
{ | |
public SpatialUnderstandingCustomMesh SpatialUnderstandingMesh; | |
private bool _timeToHideMesh; | |
// Use this for initialization | |
void Start() | |
{ | |
} | |
// Update is called once per frame | |
void Update() | |
{ | |
if (_timeToHideMesh) | |
{ | |
SpatialUnderstandingState.Instance.HideText = true; | |
HideGridEnableOcclulsion(); | |
_timeToHideMesh = false; | |
} | |
} | |
private void HideGridEnableOcclulsion() | |
{ | |
SpatialUnderstandingMesh.DrawProcessedMesh = false; | |
} | |
public void CreateScene() | |
{ | |
// Only if we're enabled | |
if (!SpatialUnderstanding.Instance.AllowSpatialUnderstanding) | |
{ | |
return; | |
} | |
_timeToHideMesh = true; | |
} | |
} |
Also modify the method Update in SpatialUnderstandingState.cs to contain the following code:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
public ObjectPlacer Placer; | |
private void Update() | |
{ | |
// Updates | |
Update_DebugDisplay(); | |
if (!_triggered && SpatialUnderstanding.Instance.ScanState == SpatialUnderstanding.ScanStates.Done) | |
{ | |
_triggered = true; | |
Placer.CreateScene(); | |
} | |
} |
In unity Select Spatial Status Billboard in the Hierarchy. Drag the Placement GameObject into the Spatial Understanding State Component into the Placer property.
These changes allow the application to call the CreateScene method of the ObjectPlacer.cs script on the specific instance that is attached to the Placement GameObject. Next select the Placement GameObject in the Hierarchy. Drag the Spatial Understanding GameObject onto the Object Placer component in the Spatial Understanding property. This will allow the Object Placer script to control the state of the Spatial Understanding by giving it a reference to the instance of the script.
Build your project and run it on either the emulator or the HoloLens. Now when you finalize the spatial understanding the text is disabled by setting SpatialUnderstandingState.Instance.HideText to true in ObjectPlacer.cs and the material showing the spatial understanding map is disabled by setting SpatialUnderstandingMesh.DrawProcessedMesh to false. This sets us up with a clean environment to display Holograms in as well as giving us an easy place to instantiate holograms that occurs immediately after spatial understanding is finalized.
Place the Hologram
Next we will use Spatial Understanding to find a location to place city_Saloon.prefab. This prefab is found here and looks like this:
Before we place the actual hologram lets draw the outline of a box where spatial understanding tells us to place our hologram. This is useful to have as an option for troubleshooting and debugging and will help demonstrate exactly what is happening. We will again borrow code from the Spatial Understanding example included with the HoloToolkit, with some small tweaks to decouple it from that example. I’m not going to go into detail explaining this code since I did not write it, but the basic idea is that every frame it draws lines representing the boxes that are passed to it. Create a new script caller “BoxDrawer.cs” and add these contents to it:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
using UnityEngine; | |
using System.Collections.Generic; | |
public class BoxDrawer | |
{ | |
// Consts | |
public const float DefaultLineWidth = 0.001f; | |
public const float DefaultBasisLength = 0.2f; | |
private readonly GameObject _callingObject; | |
public BoxDrawer(GameObject callingObject) | |
{ | |
_callingObject = callingObject; | |
} | |
// Structs | |
public class Line | |
{ | |
// Functions | |
public Line() | |
{ | |
} | |
public Line(Vector3 p0, Vector3 p1, Color c0, Color c1, float lineWidth = DefaultLineWidth) | |
{ | |
P0 = p0; | |
P1 = p1; | |
C0 = c0; | |
C1 = c1; | |
LineWidth = lineWidth; | |
IsValid = true; | |
} | |
public bool Set_IfDifferent(Vector3 p0, Vector3 p1, Color c0, Color c1, float lineWidth) | |
{ | |
IsValid = true; | |
if ((P0 != p0) || (P1 != p1) || (C0 != c0) || (C1 != c1) || (LineWidth != lineWidth)) | |
{ | |
P0 = p0; | |
P1 = p1; | |
C0 = c0; | |
C1 = c1; | |
LineWidth = lineWidth; | |
return true; | |
} | |
return false; | |
} | |
// Data | |
public Vector3 P0; | |
public Vector3 P1; | |
public Color C0; | |
public Color C1; | |
public float LineWidth; | |
public bool IsValid; | |
} | |
public class LineData | |
{ | |
public int LineIndex; | |
public List<Line> Lines = new List<Line>(); | |
public MeshRenderer Renderer; | |
public MeshFilter Filter; | |
} | |
public class AnimationCurve3 | |
{ | |
public void AddKey(float time, Vector3 pos) | |
{ | |
CurveX.AddKey(time, pos.x); | |
CurveY.AddKey(time, pos.y); | |
CurveZ.AddKey(time, pos.z); | |
} | |
public Vector3 Evaluate(float time) | |
{ | |
return new Vector3(CurveX.Evaluate(time), CurveY.Evaluate(time), CurveZ.Evaluate(time)); | |
} | |
public AnimationCurve CurveX = new AnimationCurve(); | |
public AnimationCurve CurveY = new AnimationCurve(); | |
public AnimationCurve CurveZ = new AnimationCurve(); | |
} | |
public class Box | |
{ | |
public const float InitialPositionForwardMaxDistance = 2.0f; | |
public const float AnimationTime = 2.5f; | |
public const float DelayPerItem = 0.35f; | |
public Box( | |
Vector3 center, | |
Quaternion rotation, | |
Color color, | |
Vector3 halfSize, | |
float lineWidth = DefaultLineWidth * 3.0f) | |
{ | |
Center = center; | |
Rotation = rotation; | |
Color = color; | |
HalfSize = halfSize; | |
LineWidth = lineWidth; | |
} | |
public Vector3 Center; | |
public Quaternion Rotation; | |
public Color Color; | |
public Vector3 HalfSize; | |
public float LineWidth; | |
} | |
// Config | |
public Material MaterialLine; | |
// Privates | |
private readonly LineData _lineData = new LineData(); | |
// Functions | |
protected bool DrawBox(Box box) | |
{ | |
// Animation is done, just pass through | |
return Draw_Box(box.Center, box.Rotation, box.Color, box.HalfSize, box.LineWidth); | |
} | |
protected bool Draw_Box(Vector3 center, Quaternion rotation, Color color, Vector3 halfSize, float lineWidth = DefaultLineWidth) | |
{ | |
bool needsUpdate = false; | |
Vector3 basisX = rotation * Vector3.right; | |
Vector3 basisY = rotation * Vector3.up; | |
Vector3 basisZ = rotation * Vector3.forward; | |
Vector3[] pts = | |
{ | |
center + basisX * halfSize.x + basisY * halfSize.y + basisZ * halfSize.z, | |
center + basisX * halfSize.x + basisY * halfSize.y – basisZ * halfSize.z, | |
center – basisX * halfSize.x + basisY * halfSize.y – basisZ * halfSize.z, | |
center – basisX * halfSize.x + basisY * halfSize.y + basisZ * halfSize.z, | |
center + basisX * halfSize.x – basisY * halfSize.y + basisZ * halfSize.z, | |
center + basisX * halfSize.x – basisY * halfSize.y – basisZ * halfSize.z, | |
center – basisX * halfSize.x – basisY * halfSize.y – basisZ * halfSize.z, | |
center – basisX * halfSize.x – basisY * halfSize.y + basisZ * halfSize.z | |
}; | |
// Bottom | |
needsUpdate |= Draw_Line(pts[0], pts[1], color, color, lineWidth); | |
needsUpdate |= Draw_Line(pts[1], pts[2], color, color, lineWidth); | |
needsUpdate |= Draw_Line(pts[2], pts[3], color, color, lineWidth); | |
needsUpdate |= Draw_Line(pts[3], pts[0], color, color, lineWidth); | |
// Top | |
needsUpdate |= Draw_Line(pts[4], pts[5], color, color, lineWidth); | |
needsUpdate |= Draw_Line(pts[5], pts[6], color, color, lineWidth); | |
needsUpdate |= Draw_Line(pts[6], pts[7], color, color, lineWidth); | |
needsUpdate |= Draw_Line(pts[7], pts[4], color, color, lineWidth); | |
// Vertical lines | |
needsUpdate |= Draw_Line(pts[0], pts[4], color, color, lineWidth); | |
needsUpdate |= Draw_Line(pts[1], pts[5], color, color, lineWidth); | |
needsUpdate |= Draw_Line(pts[2], pts[6], color, color, lineWidth); | |
needsUpdate |= Draw_Line(pts[3], pts[7], color, color, lineWidth); | |
return needsUpdate; | |
} | |
protected bool Draw_Line(Vector3 start, Vector3 end, Color colorStart, Color colorEnd, float lineWidth = DefaultLineWidth) | |
{ | |
// Create up a new line (unless it's already created) | |
while (_lineData.LineIndex >= _lineData.Lines.Count) | |
{ | |
_lineData.Lines.Add(new Line()); | |
} | |
// Set it | |
bool needsUpdate = _lineData.Lines[_lineData.LineIndex].Set_IfDifferent(_callingObject.transform.InverseTransformPoint(start), _callingObject.transform.InverseTransformPoint(end), colorStart, colorEnd, lineWidth); | |
// Inc out count | |
++_lineData.LineIndex; | |
return needsUpdate; | |
} | |
private void Lines_LineDataToMesh() | |
{ | |
// Alloc them up | |
Vector3[] verts = new Vector3[_lineData.Lines.Count * 8]; | |
int[] tris = new int[_lineData.Lines.Count * 12 * 3]; | |
Color[] colors = new Color[verts.Length]; | |
// Build the data | |
for (int i = 0; i < _lineData.Lines.Count; ++i) | |
{ | |
// Base index calcs | |
int vert = i * 8; | |
int v0 = vert; | |
int tri = i * 12 * 3; | |
// Setup | |
Vector3 dirUnit = (_lineData.Lines[i].P1 – _lineData.Lines[i].P0).normalized; | |
Vector3 normX = Vector3.Cross((Mathf.Abs(dirUnit.y) >= 0.99f) ? Vector3.right : Vector3.up, dirUnit).normalized; | |
Vector3 normy = Vector3.Cross(normX, dirUnit); | |
// Verts | |
verts[vert] = _lineData.Lines[i].P0 + normX * _lineData.Lines[i].LineWidth + normy * _lineData.Lines[i].LineWidth; colors[vert] = _lineData.Lines[i].C0; ++vert; | |
verts[vert] = _lineData.Lines[i].P0 – normX * _lineData.Lines[i].LineWidth + normy * _lineData.Lines[i].LineWidth; colors[vert] = _lineData.Lines[i].C0; ++vert; | |
verts[vert] = _lineData.Lines[i].P0 – normX * _lineData.Lines[i].LineWidth – normy * _lineData.Lines[i].LineWidth; colors[vert] = _lineData.Lines[i].C0; ++vert; | |
verts[vert] = _lineData.Lines[i].P0 + normX * _lineData.Lines[i].LineWidth – normy * _lineData.Lines[i].LineWidth; colors[vert] = _lineData.Lines[i].C0; ++vert; | |
verts[vert] = _lineData.Lines[i].P1 + normX * _lineData.Lines[i].LineWidth + normy * _lineData.Lines[i].LineWidth; colors[vert] = _lineData.Lines[i].C1; ++vert; | |
verts[vert] = _lineData.Lines[i].P1 – normX * _lineData.Lines[i].LineWidth + normy * _lineData.Lines[i].LineWidth; colors[vert] = _lineData.Lines[i].C1; ++vert; | |
verts[vert] = _lineData.Lines[i].P1 – normX * _lineData.Lines[i].LineWidth – normy * _lineData.Lines[i].LineWidth; colors[vert] = _lineData.Lines[i].C1; ++vert; | |
verts[vert] = _lineData.Lines[i].P1 + normX * _lineData.Lines[i].LineWidth – normy * _lineData.Lines[i].LineWidth; colors[vert] = _lineData.Lines[i].C1; ++vert; | |
// Indices | |
tris[tri + 0] = (v0 + 0); tris[tri + 1] = (v0 + 5); tris[tri + 2] = (v0 + 4); tri += 3; | |
tris[tri + 0] = (v0 + 1); tris[tri + 1] = (v0 + 5); tris[tri + 2] = (v0 + 0); tri += 3; | |
tris[tri + 0] = (v0 + 1); tris[tri + 1] = (v0 + 6); tris[tri + 2] = (v0 + 5); tri += 3; | |
tris[tri + 0] = (v0 + 2); tris[tri + 1] = (v0 + 6); tris[tri + 2] = (v0 + 1); tri += 3; | |
tris[tri + 0] = (v0 + 2); tris[tri + 1] = (v0 + 7); tris[tri + 2] = (v0 + 6); tri += 3; | |
tris[tri + 0] = (v0 + 3); tris[tri + 1] = (v0 + 7); tris[tri + 2] = (v0 + 2); tri += 3; | |
tris[tri + 0] = (v0 + 3); tris[tri + 1] = (v0 + 7); tris[tri + 2] = (v0 + 4); tri += 3; | |
tris[tri + 0] = (v0 + 3); tris[tri + 1] = (v0 + 4); tris[tri + 2] = (v0 + 0); tri += 3; | |
tris[tri + 0] = (v0 + 0); tris[tri + 1] = (v0 + 3); tris[tri + 2] = (v0 + 2); tri += 3; | |
tris[tri + 0] = (v0 + 0); tris[tri + 1] = (v0 + 2); tris[tri + 2] = (v0 + 1); tri += 3; | |
tris[tri + 0] = (v0 + 5); tris[tri + 1] = (v0 + 6); tris[tri + 2] = (v0 + 7); tri += 3; | |
tris[tri + 0] = (v0 + 5); tris[tri + 1] = (v0 + 7); tris[tri + 2] = (v0 + 4); tri += 3; | |
} | |
// Create up the components | |
if (_lineData.Filter == null) | |
{ | |
_lineData.Filter = _callingObject.AddComponent<MeshFilter>(); | |
} | |
if (_lineData.Renderer == null) | |
{ | |
_lineData.Renderer = _callingObject.AddComponent<MeshRenderer>(); | |
_lineData.Renderer.material = MaterialLine; | |
} | |
// Create or clear the mesh | |
Mesh mesh; | |
if (_lineData.Filter.mesh != null) | |
{ | |
mesh = _lineData.Filter.mesh; | |
mesh.Clear(); | |
} | |
else | |
{ | |
mesh = new Mesh(); | |
mesh.name = "LineDrawer.Lines_LineDataToMesh"; | |
} | |
// Set them into the mesh | |
mesh.vertices = verts; | |
mesh.triangles = tris; | |
mesh.colors = colors; | |
mesh.RecalculateBounds(); | |
mesh.RecalculateNormals(); | |
_lineData.Filter.mesh = mesh; | |
// If no tris, hide it | |
_lineData.Renderer.enabled = (_lineData.Lines.Count != 0); | |
// Line index reset | |
_lineData.LineIndex = 0; | |
} | |
public void UpdateBoxes(List<Box> boxes) | |
{ | |
// Lines: Begin | |
LineDraw_Begin(); | |
// Drawers | |
bool needsUpdate = false; | |
needsUpdate |= Draw_LineBoxList(boxes); | |
// Lines: Finish up | |
LineDraw_End(needsUpdate); | |
} | |
private bool Draw_LineBoxList(List<Box> boxes) | |
{ | |
bool needsUpdate = false; | |
foreach (var box in boxes) | |
{ | |
needsUpdate |= DrawBox(box); | |
} | |
return needsUpdate; | |
} | |
protected void LineDraw_Begin() | |
{ | |
_lineData.LineIndex = 0; | |
for (int i = 0; i < _lineData.Lines.Count; ++i) | |
{ | |
_lineData.Lines[i].IsValid = false; | |
} | |
} | |
protected void LineDraw_End(bool needsUpdate) | |
{ | |
if (_lineData == null) | |
{ | |
return; | |
} | |
// Check if we have any not dirty | |
int i = 0; | |
while (i < _lineData.Lines.Count) | |
{ | |
if (!_lineData.Lines[i].IsValid) | |
{ | |
needsUpdate = true; | |
_lineData.Lines.RemoveAt(i); | |
continue; | |
} | |
++i; | |
} | |
// Do the update (if needed) | |
if (needsUpdate) | |
{ | |
Lines_LineDataToMesh(); | |
} | |
} | |
} |
In your Scripts folder also create a new script called “PlacementQuery.cs”. Add the following code to it:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
using System.Collections.Generic; | |
using HoloToolkit.Unity; | |
using UnityEngine; | |
public enum ObjectType | |
{ | |
SquareBuilding, | |
WideBuilding, | |
TallBuilding, | |
Tree, | |
Tumbleweed, | |
Mine | |
} | |
public struct PlacementQuery | |
{ | |
public PlacementQuery( | |
SpatialUnderstandingDllObjectPlacement.ObjectPlacementDefinition placementDefinition, | |
Vector3 dimensions, | |
ObjectType objType, | |
List<SpatialUnderstandingDllObjectPlacement.ObjectPlacementRule> placementRules = null, | |
List<SpatialUnderstandingDllObjectPlacement.ObjectPlacementConstraint> placementConstraints = null) | |
{ | |
PlacementDefinition = placementDefinition; | |
PlacementRules = placementRules; | |
PlacementConstraints = placementConstraints; | |
Dimensions = dimensions; | |
ObjType = objType; | |
} | |
public readonly SpatialUnderstandingDllObjectPlacement.ObjectPlacementDefinition PlacementDefinition; | |
public readonly Vector3 Dimensions; | |
public readonly ObjectType ObjType; | |
public readonly List<SpatialUnderstandingDllObjectPlacement.ObjectPlacementRule> PlacementRules; | |
public readonly List<SpatialUnderstandingDllObjectPlacement.ObjectPlacementConstraint> PlacementConstraints; | |
} |
This script creates a data object that is used to keep track of a spatial understanding request. Why not just make the request right away and skip storing the request in a data structure? Right now that would work fine, but eventually we want to create many holograms, and the most efficient way to do this is to feed all the requests to spatial understanding at once in a separate thread so that the UI thread doesn’t lock up for the user which would make the HoloLens appear to lock up. For the same reason, we also will need to create the script “PlacementResult.cs”:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
using HoloToolkit.Unity; | |
using UnityEngine; | |
public class PlacementResult | |
{ | |
public PlacementResult(SpatialUnderstandingDllObjectPlacement.ObjectPlacementResult result, Vector3 dimensions, ObjectType objType) | |
{ | |
_result = result; | |
Dimensions = dimensions; | |
ObjType = objType; | |
} | |
public Vector3 Position { get { return _result.Position; } } | |
public Vector3 Normal { get { return _result.Forward; } } | |
public Vector3 Dimensions { get; private set; } | |
public ObjectType ObjType { get; private set; } | |
private readonly SpatialUnderstandingDllObjectPlacement.ObjectPlacementResult _result; | |
} |
Next make these changes to ObjectPlacer.cs:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
using System; | |
using System.Collections.Generic; | |
using HoloToolkit.Unity; | |
using UnityEngine; | |
public class ObjectPlacer : MonoBehaviour | |
{ | |
public bool DrawDebugBoxes = true; | |
public SpatialUnderstandingCustomMesh SpatialUnderstandingMesh; | |
[Tooltip("The desired size of wide buildings in the world.")] | |
public Vector3 WideBuildingSize = new Vector3(1.0f, .5f, .5f); | |
public GameObject SaloonBuildingPrefab; | |
private readonly List<BoxDrawer.Box> _lineBoxList = new List<BoxDrawer.Box>(); | |
private readonly Queue<PlacementResult> _results = new Queue<PlacementResult>(); | |
private bool _timeToHideMesh; | |
private BoxDrawer _boxDrawing; | |
// Use this for initialization | |
void Start() | |
{ | |
if (DrawDebugBoxes) | |
{ | |
_boxDrawing = new BoxDrawer(gameObject); | |
} | |
} | |
// Update is called once per frame | |
void Update() | |
{ | |
ProcessPlacementResults(); | |
if (_timeToHideMesh) | |
{ | |
SpatialUnderstandingState.Instance.HideText = true; | |
HideGridEnableOcclulsion(); | |
_timeToHideMesh = false; | |
} | |
if (DrawDebugBoxes) | |
{ | |
_boxDrawing.UpdateBoxes(_lineBoxList); | |
} | |
} | |
private void HideGridEnableOcclulsion() | |
{ | |
SpatialUnderstandingMesh.DrawProcessedMesh = false; | |
} | |
public void CreateScene() | |
{ | |
// Only if we're enabled | |
if (!SpatialUnderstanding.Instance.AllowSpatialUnderstanding) | |
{ | |
return; | |
} | |
SpatialUnderstandingDllObjectPlacement.Solver_Init(); | |
SpatialUnderstandingState.Instance.SpaceQueryDescription = "Generating World"; | |
List<PlacementQuery> queries = CreateLocationQueriesForSolver(1, WideBuildingSize, ObjectType.WideBuilding); | |
GetLocationsFromSolver(queries); | |
} | |
private void ProcessPlacementResults() | |
{ | |
if (_results.Count > 0) | |
{ | |
var toPlace = _results.Dequeue(); | |
// Output | |
if (DrawDebugBoxes) | |
{ | |
DrawBox(toPlace, Color.red); | |
} | |
} | |
} | |
private void DrawBox(PlacementResult boxLocation, Color color) | |
{ | |
if (boxLocation != null) | |
{ | |
_lineBoxList.Add( | |
new BoxDrawer.Box( | |
boxLocation.Position, | |
Quaternion.LookRotation(boxLocation.Normal, Vector3.up), | |
color, | |
boxLocation.Dimensions * 0.5f) | |
); | |
} | |
} | |
private void GetLocationsFromSolver(List<PlacementQuery> placementQueries) | |
{ | |
#if UNITY_WSA && !UNITY_EDITOR | |
System.Threading.Tasks.Task.Run(() => | |
{ | |
// Go through the queries in the list | |
for (int i = 0; i < placementQueries.Count; ++i) | |
{ | |
var result = PlaceObject(placementQueries[i].ObjType.ToString() + i, | |
placementQueries[i].PlacementDefinition, | |
placementQueries[i].Dimensions, | |
placementQueries[i].ObjType, | |
placementQueries[i].PlacementRules, | |
placementQueries[i].PlacementConstraints); | |
if (result != null) | |
{ | |
_results.Enqueue(result); | |
} | |
} | |
_timeToHideMesh = true; | |
}); | |
#else | |
_timeToHideMesh = true; | |
#endif | |
} | |
private PlacementResult PlaceObject(string placementName, | |
SpatialUnderstandingDllObjectPlacement.ObjectPlacementDefinition placementDefinition, | |
Vector3 boxFullDims, | |
ObjectType objType, | |
List<SpatialUnderstandingDllObjectPlacement.ObjectPlacementRule> placementRules = null, | |
List<SpatialUnderstandingDllObjectPlacement.ObjectPlacementConstraint> placementConstraints = null) | |
{ | |
// New query | |
if (SpatialUnderstandingDllObjectPlacement.Solver_PlaceObject( | |
placementName, | |
SpatialUnderstanding.Instance.UnderstandingDLL.PinObject(placementDefinition), | |
(placementRules != null) ? placementRules.Count : 0, | |
((placementRules != null) && (placementRules.Count > 0)) ? SpatialUnderstanding.Instance.UnderstandingDLL.PinObject(placementRules.ToArray()) : IntPtr.Zero, | |
(placementConstraints != null) ? placementConstraints.Count : 0, | |
((placementConstraints != null) && (placementConstraints.Count > 0)) ? SpatialUnderstanding.Instance.UnderstandingDLL.PinObject(placementConstraints.ToArray()) : IntPtr.Zero, | |
SpatialUnderstanding.Instance.UnderstandingDLL.GetStaticObjectPlacementResultPtr()) > 0) | |
{ | |
SpatialUnderstandingDllObjectPlacement.ObjectPlacementResult placementResult = SpatialUnderstanding.Instance.UnderstandingDLL.GetStaticObjectPlacementResult(); | |
return new PlacementResult(placementResult.Clone() as SpatialUnderstandingDllObjectPlacement.ObjectPlacementResult, boxFullDims, objType); | |
} | |
return null; | |
} | |
private List<PlacementQuery> CreateLocationQueriesForSolver(int desiredLocationCount, Vector3 boxFullDims, ObjectType objType) | |
{ | |
List<PlacementQuery> placementQueries = new List<PlacementQuery>(); | |
var halfBoxDims = boxFullDims * .5f; | |
var disctanceFromOtherObjects = halfBoxDims.x > halfBoxDims.z ? halfBoxDims.x * 3f : halfBoxDims.z * 3f; | |
for (int i = 0; i < desiredLocationCount; ++i) | |
{ | |
var placementRules = new List<SpatialUnderstandingDllObjectPlacement.ObjectPlacementRule> | |
{ | |
SpatialUnderstandingDllObjectPlacement.ObjectPlacementRule.Create_AwayFromOtherObjects(disctanceFromOtherObjects) | |
}; | |
var placementConstraints = new List<SpatialUnderstandingDllObjectPlacement.ObjectPlacementConstraint>(); | |
SpatialUnderstandingDllObjectPlacement.ObjectPlacementDefinition placementDefinition = SpatialUnderstandingDllObjectPlacement.ObjectPlacementDefinition.Create_OnFloor(halfBoxDims); | |
placementQueries.Add( | |
new PlacementQuery(placementDefinition, | |
boxFullDims, | |
objType, | |
placementRules, | |
placementConstraints | |
)); | |
} | |
return placementQueries; | |
} | |
} |
We have created some public properties on ObjectPlacer.cs, these properties keep track of the desired size of the hologram we are creating and a reference to the hologram. These are done in public properties so that changes to the hologram or its size can be made in Unity with having to change the code. In the next tutorial we will expand this concept further to allow us to create any number or holograms at different sizes. Start and Update are both updated to support boxdrawer.cs to draw boxes if the public debug boxes property is checked. We again make this a public property so that we can change it in unity without touching any code.
The CreateScene method first checks to make sure that spatial understanding is ready, if it is not it exits immediately. (In a real application you would want to do something to attempt to recover at this point) Next the Spatial Understanding solver is initialized and we update the billboard to let the user know we are generating the world. The next call creates a list of work for the solver to do containing 1 item that is trying to solve for the size of the building and then we make a call to start doing the work in GetLocationsFromSolver.
GetLocationFromSolver starts a separate thread and attempts to complete each work item in the list that was given to it by making a call to PlaceObject. The Result of that call is added to the results queue for drawing. PlaceObject makes the actual call to Spatial Understanding taking all of the information about the work to be completed as parameters.
The Update method also makes a call to ProcessPlacementResults on every frame. This method checks for any work in the work queue, and if there is any it takes one item off of the queue and draws a box for it. This method only does one item of work in the queue per frame to make sure the frame rate is not impacted.
In Unity select the Placement GameObject in the Hierarchy. Find the City_Saloon.prefab in the Assets folder in your project pane. Drag the prefab to the Saloon Building Prefab property in the Object Placer Component. Your project should now look like this:
Why did we need to reference the Saloon prefab if we are only drawing a box? The reason is because we are using the size and shape of the prefab that we eventually want to draw for the space request that we make to spatial understanding. We will also use this later when we instantiate the hologram into the world. Build the app and run the app in the emulator or on the device.
Instantiate and scale the Hologram in the world
Next make the following changes to ObjectPlacer.cs:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
using System; | |
using System.Collections.Generic; | |
using HoloToolkit.Unity; | |
using UnityEngine; | |
public class ObjectPlacer : MonoBehaviour | |
{ | |
public bool DrawDebugBoxes = false; | |
public SpatialUnderstandingCustomMesh SpatialUnderstandingMesh; | |
[Tooltip("The desired size of wide buildings in the world.")] | |
public Vector3 WideBuildingSize = new Vector3(1.0f, .5f, .5f); | |
public GameObject SaloonBuildingPrefab; | |
private readonly List<BoxDrawer.Box> _lineBoxList = new List<BoxDrawer.Box>(); | |
private readonly Queue<PlacementResult> _results = new Queue<PlacementResult>(); | |
private bool _timeToHideMesh; | |
private BoxDrawer _boxDrawing; | |
// Use this for initialization | |
void Start() | |
{ | |
if (DrawDebugBoxes) | |
{ | |
_boxDrawing = new BoxDrawer(gameObject); | |
} | |
} | |
// Update is called once per frame | |
void Update() | |
{ | |
ProcessPlacementResults(); | |
if (_timeToHideMesh) | |
{ | |
SpatialUnderstandingState.Instance.HideText = true; | |
HideGridEnableOcclulsion(); | |
_timeToHideMesh = false; | |
} | |
if (DrawDebugBoxes) | |
{ | |
_boxDrawing.UpdateBoxes(_lineBoxList); | |
} | |
} | |
private void HideGridEnableOcclulsion() | |
{ | |
SpatialUnderstandingMesh.DrawProcessedMesh = false; | |
} | |
public void CreateScene() | |
{ | |
// Only if we're enabled | |
if (!SpatialUnderstanding.Instance.AllowSpatialUnderstanding) | |
{ | |
return; | |
} | |
SpatialUnderstandingDllObjectPlacement.Solver_Init(); | |
SpatialUnderstandingState.Instance.SpaceQueryDescription = "Generating World"; | |
List<PlacementQuery> queries = CreateLocationQueriesForSolver(1, WideBuildingSize, ObjectType.WideBuilding); | |
GetLocationsFromSolver(queries); | |
} | |
private void ProcessPlacementResults() | |
{ | |
if (_results.Count > 0) | |
{ | |
var toPlace = _results.Dequeue(); | |
// Output | |
if (DrawDebugBoxes) | |
{ | |
DrawBox(toPlace, Color.red); | |
} | |
var rotation = Quaternion.LookRotation(toPlace.Normal, Vector3.up); | |
switch (toPlace.ObjType) | |
{ | |
case ObjectType.WideBuilding: | |
CreateWideBuilding(toPlace.Position, rotation); | |
break; | |
} | |
} | |
} | |
public void CreateWideBuilding(Vector3 positionCenter, Quaternion rotation) | |
{ | |
// Stay center in the square but move down to the ground | |
var position = positionCenter – new Vector3(0, WideBuildingSize.y * .5f, 0); | |
GameObject newObject = Instantiate(SaloonBuildingPrefab, position, rotation) as GameObject; | |
if (newObject != null) | |
{ | |
// Set the parent of the new object the GameObject it was placed on | |
newObject.transform.parent = gameObject.transform; | |
newObject.transform.localScale = RescaleToDesiredSizeProportional(SaloonBuildingPrefab, WideBuildingSize); | |
} | |
} | |
private Vector3 RescaleToDesiredSizeProportional(GameObject objectToScale, Vector3 desiredSize) | |
{ | |
float scaleFactor = CalcScaleFactorHelper(new List<GameObject> { objectToScale }, desiredSize); | |
return objectToScale.transform.localScale * scaleFactor; | |
} | |
private float CalcScaleFactorHelper(List<GameObject> objects, Vector3 desiredSize) | |
{ | |
float maxScale = float.MaxValue; | |
foreach (var obj in objects) | |
{ | |
var curBounds = GetBoundsForAllChildren(obj).size; | |
var difference = curBounds – desiredSize; | |
float ratio; | |
if (difference.x > difference.y && difference.x > difference.z) | |
{ | |
ratio = desiredSize.x / curBounds.x; | |
} | |
else if (difference.y > difference.x && difference.y > difference.z) | |
{ | |
ratio = desiredSize.y / curBounds.y; | |
} | |
else | |
{ | |
ratio = desiredSize.z / curBounds.z; | |
} | |
if (ratio < maxScale) | |
{ | |
maxScale = ratio; | |
} | |
} | |
return maxScale; | |
} | |
private Bounds GetBoundsForAllChildren(GameObject findMyBounds) | |
{ | |
Bounds result = new Bounds(Vector3.zero, Vector3.zero); | |
foreach (var renderer in findMyBounds.GetComponentsInChildren<Renderer>()) | |
{ | |
if (result.extents == Vector3.zero) | |
{ | |
result = renderer.bounds; | |
} | |
else | |
{ | |
result.Encapsulate(renderer.bounds); | |
} | |
} | |
return result; | |
} | |
private void DrawBox(PlacementResult boxLocation, Color color) | |
{ | |
if (boxLocation != null) | |
{ | |
_lineBoxList.Add( | |
new BoxDrawer.Box( | |
boxLocation.Position, | |
Quaternion.LookRotation(boxLocation.Normal, Vector3.up), | |
color, | |
boxLocation.Dimensions * 0.5f) | |
); | |
} | |
} | |
private void GetLocationsFromSolver(List<PlacementQuery> placementQueries) | |
{ | |
#if UNITY_WSA && !UNITY_EDITOR | |
System.Threading.Tasks.Task.Run(() => | |
{ | |
// Go through the queries in the list | |
for (int i = 0; i < placementQueries.Count; ++i) | |
{ | |
var result = PlaceObject(placementQueries[i].ObjType.ToString() + i, | |
placementQueries[i].PlacementDefinition, | |
placementQueries[i].Dimensions, | |
placementQueries[i].ObjType, | |
placementQueries[i].PlacementRules, | |
placementQueries[i].PlacementConstraints); | |
if (result != null) | |
{ | |
_results.Enqueue(result); | |
} | |
} | |
_timeToHideMesh = true; | |
}); | |
#else | |
_timeToHideMesh = true; | |
#endif | |
} | |
private PlacementResult PlaceObject(string placementName, | |
SpatialUnderstandingDllObjectPlacement.ObjectPlacementDefinition placementDefinition, | |
Vector3 boxFullDims, | |
ObjectType objType, | |
List<SpatialUnderstandingDllObjectPlacement.ObjectPlacementRule> placementRules = null, | |
List<SpatialUnderstandingDllObjectPlacement.ObjectPlacementConstraint> placementConstraints = null) | |
{ | |
// New query | |
if (SpatialUnderstandingDllObjectPlacement.Solver_PlaceObject( | |
placementName, | |
SpatialUnderstanding.Instance.UnderstandingDLL.PinObject(placementDefinition), | |
(placementRules != null) ? placementRules.Count : 0, | |
((placementRules != null) && (placementRules.Count > 0)) ? SpatialUnderstanding.Instance.UnderstandingDLL.PinObject(placementRules.ToArray()) : IntPtr.Zero, | |
(placementConstraints != null) ? placementConstraints.Count : 0, | |
((placementConstraints != null) && (placementConstraints.Count > 0)) ? SpatialUnderstanding.Instance.UnderstandingDLL.PinObject(placementConstraints.ToArray()) : IntPtr.Zero, | |
SpatialUnderstanding.Instance.UnderstandingDLL.GetStaticObjectPlacementResultPtr()) > 0) | |
{ | |
SpatialUnderstandingDllObjectPlacement.ObjectPlacementResult placementResult = SpatialUnderstanding.Instance.UnderstandingDLL.GetStaticObjectPlacementResult(); | |
return new PlacementResult(placementResult.Clone() as SpatialUnderstandingDllObjectPlacement.ObjectPlacementResult, boxFullDims, objType); | |
} | |
return null; | |
} | |
private List<PlacementQuery> CreateLocationQueriesForSolver(int desiredLocationCount, Vector3 boxFullDims, ObjectType objType) | |
{ | |
List<PlacementQuery> placementQueries = new List<PlacementQuery>(); | |
var halfBoxDims = boxFullDims * .5f; | |
var disctanceFromOtherObjects = halfBoxDims.x > halfBoxDims.z ? halfBoxDims.x * 3f : halfBoxDims.z * 3f; | |
for (int i = 0; i < desiredLocationCount; ++i) | |
{ | |
var placementRules = new List<SpatialUnderstandingDllObjectPlacement.ObjectPlacementRule> | |
{ | |
SpatialUnderstandingDllObjectPlacement.ObjectPlacementRule.Create_AwayFromOtherObjects(disctanceFromOtherObjects) | |
}; | |
var placementConstraints = new List<SpatialUnderstandingDllObjectPlacement.ObjectPlacementConstraint>(); | |
SpatialUnderstandingDllObjectPlacement.ObjectPlacementDefinition placementDefinition = SpatialUnderstandingDllObjectPlacement.ObjectPlacementDefinition.Create_OnFloor(halfBoxDims); | |
placementQueries.Add( | |
new PlacementQuery(placementDefinition, | |
boxFullDims, | |
objType, | |
placementRules, | |
placementConstraints | |
)); | |
} | |
return placementQueries; | |
} | |
} |
First we change the default for debug boxes to false. We still want the ability to turn them on and off through a check box but normally we will want them to be disabled. The method ProcessPlacementResults has been changed to calculate the correct rotation for the hologram and create the building with a call to CreateWideBuilding.
CreateWideBuilding starts by calculating the correct position for placement of the hologram. The position that spatial understanding gives us is the exact center of the area in three dimensional space, but the position that unity wants is the bottom center of the area. In other words we want the center of the desired location for the hologram in the X and Z planes, but the start of the hologram in the Y plane. Next we call instantiate passing the hologram, position and rotation desired. If the hologram is instantiated correctly, we parent the hologram to the Placement GameObject in the hierarchy and set the local scale to the desired size for the hologram.
The scaling relies on a series of utility methods that I have created called GetBoundsForAllChildren, CalcScaleFactorHelper, and RescaleToDesiredSizeProportional. GetBoundsForAllChildren examines the hologram and all its children and creates a bounding box for that hologram and its children. The bounding box is then used in CalcScaleFactorHelper to find the largest scaling factor that will allow the hologram to fit in the space requested from Spatial Understanding (desiredSize). RescaleToDesiredSizeProportional uses the results of CalcScaleFactorHelper to create a vector3 that is the local scale transform that should be applied to the hologram to get the desired size.
Build the app and run the app in the emulator or on the device. The application now displays the saloon scaled to the correct size based on the results from Spatial Understanding.
Tutorial Index
Versions: Unity 2017.1.0p5 | MixedRealityToolkit-Unity v1.2017.1.0 | Visual Studio 2017 15.3.2Unity 3D Project Creation | How to create a HoloLens project in Unity 3D |
Source Control | Configure git for HoloLens / Unity work |
Spatial Mapping | How to spatial map a Room |
Object Surface Observer | Set up fake spatial map data for the Unity editor |
TagAlongs and Billboarding | Tag along instructions to the user to force text on screen |
Spatial Understanding | Add spatial understanding to get play space detail |
Finalizing Spatial Understanding | Complete Spatial Understanding and Input Manager |
Object Placement and Scaling | Find valid locations for holograms in the play space |
Hologram Management | Manage the holograms you want to place in the world |
Cursor and Voice | Add a cursor and voice commands |
Occlusion | Add occlusion with the real world to your scene |
Colliders and Rigidbodys | Add Colliders and RigidBodys to your holograms |
- Download the completed app Western Town in the Windows Store!
- Completed Source code from the entire tutorial available on GitHub.
Hi, Cameron. How would we adapt this code to make Shape and Topology queries instead? Do you plan to cover that on another tutorial?
Hi guys! I am getting the error Placer does not exist in the current context. Is their meant to be a Placer function in Spatial Understanding State? Love the tutorials btw such a great help
I did not run into that issue, if you want to share some code that has this issue I’d be happy to take a look.
Hi Cameron! Thanks for you reply. I manged to sort the issue was a silly mistake in my code. I am currently working on my dissertation at Heriot Watt University where i am utilizing the HoloLens in order to create a companion for children. Would be awesome to sit down and pick your brains about spatial awareness. I have a working companion just need to let him use his environment to make him more life like.
That sounds really interesting Jay, I’ll shoot you an email.
Hi, I have managed to place my box in my room. The size of my debug box is 1*2*0.5 and my hologram is well inside, but it is not very large. I expected it to fill the debug box. How can I make my hologram fill the entire box ?
Thanks for your tutorials by the way.
The method RescaleToDesiredSizeProportional finds the largest possible scaling factor that fits within the bounding box of the space found by the solver. You could easily scale it to fill the box, but then your model would be stretched out of shape and most likely this isn’t desirable. The easiest path would be to request a space from the solver that has the same proportions as the model that you are rendering as a hologram. For example if your model’s actual size is 100x200x300, you could request a space that is 1x2x3 and the model would exactly fill the box. (or 2x4x6 or 1.5x3x4.5, etc) I hope that helps!
Thanks. That helps a lot. When do you think the other parts will be available ?
Great Series, thank you very much. Does SpatialUnderstanding restrict the Holograms after placement? I added a the HandDraggable class to the prefab and it seems like the building is about to move but doesn’t. That’s understandable if so, but how is done or can it be undone with worldanchor, etc.etc.
No, the spatial mapping does not place any restrictions on the holograms. It tells you where to place holograms and tracks where they have been placed, but you can choose to ignore what spatial understanding tells you. Handdraggable uses hit tests on colliders to find valid locations to move the hologram. Those colliders are created by spatial mapping and not spatial understanding, so the surfaces will not line up with the visualization of the spatial understanding. If they aren’t moving at all there is probably some other issue.
You were right, the problem was on my end. So, I’m trying to test the other SpatialUnderstandingDllObjectPlacement methods and trying to place the objects on the wall.
So I modified the placementDefinition in the CreateLocationQueriesForSolver method of the ObjectPlacer from
SpatialUnderstandingDllObjectPlacement.ObjectPlacementDefinition.Create_OnFloor(halfBoxDims);
to this:
SpatialUnderstandingDllObjectPlacement.ObjectPlacementDefinition.Create_OnWall(halfBoxDims, 0.0f, 50.0f, SpatialUnderstandingDllObjectPlacement.ObjectPlacementDefinition.WallTypeFlags.Normal, 0.0f, 0.0f);
But my custom objects (basic cubes) do not appear at all. When I switch it back to Create_OnFloor the objects do appear. What am I missing? It it the min and max height of the wall? Do I estimate those values? Or am I missing something somewhere else?
I haven’t gone through and recreated this demo yet, but do the holograms have a WorldAnchor component? If they do, it will prevent movement. In that case, you’ll need to remove the WorldAnchor component, then move the hologram to the desired location and re-add a WorldAnchor component with the new location.
Hi,
I just started this tutorial and in the section “Disable the Spatial map and the Billboard” , there are three things I noticed in a Hierarchy Panel ,
1- Basic,
2- Spatial
3- Control.
Can somebody guide me from where it come from? Is this article the continuty of another previous articles?
Or is it just prefabs of HoloToolkit?
This is a tutorial series that builds on each tutorial, this is the 8th tutorial. At the end of the article there is an index that shows them in order.
Hi, thank you for your great tutorial, now I understand more about spatial understanding.
I have a question is that after I place the holograms on the ground mesh, and if I move the camera, the holograms move with the camera, instead of staying where they are placed.
Is there some setting I have missed? thank you.
it was my mistake, I attached SimpleTagAlong on the Hologram object instead of only on billboard.
After moving SimpleTagAlong to billboard object the holograms stay where they are. thanks.
Glad the tutorials helped!
I’ve been trying to place some objects around a room, but for now this code only works with placing objects on the floor, and doesn’t work with the wall. I don’t really know why since from what I understand, the query should work with different surfaces. Do you know how can do it with walls? Thanks a lot!
In ObjectPlacer.CS in the method CreateLocationQueriesForSolver there is a call to SpatialUnderstandingDllObjectPlacement.ObjectPlacementDefinition.Create_OnFloor. If you change this to Create_OnWall you should be able to create the objects on the wall. Most likely you would want to extend some of the patterns to make this configurable per object instead of being hard coded. As with most demos/examples like this I have oversimplified these classes for the sake of making the points, but you will likely need to extend and enhance these patterns to handle the needs of your application.
Hi, i am currently following the tutorials, and using the hololens emulator.
Is there a way to make the mesh stay after I scan the room so I can place the holograms?
Sure, you can comment out the line of code that turns the mesh off, it is in objectplacer.cs in method update, there is a call to HideGridEnableOcclulsion, that would be the one to comment out.
After I added the ObjectPlacer.cs ,when I clicked the “Play” button in unity editor,the spatialUndertandingState.cs is hidden. Why? Can you help me with the problem? Does it result from the Singleton Class?