Large-scale surface reconstruction that is robust against tracking and mapping errors
1. A method for reconstructing a physical space, comprising:
- obtaining a plurality of images of the physical space using a camera carried by a rig while the rig moves in the physical space through a plurality of poses;
determining an orientation and a coordinate system of each pose;
partitioning the physical space into a plurality of volumes;
for each volume of the plurality of volumes, obtaining a depth map of the physical space using a depth sensor carried by the rig;
providing a pose graph comprising a plurality of nodes and links between the nodes, each node being associated with one of the poses, one of the images associated with the one of the poses and one of the coordinate systems associated with the one of the poses, and the links defining correspondences between the images and transforms between the coordinate systems;
for each volume, anchoring the volume to one of the nodes;
updating a state of the pose graph as the plurality of images are obtained, the updating comprising updating one or more of the nodes and one or more of the links; and
reconstructing a surface in one or more of the volumes according to the state of the pose graph.
Depth maps of a physical space are obtained using a depth sensor carried by a rig such as a robot or a head mounted display device worn by a user. Visible light images are also obtained. The images and orientation readings are used to create a pose graph which includes nodes connected by links. The nodes are associated with different poses of the rig and the corresponding images. Links between the nodes represent correspondences between the images, and transforms between coordinate systems of the nodes. As new images are captured, the pose graph is updated to reduce an accumulation of errors. Furthermore, surfaces in the physical space can be reconstructed at any time according to the current state of the pose graph. Volumes used in a surface reconstruction process are anchored to the nodes such that the positions of the volumes are adjusted as the pose graph is updated.
View as Search Results
|Methods for simultaneous localization and mapping (SLAM) and related apparatus and systems|
Patent #US 10,444,021 B2
Current AssigneeReification Inc.
Sponsoring EntityReification Inc.
|Scene reconstruction from bursts of image data|
Patent #US 10,535,156 B2
Current AssigneeMicrosoft Technology Licensing LLC
Sponsoring EntityMicrosoft Technology Licensing LLC
|Method and apparatus for identifying scale invariant features in an image and use of same for locating an object in an image|
Patent #US 6,711,293 B1
Current AssigneeBritish Columbia Cancer Agency Branch
Sponsoring EntityBritish Columbia Cancer Agency Branch
|System and method for the display of surface structures contained within the interior region of a solid body|
Patent #US 4,710,876 A
Current AssigneeGeneral Electric Company
Sponsoring EntityGeneral Electric Company
|Mobile Camera Localization Using Depth Maps|
Patent #US 20120194644A1
Current AssigneeMicrosoft Technology Licensing LLC
Sponsoring EntityMicrosoft Technology Licensing LLC
- 1. A method for reconstructing a physical space, comprising:
obtaining a plurality of images of the physical space using a camera carried by a rig while the rig moves in the physical space through a plurality of poses; determining an orientation and a coordinate system of each pose; partitioning the physical space into a plurality of volumes; for each volume of the plurality of volumes, obtaining a depth map of the physical space using a depth sensor carried by the rig; providing a pose graph comprising a plurality of nodes and links between the nodes, each node being associated with one of the poses, one of the images associated with the one of the poses and one of the coordinate systems associated with the one of the poses, and the links defining correspondences between the images and transforms between the coordinate systems; for each volume, anchoring the volume to one of the nodes; updating a state of the pose graph as the plurality of images are obtained, the updating comprising updating one or more of the nodes and one or more of the links; and reconstructing a surface in one or more of the volumes according to the state of the pose graph.
- View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11)
- 12. An apparatus for reconstructing a physical space, comprising:
an orientation sensor; a depth sensor; a camera; and a processor in communication with the orientation sensor, the depth sensor and the camera, the processor; obtains a plurality of images of the physical space from the camera, associates a pose and a coordinate system with each image based on the orientation sensor, partitions the physical space into a plurality of volumes; for each volume of the plurality of volumes, obtains a depth map of the physical space from the depth sensor, provides a pose graph comprising a plurality of nodes and links between the nodes, each node being associated with one of the poses, one of the images associated with the one of the poses and one of the coordinate systems associated with the one of the poses, and the links define correspondences between the images and transforms between the coordinate systems; for each volume, anchor the volume to one of the nodes; update a state of the pose graph as the plurality of images are obtained; and reconstructs a surface in one or more of the volumes according to the state of the pose graph.
- View Dependent Claims (13, 14, 15)
- 16. A processor-readable storage device having processor-readable software embodied thereon for programming a processor to perform a method for reconstructing a physical space, the method comprising:
obtaining a plurality of images of the physical space using a camera carried by a rig while the rig moves in the physical space; determining poses of the rig, each pose being associated with an orientation and a coordinate system; providing a pose graph based on the poses and the images, the pose graph comprising a plurality of nodes and links between the nodes; obtaining a depth map of a volume in the physical space using a depth sensor carried by the rig; anchoring the volume to one of the nodes; updating a state of the pose graph as the plurality of images are obtained, the updating comprises updating one of the coordinate systems which is associated with the one of the nodes; and reconstructing a surface in the volume according to the state of the pose graph.
- View Dependent Claims (17, 18, 19, 20)
Surface reconstruction (SR) involves identifying surfaces of objects in a physical space. For example, the surfaces can represents walls, floors, a ceiling, furniture or other objects in a room. The surfaces provide a model of the physical environment. Such a model can be used in a wide range of applications. One example is guiding a robot through a physical space. Another example is displaying a virtual object to a user, such by using a head mounted display device.
Technology described herein provides various embodiments for reconstructing surfaces in a physical space.
Depth maps of a physical space are obtained using a depth sensor. A depth map contains depth values which represent a distance from the depth sensor to surfaces in a physical space. The depth maps are defined with respect to volumes in the physical space, and the location of the volumes is defined with respect to a coordinate system. For example, the coordinate system can be defined according to a pose (e.g., position and orientation) of a rig on which the depth sensor is carried.
However, errors in the position of the volumes can accumulate as additional depth maps are obtained and additional volumes are defined. To reduce the accumulation of errors, visible light images of the physical space are obtained and used to form a pose graph. The pose graph comprises nodes which are associated with different poses of the rig and the corresponding images. Links between the nodes represent correspondences between the images, and transforms between coordinate systems of the nodes. From time to time, as new images are captured, the pose graph is updated. For example, a node can be updated by updating the coordinate system associated with the node.
By anchoring the volumes to the nodes of the pose graph, the surface reconstruction process benefits from the increased accuracy which results from updating of the coordinate systems. As a result, the volumes and surfaces can be placed more accurately. Surfaces can be reconstructed at any time according to the current state of the pose graph.
This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.
In the drawings, like-numbered elements correspond to one another.
Surface reconstruction can be used in many applications, including placing a virtual object, defining a play space, path-finding, collision detection or occlusion of virtual objects a physical space.
Surface reconstruction over large scales requires robustness against tracking and mapping errors. For example, a physical space having a large scale can be a room in a home, office or museum, or an outdoor space. In a surface reconstruction process, surfaces are acquired using multiple depth maps which are acquired by a depth sensor carried on a rig. The surfaces are localized with respect to each other using computer vision tracking and mapping techniques which are prone to cumulative inaccuracies. Furthermore, pose estimates of the rig change over time as more information about the environment becomes available.
Some SR solutions operate on a small scale or ignore tracking errors. However, this results in SR data that is not self-consistent. The techniques provided herein address the above and other issues. In one approach, an SR process involves creating multiple separate SR volumes to reconstruct. Each volume is anchored to (e.g., associated with) a node (e.g., a key frame) in a tracking and mapping pose graph. When the pose graph updates, such as due to a bundle adjustment or a re-localization, the volumes automatically adjust their pose accordingly.
Additional aspects include allowing the volumes to overlap somewhat when they are initially placed, to avoid gaps which might otherwise occur when the position of the volumes is adjusted based on an update to the pose graph. New volumes can be created, or existing volumes can be expanded in size, to remove any gaps in the coverage of the physical space which may occur. Moreover, when SR volumes overlap excessively, redundant volumes can be deleted. Volumes can be occasionally re-anchored to a closest node if appropriate.
A depth sensor 25 obtains a depth map of the physical space. Typically, the depth maps are obtained less frequently than the camera images. The depth sensor can be a time-of-flight camera or a stereoscopic camera, for instance. A gaze detection system 26 may be used to determine the user'"'"'s gaze in the case of the rig being a head mounted display device. For example, this can be useful in placing virtual objects in the physical space. An orientation sensor 27 obtains readings regarding an orientation of the rig. For, example, an orientation reading can be associated with each image and depth map. These can be inertial sensors, for instance, discussed further below. A memory 28 can store executable code which is executed by the control circuits to provide the functionality described herein, as well as storing information which is obtained by the visible light camera 24, the depth sensor 25, the gaze detection system 26 and the orientation sensor 27.
Optionally, the rig communicates with a server 10. For example, the rig could upload some of the information that it gathers or download information such as executable code, data for the augmented reality projection system, or other content. In one approach, the server is remote from the rig. In another approach, the rig communicates with a local hub computing system 50, such as in the user'"'"'s home. The hub computing system could be a gaming console which runs various gaming and non-gaming applications, for instance. See also
In one approach, the network 30 is a wireless network such as WI-FI®, BLUETOOTH®, infra-red (e.g., IrDA®) or cellular (e.g., GSM). Optionally, multiple rigs can communicate with one another within a common physical space.
Further details of these system are provided in connection with the following figures.
Generally, the communication interfaces allow communication between computing devices. The control circuits provide control of hardware and/or software of the respective computing devices. For example, the control circuits can include one or more processors which execute instructions stored on one or more tangible, non-transitory processor-readable storage devices having processor-readable software embodied thereon for programming a processor to perform processor- or computer-implemented methods as described herein. The memories can store the instructions as code, and can provide the processor-readable storage devices. The memories can provide databases, data stores or other sources of data which are accessed to perform the techniques described herein. The memories can be hardware memory devices.
The HMD device can be worn on the head of a user so that the user can see through a display and thereby see a real-world scene which includes an image which is not generated by the HMD device. The HMD device can be self-contained so that all of its components are carried by the frame. Optionally, one or more components of the HMD device are not carried by the frame. For example, one of more components which are not carried by the frame can be physically attached by a wire to a component carried by the frame.
Further, one of more components which are not carried by the frame can be in wireless communication with a component carried by the frame, and not physically attached by a wire or otherwise to a component carried by the frame. The one or more components which are not carried by the frame can be carried by the user, in one approach, such as on the wrist. For example, the processing unit 105 could be connected to a component in the frame via a wire or via a wireless link. The term “HMD device” can encompass both on-frame components and associated off-frame components.
The processing unit 105 includes much of the computing power used to operate the HMD device. The processor may execute instructions stored on a processor readable storage device for performing the processes described herein. In one embodiment, the processing unit communicates wirelessly with one or more servers, hub computing systems, other HMDs or other computing devices. Control circuits 136 provide various electronics that support the other components of the HMD device.
At the front of HMD device are one or more forward- or room-facing visible light cameras 113. The cameras can include at least one visible light video camera that can capture video and still images, and transmit the images to the processing unit. The camera can be used to identify features in the physical space, such as colors, patterns, shapes, lines and so forth. A depth sensor can be formed by the combination of an infrared emitter 115 and an infrared sensor/detector 117. The visible light camera may also be part of the depth sensor. The visible light camera 113 faces outward and has a viewpoint similar to that of the user.
A portion of the frame of the HMD device surrounds a display that includes one or more lenses. The display includes a light guide optical element 112, opacity filter 114, a front, right-side see-through lens 116 and a rear, right-side see-through lens 118. In one embodiment, opacity filter 114 is behind and aligned with see-through lens 116, light guide optical element 112 is behind and aligned with opacity filter 114, and see-through lens 118 is behind and aligned with light guide optical element 112. Opacity filter 114 filters out natural light (either on a per pixel basis or uniformly) to enhance the contrast of the augmented reality imagery. Light guide optical element 112 channels artificial light to the eye. Similarly, the left side of the HMD includes a front, left-side see-through lens 119 and a rear, left-side see-through lens 121.
Mounted to or inside temple 102 is an image source, which (in one embodiment) includes microdisplay 120 for projecting an augmented reality image and lens 122 for directing images from the microdisplay into light guide optical element 112. In one embodiment, lens 122 is a collimating lens. An augmented reality emitter can include the microdisplay, one or more optical components such as the lens 122 and light guide optical element 112, and associated electronics such as a driver. Such an augmented reality emitter is associated with the HMD device, and emits light to a user'"'"'s eye, where the light represents augmented reality still or video images. This can be used to display a virtual object in a physical space.
Control circuits 136, discussed further in connection with
The microdisplay projects an image through lens 122. Light guide optical element 112 transmits light from the microdisplay to the eye 140 of the user while allowing light from in front of the HMD device to be transmitted through light guide optical element to eye 140, as depicted by arrow 142, allowing the user to have an actual direct view of the space in front of HMD device, in addition to receiving an augmented reality image from the microdisplay. Thus, the walls of light guide optical element are see-through. Light guide optical element includes a first reflecting surface 124 (e.g., a mirror or other surface). Light from the microdisplay passes through lens 122 and is incident on reflecting surface 124, which reflects the incident light such that light is trapped inside a planar, substrate comprising light guide optical element by internal reflection. After several reflections off the surfaces of the substrate, the trapped light waves reach an array of selectively reflecting surfaces, including example surface 126.
Reflecting surfaces 126 couple the incident light waves out of the substrate into the eye 140 of the user. Different light rays will travel and bounce off the inside of the substrate at different angles as they hit the various reflecting surface 126 at different angles. Therefore, different light rays will be reflected out of the substrate by different ones of the reflecting surfaces. The selection of which light rays will be reflected out of the substrate by which surface 126 is engineered by selecting an appropriate angle of the surfaces 126. In one embodiment, each eye will have its own light guide optical element. When the HMD device has two light guide optical elements, each eye can have its own microdisplay that can display the same image in both eyes or different images in the two eyes. In another embodiment, there can be one light guide optical element which reflects light into both eyes.
Note that some of the components (e.g., eye tracking camera 134B, microdisplay 120, opacity filter 114, eye tracking illumination 134A and earphones 130) are shown in shadow to indicate that there are two of each of those devices, one for the left side and one for the right side of HMD device. Similarly, the depth sensor 313 can include an infrared emitter 115 and an infrared sensor/detector 117, for instance. In another approach, two or more cameras with a known spacing between them are used as a depth camera to obtain depth data for objects in a room, indicating the distance from the cameras/HMD device to the object. Optionally, one visible light camera 113 is used. The eye tracking camera 134B and the eye tracking illumination 134A are part of an eye tracking component 134.
Images from the forward-facing cameras can be used to identify people, hand gestures and other objects in a field of view of the user. For example, it can be determined when the user makes a hand gesture such as pointing at an object. The real-world object can be identified and associated with a data stream, or used as the location to display a previously-associated data stream.
A control circuit 300 communicates with the power management circuit 302. Control circuit 300 includes processor 310, memory controller 312 in communication with memory 344 (e.g., DRAM), camera interface 316, camera buffer 318, display driver 320, display formatter 322, timing generator 326, display out interface 328, and display in interface 330. A GPS circuit 317 can be used to identify the location of the HMD device.
In one embodiment, all of components of control circuit 300 are in communication with each other via dedicated lines or one or more buses. In another embodiment, each of the components of control circuit 300 communicates with processor 310. A camera interface/buffer 316 provides an interface to, and stores images from, the visible light camera. A depth sensor interface/buffer 318 provides an interface to, and stores images from, the depth sensor.
Display driver 320 drives the microdisplay. Display formatter 322 provides information, about the augmented reality image being displayed on the microdisplay, to opacity control circuit 324, which controls opacity filter 114. Timing generator 326 is used to provide timing data for the system. Display out interface 328 is a buffer for providing images from forward-facing cameras 113 to the processing unit 105. Display in interface 330 is a buffer for receiving images such as an augmented reality image to be displayed on the microdisplay.
Display out interface 328 and display in interface 330 communicate with band interface 332 which is an interface to processing unit, when the processing unit is attached to the frame of the HMD device by a wire, or communicates by a wireless link, and is worn on the wrist of the user on a wrist band. This approach reduces the weight of the frame-carried components of the HMD device. In other approaches, as mentioned, the processing unit can be carried by the frame and a band interface is not used.
Power management circuit 302 includes voltage regulator 334, eye tracking illumination driver 337, audio DAC and amplifier 338, microphone preamplifier audio ADC 340 and clock generator 345. Voltage regulator 334 receives power from processing unit via band interface 332 and provides that power to the other components of HMD device. Eye tracking illumination driver provides the infrared (IR) light source for eye tracking illumination 134A, as described above. Audio DAC and amplifier 338 receives the audio information from earphones 130. Microphone preamplifier and audio ADC 340 provides an interface for microphone 110. Power management unit 302 also provides power and receives data back from three-axis magnetometer 132A, three-axis gyroscope 132B and three axis accelerometer 132C, as part of an orientation sensor.
In one embodiment, wireless communication component 446 can include a Wi-Fi® enabled communication device, BLUETOOTH® communication device, or infrared communication device. The wireless communication component 446 is a wireless communication interface which, in one implementation, receives data in synchronism with the content displayed by the HMD device. Further, augmented reality images may be displayed in response to the received data. In one approach, such data is received from a server, a hub computing system, a local Wi-Fi® network or WLAN, a cell phone network, a local storage device or other source.
The USB port can be used to dock the processing unit to hub computing system 50 to load data or software onto processing unit, as well as charge processing unit. In one embodiment, CPU 420 and GPU 422 are the main workhorses for determining where, when and how to insert augmented reality images into the view of the user.
Power management circuit 406 includes clock generator 460, analog to digital converter 462, battery charger 464, voltage regulator 466 and HMD power source 476. Analog to digital converter 462 is connected to a charging jack 470 for receiving an AC supply and creating a DC supply for the system. Voltage regulator 466 communicates with battery 468 for supplying power to the system. Battery charger 464 is used to charge battery 468 (via voltage regulator 466) upon receiving power from charging jack 470. HMD power source 476 provides power to the HMD device.
The calculations that determine where, how and when to insert an augmented reality image can be performed by the HMD device and/or by a computing device which communicates with the HMD device.
In one example embodiment, the HMD device will create a model of the environment that the user is in and track various objects in that environment, based on the field of view of the HMD device. The model and the tracking information are provided to processing unit. Sensor information obtained by HMD device is transmitted to processing unit. Processing unit refines the field of view of the user and provide instructions to HMD device on how, where and when to insert augmented reality images.
CPU, memory controller, and various memory devices are interconnected via one or more buses (not shown).
In one implementation, CPU, memory controller, ROM, and RAM are integrated onto a common module 514. In this implementation, ROM is configured as a flash ROM that is connected to memory controller via a PCI bus and a ROM bus (neither of which are shown). RAM is configured as multiple Double Data Rate Synchronous Dynamic RAM (DDR SDRAM) modules that are independently controlled by memory controller via separate buses (not shown). Hard disk drive and portable media drive are shown connected to the memory controller via the PCI bus and an AT Attachment (ATA) bus 516.
A GPU 520 and a video encoder 522 form a video processing pipeline for high speed and high resolution graphics processing.
An audio processing unit 524 and an audio codec (coder/decoder) 526 form a corresponding audio processing pipeline for multi-channel audio processing of various digital audio formats. Audio data are carried between audio processing unit and the audio codec via a communication link (not shown). The video and audio processing pipelines output data to an A/V (audio/video) port 528 for transmission to a television or other display. In the illustrated implementation, video and audio processing components 520-528 are mounted on the module.
The module 514 includes a USB host controller 531 and a network interface 532. USB host controller is shown in communication with CPU and memory controller via a bus (e.g., PCI bus) and serves as host for peripheral controllers 504(1)-504(4). Network interface provides access to a network (e.g., Internet, home network, etc.) and may be any of a wide variety of various wired or wireless interface components.
In the implementation depicted, the console includes a controller support subassembly 540 for supporting the four peripheral controllers. The controller support subassembly includes any hardware and software components needed to support wired and wireless operation with an external control device, such as for example, a media and game controller. A front panel I/O subassembly 542 supports the multiple functionalities of power button 512, the eject button 541, as well as any LEDs (light emitting diodes) or other indicators exposed on the outer surface of console. Subassemblies and are in communication with module via one or more cable assemblies 544. In other implementations, the console can include additional controller subassemblies. An optical I/O interface 535 sends and receives signals that can be communicated to module 514. The interface may be responsive to a remote control 590.
Memory units (MUs) 540(1) and 540(2) are connectable to MU ports “A” 530(1) and “B” 530(2) respectively. Additional MUs (e.g., MUs 540(3)-540(6)) are illustrated as being connectable to the peripheral controllers 504(1) and 504(3), i.e., two MUs for each controller. Controllers 504(2) and 504(4) can also be configured to receive MUs (not shown). Each MU offers additional storage on which games, game parameters, and other data may be stored. In some implementations, the other data can include any of a digital game component, an executable gaming application, an instruction set for expanding a gaming application, and a media file. When inserted into the console or a controller, MU can be accessed by memory controller. A system power supply module 550 provides power to the components of the console. A fan 552 cools the circuitry within the console. A microcontroller unit 554 is also provided.
An application 560 comprising machine instructions is stored on hard disk drive. When the console is powered on, various portions of application are loaded into the RAM, and/or the caches, for execution on the CPU, wherein application is one such example. Various applications can be stored on hard disk drive for execution on CPU.
The console may be operated as a standalone system by simply connecting the system to a monitor, a television, a video projector, or other display device. In this standalone mode, the console enables one or more players to play games, or enjoy digital media, e.g., by watching movies, or listening to music. However, with the integration of broadband connectivity made available through network interface, the console may further be operated as a participant in a larger network gaming community.
For instance, in the SIFT technique, keypoints of objects in a physical space are first extracted from a set of reference images and stored in a database. An object is recognized in a new image by individually comparing each feature from the new image to this database and finding candidate matching features based on Euclidean distance of their feature vectors. From the full set of matches, subsets of keypoints that agree on the object and its location, scale, and orientation in the new image are identified to filter out good matches. The determination of consistent clusters is performed rapidly by using an efficient hash table implementation of the generalized Hough transform. Each cluster of three or more features that agree on an object and its pose is then subject to further detailed model verification and, subsequently, outliers are discarded. Finally, the probability that a particular set of features indicates the presence of an object is computed, given the accuracy of fit and number of probable false matches. Object matches that pass all these tests can be identified as correct with high confidence. See, e.g., U.S. Pat. No. 6,711,293, “Method and apparatus for identifying scale invariant features in an image and use of same for locating an object in an image,” David Lowe, issued Mar. 23, 2004 and incorporated herein by reference.
Decision step 611 determines whether the pose is new. In one approach, if the pose is not sufficiently different than a previous pose or another pose, it may not be considered to be new, in which case the flow returns to step 610. For example, the pose may be new if the orientation is significantly different than in other nodes. Or, the pose may be new if features of the image are significantly different than in other nodes or if a specified amount of time has passed since last new pose. In one approach, the camera provides a sequence of video images and selected images are suitable for use in defining a node.
If the pose is new, step 612 adds a new node to the pose graph. Step 613 determines a coordinate system associated with the orientation of the pose. Or, the coordinate system may already have been determined. Step 614 associates the image taken with the pose (e.g., the image captured by the camera on the rig when the rig/camera has assumed the pose) and the coordinate system with the new node. For example, this can involve storing the information in a database, as discussed further below. An image can be represented and stored in any form, including pixel data and/or extracted feature data. Step 615 determines a correspondence between the image associated with the new node and images of one or more other nodes in the pose graph. For instance, this can involve comparing the features of the image of the new node to features of images of one or more other nodes which are close to the new node in the pose graph. The correspondence between two images is relatively high when the features in the two images are relatively similar. For example, the degree of correspondence can be determined using the SIFT technique.
Step 616 determines a transform between the coordinate system of the new node and the coordinate systems of the one or more other nodes. Step 617 associates the transform and the correspondence with one or more links between the new node and the one or more other nodes. Step 618 updates the other nodes and links in the pose graph if appropriate. For example, updating of the state of the pose graph can involve performing a bundle adjustment in which the coordinate systems and the transforms are adjusted, or updating one or more of the correspondences, or performing a relocalization in which there is a change in topology in the pose graph. For instance, the relocalization can comprise a loop closure, in which a closed loop of the nodes is formed in the pose graph.
A tracking subsystem 720 converts sensor data into poses and a map of interconnected coordinate frames. For example, this can be a head tracking subsystem in the example case of an HMD device. The tracking subsystem 720 is responsive to a map service subsystem 741, and includes a tracking and mapping process 721. The tracking subsystem 720 and the map service subsystem 741 together form a tracking and mapping subsystem 742.
An SR Fusion and Extraction Kernel (SRFEK) converts depth maps and poses into surface data. The SRFEK runs on the ASIC, and communicates with other processes in both the ASIC and the SoC. The SRFEK takes in two pieces of input data: depth maps and poses. A raw to depth subsystem 731 captures raw infrared images from the depth sensor and converts them into depth values in depth maps. The depth values indicates a distance of one or more surfaces in the physical space from the depth sensor. These depth maps have lens distortion. To address this, an undistortion subsystem 732 removes the effects of distortion from the lens of the camera from the depth images, to provide an undistorted depth map. This can involve applying a model of lens distortion, whose parameters are measured through calibration, to yield a flat undistorted image. At runtime, this can involve creating an undistortion map (a lookup table) and applying it to each depth map pixel.
A 3D fusion subsystem 734 combines the undistorted depth map with a pose to fuse it into a 3D volume. Fusion involves projecting depth maps (which are 2D images) into a volume (which is 3D). For example, a signed distance function (SDF) can be used which represents a distance from the depth sensor to the surface of an object. See
An extraction algorithm 733 creates a polygon mesh from that volumetric data. Not all applications require a polygon mesh, but they are often useful in applications such as game engines which use meshes for rendering and collision. One way to extract a mesh after populating the SDF volume uses an algorithm called Marching Cubes. The idea behind Marching Cubes involves turning each box of volumetric data into a set of polygons, where the vertices of those polygons lie on edges of the boxes. For further details, see U.S. Pat. No. 4,710,876, “System and method for the display of surface structures contained within the interior region of a solid body,” incorporated herein by reference.
Outputs of the 3D fusion subsystem 734 and the extraction algorithm 733 can be stored in a SRFEK cache 735. This output includes the reconstructed surfaces which represent the geometry of the physical space.
A SR service 750 is a higher level process which controls SRFEK, giving it chunks of data and commands to process that data. The SR service also feeds surface data which is obtained from surface reconstruction to applications. The service includes a depth map acquisition subsystem 751 which decides when to trigger the acquisition of depth maps, based on where and when they are needed. A volume placement subsystem 752 decides how to partition the physical space into volumes, e.g., how many volumes to create and where to place them. Deciding where to place volumes is the initial problem. We can start with a set of previously-defined SR volumes. See
SR data can occupy a lot of memory, so the service decides which data to keep in memory, e.g., an SR service cache 754, and which to store for later use, e.g., in a backing store 755, using a cache eviction and replenishment subsystem 753. The SR service decides what SR data to evict and replenish, when to evict and replenish the data, and supplies applications with SR data. The SR service is useful in managing scale, both in terms of the size of the physical space and the quantity of data which is generated. The system should work on physical scales that extend beyond the immediate vicinity of the user, e.g., in physical spaces of several meters such as in an office building. Brute-force solutions will not work because even modestly-sized regions uses hundreds or thousands of SR volumes and each SR volume can use a few megabytes. The SR system can scale by partitioning a space into volumes and paging (e.g., transferring) those volumes between memory and disk.
The SR Service can accomplish both persistence and data scalability with the cache eviction and replenishment subsystem. For example, this subsystem can establish a maximum number of volumes. This provides a cap on the amount of SoC main memory the SR service consumes. When that cap would be exceeded by adding more data, or more SR volumes, we can evict some other volumes. This is analogous to a least-recently-used (LRU) eviction policy, except the SR Service can evict the farthest volumes, e.g., farthest from the rig. Moreover, when the rig re-enters a region for which it previously captured SR data, the cache can be replenished with that data. Thus, we can intermittently replenish the cache from the backing store by loading the closest volumes to the rig. Note that new volumes can still be added so they also contribute to the cache population.
Many applications can consume SR data in similar ways. An SR extensions and utilities subsystem 760 provides common functionality for application developers. Extensions and utilities can include occlusion rendering 761 for occluding holograms with real-world geometry, mesh post-processing 762 to add useful auxiliary information, collision detection 763 between real and virtual objects, play space definition 764 and negative space allocation 765 to help with hologram placement. Examples of SR mesh post-processing routines include welding vertices, zippering meshes, computing surface normals, clipping triangles from overlapping meshes, smoothing meshes, and simplifying meshes to reduce the number of polygons.
Regarding hologram placement, meshes can be too fine and complicated for certain kinds of hologram placement logic. Instead, application developers can find it easier to deal with abstractions such as walls, ceilings and floors. One solution is to find flat triangles, identify connected flat regions, and then fit planes to those regions. To find contiguous regions of low curvature, multiple separate meshes are first combined into a single contiguous mesh. This is where welding and zippering are useful.
Regarding computing surface normals, to find flat regions, it is useful to compute surface curvature. The SR pipeline can generate per-vertex normals, either from the SDF (using finite differences) or from the mesh (using per-face normals).
Another example of SR mesh post-processing is applying texture as captured from the camera.
In contrast, suppose we initially placed SR volumes to be perfectly aligned with each other, with no gap or overlap. Then, after pose adjustment, the SR volumes could have gaps between them. We therefore initially place SR volumes to have some moderate overlap. That way, when they move due to pose adjustment, there is less of a chance they will have gaps between them. Gaps are generally undesirable since they result in discontinuities in the reconstructed surfaces.
Clipping of blocks in a volume can be useful as a counterpart to providing overlap between volumes. As mentioned, pose adjustments cause SR volumes to move relative to each other. To pre-emptively avoid gaps, the SR service places volumes to have some overlap. But the overlap is at best redundant and at worst conflicting. The SR service deletes blocks that are entirely redundant, but this can still leave extraneous triangles or other shaped regions in those blocks which only partially overlap. Applications often want to omit the overlap regions. The SR utilities therefore can include a fast and efficient mesh clipping procedure to eliminate overlapping triangles. The clipping routine can provide the option to allow either a small overlap or a small gap. Both are useful. For example, a small overlap provides an easy and fast way to generate a mesh that appears to have no gaps. This is useful for occlusion rendering. On the other hand, a small gap facilitates zippering multiple separate meshes into a single contiguous mesh, which facilitates further mesh processing.
In particular, a pose graph of a rig can be used to optimize the placement of volumes in a physical space for surface reconstruction. The problem of spatial partitioning is not specific to SR, and there are many spatial partitioning schemes. To understand why the techniques described herein work well for SR, first consider various aspects of how rig tracking works. The tracking and mapping subsystem 742 (
The link between nodes represents two things: correspondences between images and a transform between the coordinate systems of the two nodes. However, when the mapper can obtain multiple estimates for the transform between nodes, those transforms can be inconsistent. The tracking and mapping system minimizes the inconsistency by adjusting the coordinate transforms and feature locations. This is called bundle adjustment. As the rig moves around, the mapper continues to add nodes and links, and this results in additional pose adjustments. As a result, the map of the physical space changes over time. That is, estimates of feature positions move and the pose graph is refined. For example, the links in the pose graph can change distance or orientation. SR should be robust against pose adjustments.
To understand how severe this problem can be and how far a feature in an image can appear to move, consider loop closure—an abrupt form of pose graph adjustment. Loop closure is a form of relocalization. For example, imagine the rig visits separate spaces without visiting the regions that connect them. The mapper will have separate map components for each space. Then, imagine the rig visits the regions between the spaces. The tracking and mapping subsystem will recognize that the regions are connected, and will add links to the pose graph. Relocalization refers to any topological change to the pose graph.
One form of loop closure occurs when the rigs ends up at a point in space where it already visited, and the tracking and mapping subsystem observes that fact. For example, imagine a user wearing an HMD device walks into a room such as in an art gallery and always faces the wall nearest to the user, beginning in a starting position. The user does not look across the room or behind the user. The user then circumnavigates the room, looking at each wall in turn, as the tracking and mapping subsystem creates new nodes. Just before the user returns to the starting position, the tracking and mapping subsystem does not yet recognize that it is at the starting point again. But, eventually, the tracking and mapping subsystem will recognize features captured while the user was at the starting point and will identify correspondences between images taken at the beginning and end of this loop. It will therefore create a new link in the pose graph, thus forming a closed loop. Just before loop closure, the transform between the start and end of the loop entails many hops. Just after loop closure, the transform between the start and end of the loop entails only a single hop. Each hop has some error, and the errors accumulate. The amount of accumulated errors is smaller just after loop closure. So, just before and just after loop closure, the transforms between nodes will abruptly change. The techniques provided herein allow the SR process to benefit from adjustments to the pose graph by anchoring volumes to the nodes and their coordinate systems in a pose graph. When the pose graph changes, the SR volumes come along for the ride since the volumes follow the pose graph node to which they are anchored.
Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.