Room based sensors in an augmented reality system

• US 10,127,723 B2
• Filed: 05/06/2015
• Issued: 11/13/2018
• Est. Priority Date: 04/18/2014
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Miller, Samuel A.
• Primary Examiner(s): Patel, Jitesh
• Application Number:
  US14/705,920
• Publication Number:
  US 20150302642A1
• Time in Patent Office: 1,287 Days
• CPC Class Codes: A63F 13/56 Computing the motion of game characters with respect to other game characters, game objects or elements of the game scene, e.g. for simulating the behaviour of a group of virtual soldiers or for path finding
• CPC Class Codes: A63F 13/57 Simulating properties, behaviour or motion of objects in the game world, e.g. computing tyre load in a car race game A63F13/56 takes precedence
• CPC Class Codes: A63F 13/577 using determination of contact between game characters or objects, e.g. to avoid collision between virtual racing cars
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/0101 characterised by optical features G02B27/0172 takes precedence
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G02B 27/0179 Display position adjusting means not related to the information to be displayed
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G06F 3/167 : Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06Q 30/02 : Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06T 11/60 : Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 13/40 : of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 13/80 : 2D [Two Dimensional] animation, e.g. using sprites
• CPC Class Codes: G06T 15/10 : Geometric effects
• CPC Class Codes: G06T 19/003 : Navigation within 3D models or images
• CPC Class Codes: G06T 19/006 : Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/04 : involving 3D image data
• CPC Class Codes: G06T 2213/08 : Animation software package
• CPC Class Codes: G06T 2215/16 : Using real world measurements to influence rendering
• CPC Class Codes: G06T 2219/024 : Multi-user, collaborative environment
• CPC Class Codes: G06T 7/60 : Analysis of geometric attributes
• CPC Class Codes: G06T 7/70 : Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 20/10 : Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/20 : in augmented reality scenes
• CPC Class Codes: G06V 20/653 : by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces
• CPC Class Codes: G06V 20/80 : Recognising image objects characterised by unique random patterns
• CPC Class Codes: H04S 2400/11 : Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 2400/15 : Aspects of sound capture and related signal processing for recording or reproduction
• CPC Class Codes: H04S 7/304 : For headphones

Abstract

An augmented reality display system comprises a passable world model data comprises a set of map points corresponding to one or more objects of the real world. The augmented reality system also comprises a processor to communicate with one or more individual augmented reality display systems to pass a portion of the passable world model data to the one or more individual augmented reality display systems, wherein the piece of the passable world model data is passed based at least in part on respective locations corresponding to the one or more individual augmented reality display systems.

Claims

1. A sensor system, comprising:

a plurality of sensors or transducers to respectively capture information pertaining to one or more images or keyframes of a space, wherein respective pose information of the plurality of sensors or transducers relative to the space is known and comprises an orientation of the plurality of sensors or transducers within the space;

at least one processor of a remote computing system configured to receive the information captured by the plurality of sensors or transducers and to determine whether the information is to be used in constructing or updating a map residing on the remote computing system and to select and discard at least a first part of the information that is determined not to be used to construct or update the map;

the at least one processor further configured to identify a plurality of layers of signature information pertaining to at least the space;

the at least one processor further configured to modify the map into an overlaid map at least by placing the plurality of layers of signature information in context with each other in relation to the map;

the at least one processor further configured to determine multiple pieces of location information from at least the plurality of layers of signature information that are aligned in the overlaid map, the multiple pieces of location information respectively and uniquely corresponding to a plurality of locations in the space;

the at least one processor further configured to determine multiple pieces of fingerprint data for the plurality of locations in the space using at least some of the multiple pieces of location information;

the at least one processor further configured to construct or update at least a portion of the map residing on the remote computing system and corresponding to the space at least by selecting and using the at least a second part of the information from the information captured by the plurality of sensors or transducers to represent the space as a node and by storing one or more pieces of fingerprint data for the node in a set of fingerprint data, rather than storing a keyframe or an image from which the fingerprint data is derived, based at least in part on the at least the second part of the information; and

a wireless network to transmit the overlaid map to a virtual or augmented reality display system that determines a real-world location in the space at least by using at least the set of fingerprint data derived at least from images or keyframes pertaining to a plurality of nodes in the overlaid map, and displays virtual content to the user of the virtual or augmented reality display system in relation to the real-world location.

2. The sensor system of claim 1, wherein the plurality of sensors or transducers comprise an image-based sensor, and the wireless network is further configured to transmit at least the keyframe or image corresponding to the node but not keyframes or images that are not relevant to the location from the cloud to the virtual or augmented reality system after the real-world location is determined, and the plurality of sensors or transducers comprise an audio sensor or an environment sensor.

3. The sensor system of claim 1, wherein the real-world location represents a room in the space, and the one or more pieces of fingerprint data for the node comprise at least one piece of fingerprint data that uniquely identifies the room.

4. The sensor system of claim 1, wherein the virtual or augmented reality display system is further configured to determine the real-world location of the user at least by using the map, rather than by processing geometric information pertaining to geometries of a world model that is constructed to represent a real-world, wherein the map does not include feature information about all features or points captured in the images or keyframes.

5. The sensor system of claim 1, wherein the plurality of sensors or transducers comprise a temperature-based sensor.

6. The sensor system of claim 1, wherein the plurality of sensors or transducers comprise a humidity-based sensor.

7. The sensor system of claim 1, wherein the pose comprises a position of the plurality of sensors or transducers within a room in the space.

8. The sensor system of claim 1, wherein the information is captured with respect to a reference frame corresponding to the space.

9. The sensor system of claim 1, wherein the sensor system is stationary.

10. The sensor system of claim 8, wherein the at least one processor performs a transformation to relate a sensor reference frame of the plurality of sensors or transducers to a space reference frame corresponding to the space.

11. The sensor system of claim 10, wherein the transformation comprises a translation matrix.

12. The sensor system of claim 10, wherein the transformation comprises a rotation matrix.

13. The sensor system of claim 1, wherein the at least one processor is further configured to dispose of the at least the first part of the information captured by at least one of the plurality of sensors or transducers based in part or in whole upon one or more criteria.

14. The sensor system of claim 13, wherein the one or more criteria comprise redundancy or quality of the at least the first part of the information or the at least the second part of the information.

15. The sensor system of claim 10, wherein the at least one processor is further configured to construct or update the map at least by adding a separate node representing a second space in a real world to the map and by connecting the separate node to the node with an edge.

16. The sensor system of claim 15, wherein the at least one processor is further configured to represent the edge with a graphical emphasis in the map, and the graphical emphasis is indicative of a number of real world features or nodes shared between the node and the separate node.

17. The sensor system of claim 15, wherein the wireless network is further configured to receive environment information of a physical environment in which the user is located, to upload the environment information into a world model corresponding to the map, and to display interactions between the virtual content and the physical environment to the user based in part or in whole upon the environment information via the virtual or augmented reality display system.

18. A method for displaying virtual contents with a sensor system, the method comprising:

capturing, at a plurality of sensors or transducers, information pertaining to one or more images or keyframes of a space, wherein a respective pose information of the plurality of sensors or transducers relative to the space is known and comprises an orientation of the plurality of sensors or transducers within the space;

receiving, by at least one processor of a remote computing system operatively coupled to the sensor or transducer, the information and determining whether the information is to be used in constructing or updating a map residing on the remote computing system and to select and discard at least a first part of the information that is determined not to be used to construct or update the map;

identifying a plurality of layers of signature information pertaining to at least the space;

modifying the map into an overlaid map at least by placing the plurality of layers of signature information in context with each other in relation to the map;

determining multiple pieces of location information from at least the plurality of layers of signature information that are aligned in the overlaid map, the multiple pieces of location information respectively and uniquely corresponding to a plurality of locations in the space;

determining multiple pieces of fingerprint data for the plurality of locations in the space using at least some of the multiple pieces of location information;

constructing or updating, at the at least one processor, at least a portion of the map residing on the remote computing system and corresponding to the space at least by selecting and using the at least a second part of the information from the information captured by the plurality of sensors or transducers to represent the space as a node and by storing one or more pieces of fingerprint data for the node in a set of fingerprint data, rather than storing a keyframe or an image from which the fingerprint data is derived, based at least in part on the at least the second part of the information; and

transmitting, via a wireless network, the overlaid map to a virtual or augmented reality display system that determines a real-world location in the space at least by using at least the set of fingerprint data derived at least from images or keyframes pertaining to a plurality of nodes in the overlaid map, and displays virtual content to a user of the virtual or augmented reality display system in relation to the real-world location.

19. The method of claim 18, further comprising at least one of:

determining the real-world location of a user at least by using the map, rather than by processing geometric information pertaining to geometries of a world model that is constructed to represent a real-world, wherein the map does not include feature information about all features or points captured in the images or keyframes;

transmitting at least the keyframe or image corresponding to the node but not keyframes or images that are not relevant to the location from the cloud to the virtual or augmented reality system after the real-world location is determined;

disposing the at least the first part of the information captured by the plurality of sensors or transducers based in part or in whole upon one or more criteria;

or

updating the map at least by adding a separate node representing a second space in the real world to the overlaid map and by connecting the separate node to the node with an edge, wherein the edge is indicative of a number of real world features or nodes shared between the node and the separate node.