Locating and orienting device in space

• US 9,144,744 B2
• Filed: 06/10/2013
• Issued: 09/29/2015
• Est. Priority Date: 06/10/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Langlois, Eric, Pinto, Ed, Lopez Ruiz, Marcelo, Manion, Todd, Mathe, Zsolt
• Primary Examiner(s): TORIMIRO, ADETOKUNBO OLUSEGUN
• Application Number:
  US13/914,600
• Publication Number:
  US 20140364227A1
• Time in Patent Office: 841 Days
• Field of Search: 463/20, 463/25, 463/30, 463/32, 463/36, 463/37, 463/39, 345/158
• US Class Current: 1/1
• CPC Class Codes: A63F 13/00 Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: A63F 13/20 Input arrangements for video game devices
• CPC Class Codes: A63F 13/28 responding to control signals received from the game device for affecting ambient conditions, e.g. for vibrating players' seats, activating scent dispensers or affecting temperature or light controlling the output signals based on the game progress A63F13/50
• CPC Class Codes: A63F 13/52 involving aspects of the displayed game scene
• CPC Class Codes: A63F 13/56 Computing the motion of game characters with respect to other game characters, game objects or elements of the game scene, e.g. for simulating the behaviour of a group of virtual soldiers or for path finding
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883

Abstract

Example apparatus and methods concern an improved immersive experience for a video gamer that is provided by controlling a game based on the three dimensional location and orientation of a control and display device held by or otherwise associated with the gamer. The location is determined from data comprising a three dimensional position and an orientation of a portion of a player in a three dimensional space associated with a computerized game. The facing and rotation of the device is determined as a function of both the location of the device and the orientation of the device. The orientation may be determined by data from motion sensors in or on the device. Example apparatus and methods control the computerized game based, at least in part, on the position of the device, the facing of the device, and the rotation of the device.

Claims

1. A computer-readable storage device storing computer-executable instructions that when executed by a computer control the computer to perform a method for locating and orienting a device in space, the method comprising:

acquiring first electronic data comprising a three dimensional position and an orientation of a portion of a user in a three dimensional space associated with a computerized interactive experience, where the first data is acquired from a first system configured to produce a depth map of the user in the three dimensional space, or where the first data is generated from an infrared interrogation of the three dimensional space or where the first data is generated from a camera detecting visible light associated with the three dimensional space;

determining, as a function of the first data, a position in the three dimensional space of a device held by the user, the device being a controller configured to control the computerized interactive experience or to display information associated with the computerized interactive experience;

acquiring second electronic data comprising an orientation of the device in the three dimensional space, where the second data is generated by a sensor in the device or where the second data is generated by an apparatus on the device, where the sensor is an accelerometer, a gyroscope, or a compass, and where the apparatus is an infrared reflector, an infrared transmitter, a sound reflector, a sound transmitter, a visible light reflector, or a visible light transmitter;

determining a facing of the device and a rotation of the device as a function of the first data or the second data, and

controlling the computerized interactive experience based, at least in part, on the position of the device, the facing of the device, and the rotation of the device.

2. The computer-readable storage device of claim 1, comprising:

constructing a representation of a skeleton of the user from the depth map, where the skeleton comprises a plurality of joints of the user, where a joint is described by a location, a bend angle, and an orientation, and

determining the position of the device as a function of a relationship between two or more joints in the representation.

3. The computer-readable storage device of claim 1, comprising:

upon determining that the depth map is missing information concerning a joint that is relevant to determining the position of the device, estimating the missing information based on a selected subset of the first data.

4. The computer-readable storage device of claim 1, comprising:

acquiring the second data from the device,

computing, from the second data, a three dimensional facing vector that describes a facing axis perpendicular to a display on the device, and

computing, from the second data, a rotation of the device about the facing axis.

5. The computer-readable storage device of claim 1, comprising:

filtering the first data to produce a filtered and stable location data;

filtering the second data to produce a filtered and stable orientation data; and

computing the position of the device, the facing of the device, and the rotation of the device as a function of the filtered location data or the filtered orientation data,

where filtering the first data to produce the filtered location data includes passing the first data through a Kalman filter, an extended Kalman filter, an unscented Kalman Filter, a high pass filter, a low pass filter, a Bayesian network filter, or a particle filter, and

where filtering the second data to produce the filtered orientation data includes passing the second data through a Kalman filter, a high pass filter, a low pass filter, a Bayesian network filter, or a particle filter.

6. The computer-readable storage device of claim 5, comprising:

combining information derived from the filtered and stable location data with information derived from the filtered and stable orientation data according to a weighted blending function, where the weights for the weighted blending function are based, at least in part, on a completeness of the first data, a completeness of the second data, an accuracy of the first data, an accuracy of the second data, a timeliness of the first data, or a timeliness of the second data.

7. The computer-readable storage device of claim 1, comprising:

dynamically reconfiguring a function for determining the position based, at least in part, on a completeness of the first data, an accuracy of the first data, or a timeliness of the first data, and

dynamically reconfiguring a function for determining a facing of the device and a rotation of the device based, at least in part, on a completeness of the first data, a completeness of the second data, an accuracy of the first data, an accuracy of the second data, a timeliness of the first data, or a timeliness of the second data.

8. The computer-readable storage device of claim 1, comprising:

determining the position of the device, the facing of the device, or the rotation of the device as a function of a location and an orientation of a hand of the player, or

determining the position of the device, the facing of the device, or the rotation of the device as a function of a location and an orientation of the head of the player or as a function of a direction of a gaze of the player.

9. The computer-readable storage device of claim 1, where the computerized interactive experience is a computerized game or a digital media with which the user interacts.

10. The computer-readable storage device of claim 1, where controlling the computerized interactive experience includes selectively altering a display of a game scene, selectively altering a display of an avatar associated with the user, providing an alternate view into a media simulation, changing the alternative view in a simulation space as the user moves through real space, or presenting an item in the alternative view in the context of a real space via the media simulation.

11. A computer-readable storage device storing computer-executable instructions that when executed by a computer control the computer to perform a method, the method comprising:

acquiring, from a first system configured to produce a depth map of a player in a three dimensional space associated with a computerized game, first data comprising a three dimensional position and an orientation of a portion of the player in the three dimensional space;

constructing a representation of a skeleton of the player from the depth map, where the representation of the skeleton is a function of locations, bends, and orientations of a plurality of joints of the player;

determining, as a function of a relationship between two or more joints in the representation of the skeleton, a position in the three dimensional game space of a control and display device held by the player;

acquiring, from a sensor in the control and display device or from an apparatus on the control and display device, second data comprising an orientation of the control and display device in the three dimensional game space, the sensor being an accelerometer, a gyroscope, or a compass, the apparatus being an infrared reflector, an infrared transmitter, a sound reflector, a sound transmitter, a visible light reflector, or a visible light transmitter;

determining, from the representation of the skeleton or from the second data, a facing of the control and display device and a rotation of the control and display device as a function of the first data or the second data, where the facing is described by a three dimensional facing vector that describes a facing axis perpendicular to a display on the control and display device;

determining, from the representation of the skeleton or from the second data, a rotation of the control and display device about the facing axis;

dynamically reconfiguring a function for determining the position based, at least in part, on a completeness of the first data, an accuracy of the first data, or a timeliness of the first data;

dynamically reconfiguring a function for determining a facing of the device and a rotation of the device based, at least in part, on a completeness of the first data, a completeness of the second data, an accuracy of the first data, an accuracy of the second data, a timeliness of the first data, or a timeliness of the second data, and

controlling the computerized game based, at least in part, on the position of the control and display device, the facing of the control and display device, and the rotation of the control and display device, where controlling the computerized game comprises selectively altering a game scene display as a function of the position of the control and display device, the facing of the control and display device, or the rotation of the control and display device, selectively altering a display of an avatar associated with the player as a function of the position of the control and display device, the facing of the control and display device, or the rotation of the control and display device, providing an alternate view into a media simulation, changing the alternative view in a simulation space as the player moves through real space, or presenting an item in the alternative view in the context of a real space via the media simulation.

12. An apparatus, comprising:

a processor;

a memory;

a set of logics configured to control a computerized interactive experience; and

an interface to connect the processor, the memory, and the set of logics;

the set of logics comprising;

a first logic that determines a three-dimensional position of a controller held by a user in a three-dimensional space associated with the computerized interactive experience using a depth map, where the depth map comprises location and aspect information concerning a head/body joint of the user, a shoulder joint of the user, an elbow joint of the user, a wrist joint of the user, a finger joint of the user, a body/pelvis joint of the user, a hip joint of the user, a knee joint of the user, or a foot joint of the user, the controller being configured with a display to display information associated with the computerized interactive experience;

a second logic that determines an orientation of the controller; and

a third logic that controls the computerized interactive experience based on the three-dimensional position and the orientation.

13. The apparatus of claim 12, where the second logic determines the orientation as a function of motion sensor data acquired from the controller, as a function of the depth map, or as a function of a combination of the motion sensor data and the depth map.

14. The apparatus of claim 13, where the second logic determines the three-dimensional position using first data from the depth map and determines the orientation using second data from the motion sensor data, where the first data and the second data share a standard system for representing the three-dimensional position and the orientation.

15. The apparatus of claim 14, where the first logic and the second logic dynamically reconfigure how to determine the three-dimensional position of a controller and to dynamically reconfigure how to determine the orientation of the controller based on the quality, accuracy, or completeness of the depth map or the motion sensor data.

16. The apparatus of claim 14, where the third logic controls the computerized interactive experience to selectively alter a scene displayed in the computerized interactive experience as a function of the position of the device and the orientation of the controller, to selectively alter an avatar associated with the user as a function of the position of the device and the orientation of the controller, to selectively alter a display attribute associated with a virtual reality aspect of the computerized interactive experience, to selectively alter a display attribute associated with an augmented reality aspect of the computerized interactive experience, to selectively alter an x-ray attribute of an item in the computerized interactive experience, to provide an alternate view into a media simulation, to change the alternative view in a simulation space as the user moves through real space, or to present an item in the alternative view in the context of a real space via the media simulation.