NATURAL USER INTERFACE CAMERA CALIBRATION

• US 20160170603A1
• Filed: 12/10/2014
• Published: 06/16/2016
• Est. Priority Date: 12/10/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Kennedy, Kevin Alan, Bastien, David
• Granted Patent: US 10,088,971 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 2203/012 Walk-in-place systems for allowing a user to walk in a virtual environment while constraining him to a given position in the physical environment
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 15/55 Radiosity
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: H04N 13/207 using a single 2D image sensor
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 2013/0081 : Depth or disparity estimation from stereoscopic image signals

Abstract

A method of calibrating a six-degree-of-freedom (6DoF) pose of a natural user interface (NUI) camera relative to a display is provided. Calibration video imaging an environment from a calibration perspective, which sites the display and one or more features, is received from the NUI camera or a calibration camera. A three-dimensional map of the environment, which defines a 6DoF pose of the display and a three-dimensional location of each of the one or more features, is modeled from the calibration video. Primary video imaging the environment from an operation perspective, which sites the one or more features, is received from the NUI camera. A 6DoF pose of the NUI camera is found within the three-dimensional map of the environment based on the operation perspective view of the one or more features.

Claims

1. A method of calibrating a six-degrees-of-freedom (6DoF) pose of a natural user interface (NUI) camera relative to a display:

receiving, from a calibration camera, calibration video imaging an environment from a calibration perspective that sites the display and one or more features;

three-dimensionally modeling, from the calibration video, a three-dimensional map of the environment, the three-dimensional map of the environment defining a 6DoF pose of the display within the environment and a three-dimensional location of each of the one or more features within the environment;

receiving, from the NUI camera, primary video imaging the environment from an operation perspective that sites the one or more features; and

finding a 6DoF pose of the NUI camera within the three-dimensional map of the environment based on the operation perspective view of the one or more features.

2. The method of claim 1, further comprising:

receiving, from the NUI camera, primary video imaging a user in the environment;

identifying three-dimensional locations of a first body part of the user and a second body part of the user relative to the display; and

visually indicating, on the display, an intersection point of the display and a ray extending from the first body part through the second body part.

3. The method of claim 1, further comprising:

translating movement of a second body part of a user towards the display as an input command directed to an object displayed at an intersection point of the display and a ray extending from a first body part of the user through the second body part.

4. The method of claim 3, wherein movement of the second body part towards the display is determined based on a distance between the second body part and a body of the user.

5. The method of claim 3, wherein movement of the second body part towards the display is determined based on a distance between the second body part and the first body part.

6. The method of claim 3, further comprising:

presenting a display object on the display; and

selecting the display object responsive to movement of the second body part towards the display when the intersection point is coincident with the display object.

7. The method of claim 2, wherein the first body part is an eye of the user and the second body part is a finger of the user.

8. The method of claim 2, wherein the first body part is an elbow of the user and the second body part is a finger of the user.

9. The method of claim 1, wherein the primary video sites the calibration camera.

10. The method of claim 9, wherein the calibration camera includes two or more reference markings visible to the NUI camera.

11. The method of claim 9, wherein the calibration camera includes a display screen configured to present a reference image visible to the NUI camera.

12. The method of claim 1, wherein the NUI camera is a depth camera.

13. The method of claim 1, wherein the display is configured to present a calibration image visible to the calibration camera.

14. The method of claim 1, wherein the calibration camera is included in a head-mounted augmented reality device wearable by a user.

15. A method of using a depth camera to control a computing system, comprising:

receiving, at the computing system, a three-dimensional model defining a 6DoF pose of a display relative to a 6DoF pose of the depth camera within an environment;

receiving, at the computing system, a machine-readable body model of a user within the environment, the machine-readable body model including a three-dimensional location of a first body part of the user and a three-dimensional location of a second body part of the user;

during a first condition, translating a position of the first body part of the user and a position of the second body part of the user into a physical interaction zone gesture that controls the computing system; and

during a second condition, translating a position of the first body part of the user and a position of the second body part of the user relative to the display into a pointing gesture that controls the computing system.

16. The method of claim 15, further comprising:

transitioning from the first condition to the second condition responsive to receiving a command from the user.

17. The method of claim 15, further comprising:

transitioning from the first condition to the second condition responsive to receiving instructions from an application executing on the computing system.

18. The method of claim 15, further comprising:

transitioning from the first condition to the second condition responsive to recognizing that a distance of the second body part relative to the display has decreased below a threshold.

19. The method of claim 15, further comprising:

during the second condition, visually indicating, on the display, an intersection point of the display and a ray extending from the first body part through the second body part.

20. A method of calibrating a six-degrees-of-freedom (6DoF) pose of a natural user interface (NUI) camera relative to a display:

receiving, from the NUI camera, calibration video imaging an environment from a calibration perspective that sites the display and one or more features;

three-dimensionally modeling, from the calibration video, a three-dimensional map of the environment, the three-dimensional map of the environment defining a 6DoF pose of the display within the environment and a three-dimensional location of each of the one or more features within the environment;

receiving, from the NUI camera, primary video imaging the environment, from an operation perspective that sites the one or more features; and

finding a 6DoF pose of the NUI camera within the three-dimensional map of the environment based on the operation perspective view of the one or more features.