Calibration systems and methods for depth-based interfaces with disparate fields of view

• US 10,437,342 B2
• Filed: 12/05/2016
• Issued: 10/08/2019
• Est. Priority Date: 12/05/2016
• Status: Active Grant

• Current Assignee: Hasivision, LLC
• Original Assignee: YouSpace, Inc.
• Inventors: Brunner, Ralph
• Primary Examiner(s): Butcher, Brian M
• Application Number:
  US15/369,799
• Publication Number:
  US 20180157328A1
• Time in Patent Office: 1,037 Days
• Field of Search: None
• CPC Class Codes: G06F 2203/0382 Plural input, i.e. interface arrangements in which a plurality of input device of the same type are in communication with a PC
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20 Special algorithmic details
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration

Abstract

Various of the disclosed embodiments provide Human Computer Interfaces (HCI) that incorporate depth sensors at multiple positions and orientations. The depth sensors may be used in conjunction with a display screen to permit users to interact dynamically with the system, e.g., via gestures. Calibration methods for orienting depth values between sensors are also presented. The calibration methods may generate both rotation and translation transformations that can be used to determine the location of a depth value acquired in one sensor from the perspective of another sensor. The calibration process may itself include visual feedback to direct a user assisting with the calibration. In some embodiments, floor estimation techniques may be used alone or in conjunction with the calibration process to facilitate data processing and gesture identification.

Claims

1. A display structure for gesture interactions comprising:

a display screen;

a plurality of sensors, at least two of the plurality of sensors configured to acquire depth information and at least two of the plurality of sensors configured to acquire pixel image information in a region before the display screen;

a computer system configured to perform a method comprising;

capturing a first pixel image at a primary sensor of at least a portion of the primary sensor'"'"'s field of view;

capturing a second pixel image at a first secondary sensor of at least a portion of the first secondary sensor'"'"'s field of view;

causing the display screen to display the first pixel image;

causing the display screen to display the second pixel image;

capturing a first set of depth values associated with a calibration object at the primary sensor when the calibration object is in both the primary sensor'"'"'s field of view and the first secondary sensor'"'"'s field of view;

capturing a second set of depth values associated with the calibration object at the first secondary sensor when the calibration object is in both the primary sensor'"'"'s field of view and the first secondary sensor'"'"'s field of view;

determining a first normal associated with a plane of the calibration object for the first set of depth values;

determining a first on-plane point associated with the plane of the calibration object for the first set of depth values;

determining a second normal associated with the plane of the calibration object for the second set of depth values;

determining a second on-plane point associated with the plane of the calibration object for the second set of depth values;

generating a first system of linear equations, in part, using a linear equation derived from the first normal and the second normal;

solving for components of a rotation transform using the first system of linear equations;

generating a second system of linear equations, in part, using a linear equation derived from the first on-plane point and the second on-plane point; and

solving for components of a translation transform using the second system of linear equations,

wherein the rotation transform and translation transform move a point from the perspective of the first secondary sensor to the perspective of the primary sensor.

2. The display structure of claim 1, wherein the first on-plane point is an average of the depth values associated with the plane visible to the primary sensor.

3. The display structure of claim 1, wherein the calibration object is a planar board.

4. The display structure of claim 1, wherein the primary sensor'"'"'s field of view and the first secondary sensor'"'"'s field of view do not overlap.

5. The display structure of claim 1, the method further comprising:

capturing a third set of depth values associated with the calibration object at a second secondary sensor when the calibration object is in both the second secondary sensor'"'"'s field of view and the first secondary sensor'"'"'s field of view, but not within the primary sensor'"'"'s field of view;

capturing a fourth set of depth values associated with the calibration object at the first secondary sensor when the calibration object is in both the second secondary sensor'"'"'s field of view and the first secondary sensor'"'"'s field of view, but not within the primary sensor'"'"'s field of view;

determining a third normal associated with a plane of the calibration object for the third set of depth values;

determining a third on-plane point associated with the plane of the calibration object for the third set of depth values;

determining a fourth normal associated with a plane of the calibration object for the fourth set of depth values; and

determining a fourth on-plane point associated with the plane of the calibration object for the fourth set of depth values,

wherein generating the first system of linear equations further comprises inserting a linear equation derived from the third normal and the fourth normal, and

wherein generating the second system of linear equations further comprises inserting a linear equation derived from the third on-plane point and the fourth on-plane point.

6. The display structure of claim 5, wherein at least two of the plurality of sensors are vertically offset relative to one another by a bracket.

20. The display structure of claim 1, wherein the display screen comprises multiple screens.

7. A computer-implemented method for calibrating a plurality of sensors, comprising:

capturing a first pixel image at a primary sensor of at least a portion of the primary sensor'"'"'s field of view;

capturing a second pixel image at a first secondary sensor of at least a portion of the first secondary sensor'"'"'s field of view;

causing a first display screen to display the first pixel image;

causing a second display screen to display the second pixel image;

capturing a first set of depth values associated with a calibration object at the primary sensor when the calibration object is in both the primary sensor'"'"'s field of view and the first secondary sensor'"'"'s field of view;

capturing a second set of depth values associated with the calibration object at the first secondary sensor when the calibration object is in both the primary sensor'"'"'s field of view and the first secondary sensor'"'"'s field of view;

determining a first normal associated with a plane of the calibration object for the first set of depth values;

determining a first on-plane point associated with the plane of the calibration object for the first set of depth values;

determining a second normal associated with the plane of the calibration object for the second set of depth values;

determining a second on-plane point associated with the plane of the calibration object for the second set of depth values;

generating a first system of linear equations, in part, using a linear equation derived from the first normal and the second normal;

solving for components of a rotation transform using the first system of linear equations;

generating a second system of linear equations, in part, using a linear equation derived from the first on-plane point and the second on-plane point; and

solving for components of a translation transform using the second system of linear equations,

wherein the rotation transform and translation transform move a point from the perspective of the first secondary sensor to the perspective of the primary sensor.

8. The computer-implemented method of claim 7, wherein the first display and the second display are the same display.

9. The computer-implemented method of claim 7, wherein the first on-plane point is an average of the depth values associated with the plane visible to the primary sensor.

10. The computer-implemented method of claim 7, wherein the calibration object is a planar board.

11. The computer-implemented method of claim 7, wherein the primary sensor'"'"'s field of view and the first secondary sensor'"'"'s field of view do not overlap.

12. The computer-implemented method of claim 7, the method further comprising:

capturing a third set of depth values associated with the calibration object at a second secondary sensor when the calibration object is in both the second secondary sensor'"'"'s field of view and the first secondary sensor'"'"'s field of view, but not within the primary sensor'"'"'s field of view;

capturing a fourth set of depth values associated with the calibration object at the first secondary sensor when the calibration object is in both the second secondary sensor'"'"'s field of view and the first secondary sensor'"'"'s field of view, but not within the primary sensor'"'"'s field of view;

determining a third normal associated with a plane of the calibration object for the third set of depth values;

determining a third on-plane point associated with the plane of the calibration object for the third set of depth values;

determining a fourth normal associated with a plane of the calibration object for the fourth set of depth values; and

determining a fourth on-plane point associated with the plane of the calibration object for the fourth set of depth values,

wherein generating the first system of linear equations further comprises inserting a linear equation derived from the third normal and the fourth normal, and

wherein generating the second system of linear equations further comprises inserting a linear equation derived from the third on-plane point and the fourth on-plane point.

13. The computer-implemented method of claim 12, wherein at least two of the plurality of sensors are vertically offset relative to one another by a bracket.

14. A non-transitory computer-readable medium comprising instructions configured to cause a computer system to perform a method comprising:

capturing a first set of depth values associated with a calibration object at a primary sensor when the calibration object is in both the primary sensor'"'"'s field of view and a first secondary sensor'"'"'s field of view;

capturing a second set of depth values associated with the calibration object at the first secondary sensor when the calibration object is in both the primary sensor'"'"'s field of view and the first secondary sensor'"'"'s field of view;

determining a first normal associated with a plane of the calibration object for the first set of depth values;

determining a first on-plane point associated with the plane of the calibration object for the first set of depth values;

determining a second normal associated with the plane of the calibration object for the second set of depth values;

determining a second on-plane point associated with the plane of the calibration object for the second set of depth values;

generating a first system of linear equations, in part, using a linear equation derived from the first normal and the second normal;

solving for components of a rotation transform using the first system of linear equations;

generating a second system of linear equations, in part, using a linear equation derived from the first on-plane point and the second on-plane point; and

solving for components of a translation transform using the second system of linear equations,

wherein the rotation transform and translation transform move a point from the perspective of the first secondary sensor to the perspective of the primary sensor.

15. The non-transitory computer-readable medium of claim 14, wherein the first on-plane point is an average of the depth values associated with the plane visible to the primary sensor.

16. The non-transitory computer-readable medium of claim 14, wherein the calibration object is a planar board.

17. The non-transitory computer-readable medium of claim 14, wherein the primary sensor'"'"'s field of view and the first secondary sensor'"'"'s field of view do not overlap.

18. The non-transitory computer-readable medium of claim 14, the method further comprising:

capturing a third set of depth values associated with the calibration object at a second secondary sensor when the calibration object is in both the second secondary sensor'"'"'s field of view and the first secondary sensor'"'"'s field of view, but not within the primary sensor'"'"'s field of view;

capturing a fourth set of depth values associated with the calibration object at the first secondary sensor when the calibration object is in both the second secondary sensor'"'"'s field of view and the first secondary sensor'"'"'s field of view, but not within the primary sensor'"'"'s field of view;

determining a third normal associated with a plane of the calibration object for the third set of depth values;

determining a third on-plane point associated with the plane of the calibration object for the third set of depth values;

determining a fourth normal associated with a plane of the calibration object for the fourth set of depth values; and

determining a fourth on-plane point associated with the plane of the calibration object for the fourth set of depth values,

wherein generating the first system of linear equations further comprises inserting a linear equation derived from the third normal and the fourth normal, and

wherein generating the second system of linear equations further comprises inserting a linear equation derived from the third on-plane point and the fourth on-plane point.

19. The non-transitory computer-readable medium of claim 18, wherein at least two of the plurality of sensors are vertically offset relative to one another by a bracket.