Floor estimation for human computer interfaces

• US 10,030,968 B2
• Filed: 02/08/2016
• Issued: 07/24/2018
• Est. Priority Date: 02/08/2016
• Status: Active Grant

• Current Assignee: Hasivision, LLC
• Original Assignee: YouSpace, Inc.
• Inventors: Brunner, Ralph
• Primary Examiner(s): Menz, Laura
• Application Number:
  US15/018,048
• Publication Number:
  US 20170227353A1
• Time in Patent Office: 897 Days
• CPC Class Codes: G01B 11/22 for measuring depth
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06V 10/267 by performing operations on regions, e.g. growing, shrinking or watersheds
• CPC Class Codes: G06V 40/113 Recognition of static hand signs
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

Human Computer Interfaces (HCI) may allow a user to interact with a computer via a variety of mechanisms, such as hand, head, and body gestures. Various of the disclosed embodiments allow information captured from a depth camera on an HCI system to be used to recognize such gestures. Particularly, the HCI system'"'"'s depth sensor may capture depth frames of the user'"'"'s movements over time. To discern gestures from these movements, the system may group portions of the user'"'"'s anatomy represented by the depth data into classes. This grouping may require that the relevant depth data be extracted from the depth frame. Such extraction may itself require that appropriate clipping planes be determined. Various of the disclosed embodiments better establish floor planes from which such clipping planes may be derived.

Claims

1. A computer system configured to determine a floor plane estimation from depth frame data comprising:

a depth sensor device configured to acquire a frame of depth data;

at least one processor;

at least one memory comprising instructions configured to cause the at least one processor to cause the computer system to perform a method comprising;

receiving a frame of depth data from the depth sensor, the depth data comprising a point cloud;

determining an initial floor plane;

generating a plurality of candidate floor planes by;

rotating a normal associated with a reference floor plane;

determining a metric value for each of the plurality of candidate floor planes, wherein determining a metric value for a candidate floor plane of the plurality of candidate for planes comprises;

determining a first plurality of distances by projecting points from the point cloud upon the candidate floor plane;

determining an origin of the candidate floor plane based upon a portion of the first plurality of distances;

determining a second plurality of distances between points in the point cloud and the origin; and

determining the metric value based upon the second plurality of distances; and

selecting the candidate floor plane associated with the best metric value as the determined floor plane.

2. The computer system of claim 1, wherein the reference floor plane is the initial floor plane in a first iteration considering a first candidate floor plane of the plurality of candidate floor planes and a previously considered candidate floor plane in a subsequent iteration considering a second candidate floor plane of the plurality of candidate floor planes.

3. The computer system of claim 1, wherein determining an origin of the candidate floor plane based upon a portion of the first plurality of distances comprises selecting a point corresponding with a distance at a percentage boundary of the first plurality of distances.

4. The computer system of claim 1, wherein rotating the normal associated with the reference floor plane comprises rotating the normal by a random amount within a range.

5. The computer system of claim 1, wherein determining an origin of the candidate floor plane based upon a portion of the first plurality of distances comprises determining an average of points corresponding to the portion of the first plurality of distances, and wherein

determining the metric value based upon the second plurality of distances comprises summing the second plurality of distances.

6. The computer system of claim 5, wherein the portion of the first plurality of distances comprises a smallest first percentage of distances of the first plurality of distances, wherein

the points in the point cloud used to determine the second plurality of distances are associated with a smallest second percentage of distances of the first plurality of distances, and wherein

the first percentage is less than the second percentage.

7. The computer system of claim 6, wherein the distances comprise absolute magnitudes of the distances from depth points to the candidate floor plane.

8. The computer system of claim 6, wherein the first percentage is 5 percent and the second percentage is 10 percent.

9. The computer system of claim 6, the method further comprising:

excluding a portion of the point cloud by raising the determined floor plane and excluding points in the point cloud below the raised determined floor plane; and

performing gesture recognition on point cloud data above the raised determined floor plane.

10. The computer system of claim 1, wherein determining an origin of the candidate floor plane based upon a portion of the first plurality of distances comprises determining an average of points corresponding to the portion of the first plurality of distances, and wherein

determining the metric value based upon the second plurality of distances comprises determining a variance of the second plurality of distances.

11. A computer-implemented method for determining a floor plane estimation from depth frame data, the depth frame data comprising a point cloud, the method comprising:

receiving a frame of depth data captured by a depth sensor;

determining an initial floor plane;

generating a plurality of candidate floor planes by;

rotating a normal associated with a reference floor plane;

determining a metric value for each of the plurality of candidate floor planes, wherein determining a metric value for a candidate floor plane of the plurality of candidate for planes comprises;

determining a first plurality of distances by projecting points from the point cloud upon the candidate floor plane;

determining an origin of the candidate floor plane based upon a portion of the first plurality of distances;

determining a second plurality of distances between points in the point cloud and the origin; and

determining the metric value based upon the second plurality of distances; and

selecting the candidate floor plane associated with the best metric value as the determined floor plane.

12. The computer-implemented method of claim 11, wherein the reference floor plane is the initial floor plane in a first iteration considering a first candidate floor plane of the plurality of candidate floor planes and a previously considered candidate floor plane in a subsequent iteration considering a second candidate floor plane of the plurality of candidate floor planes.

13. The computer-implemented method of claim 11, wherein rotating the normal associated with the initial floor plane comprises rotating the normal by a random amount within a range.

14. The computer-implemented method of claim 11, wherein determining an origin of the candidate floor plane based upon a portion of the first plurality of distances comprises determining an average of points corresponding to the portion of the first plurality of distances, and wherein

determining the metric value based upon the second plurality of distances comprises summing the second plurality of distances.

15. The computer-implemented method of claim 14, wherein the portion of the first plurality of distances comprises a first percentage portion of the smallest distances of the first plurality of distances, wherein

the points in the point cloud used to determine the second plurality of distances are associated with a smallest second percentage of distances of the first plurality of distances, and wherein

the first percentage is less than the second percentage.

16. The computer-implemented method of claim 15, wherein the first percentage is 5 percent and the second percentage is 10 percent.

17. The computer-implemented method of claim 15, the method further comprising:

excluding a portion of the point cloud by raising the determined floor plane and excluding points in the point cloud below the raised determined floor plane; and

performing gesture recognition on point cloud data above the raised determined floor plane.

18. A non-transitory computer readable medium comprising instructions configured to cause a computer system to perform a method, comprising:

receiving a frame of depth data from a depth sensor;

determining an initial floor plane;

generating a plurality of candidate floor planes by;

rotating a normal associated with a reference floor plane;

determining a metric value for each of the plurality of candidate floor planes, wherein determining a metric value for a candidate floor plane of the plurality of candidate for planes comprises;

determining a first plurality of distances by projecting points from the point cloud upon the candidate floor plane;

determining an origin of the candidate floor plane based upon a portion of the first plurality of distances;

determining a second plurality of distances between points in the point cloud and the origin; and

determining the metric value based upon the second plurality of distances; and

selecting the candidate floor plane associated with the best metric value as the determined floor plane.

19. The non-transitory computer readable medium of claim 18, wherein determining an origin of the candidate floor plane based upon a portion of the first plurality of distances comprises determining an average of points corresponding to the portion of the first plurality of distances, and wherein

determining the metric value based upon the portion of the second plurality of distances comprises summing the second plurality of distances.

20. The non-transitory computer readable medium of claim 19, wherein the portion of the first plurality of distances comprises a first percentage portion of the smallest distances of the first plurality of distances, wherein

the points in the point cloud used to determine the second plurality of distances are associated with a smallest second percentage of distances of the first plurality of distances, and wherein

the first percentage is less than the second percentage.

21. The non-transitory computer readable medium of claim 20, the method further comprising:

excluding a portion of the point cloud by raising the determined floor plane and excluding points in the point cloud below the raised determined floor plane; and

performing gesture recognition on point cloud data above the raised determined floor plane.

22. The non-transitory computer readable medium of claim 18, wherein the reference floor plane is the initial floor plane in a first iteration considering a first candidate floor plane of the plurality of candidate floor planes and a previously considered candidate floor plane in a subsequent iteration considering a second candidate floor plane of the plurality of candidate floor planes.