Cross-user hand tracking and shape recognition user interface

• US 8,890,813 B2
• Filed: 05/06/2013
• Issued: 11/18/2014
• Est. Priority Date: 04/02/2009
• Status: Active Grant

• Current Assignee: Oblong Industries, Inc. (Oblong, Inc.)
• Original Assignee: Oblong Industries, Inc. (Oblong, Inc.)
• Inventors: Minnen, David
• Primary Examiner(s): Shankar, Vijay
• Application Number:
  US13/888,174
• Publication Number:
  US 20140145929A1
• Time in Patent Office: 561 Days
• Field of Search: 345156-178, 178 1801- 1809, 178 2001- 2004
• US Class Current: 345/158
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/107 Static hand or arm
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

Embodiments include vision-based interfaces performing hand or object tracking and shape recognition. The vision-based interface receives data from a sensor, and the data corresponds to an object detected by the sensor. The interface generates images from each frame of the data, and the images represent numerous resolutions. The interface detects blobs in the images and tracks the object by associating the blobs with tracks of the object. The interface detects a pose of the object by classifying each blob as corresponding to one of a number of object shapes. The interface controls a gestural interface in response to the pose and the tracks.

Claims

1. A system comprising:

a detection and tracking algorithm running on a processor that is coupled to a sensor, wherein the detection and tracking algorithm is coupled to a gestural interface;

a gestural interface application running on a processor that is coupled to a sensor, the gestural interface application receiving data from the sensor corresponding to an object detected by the sensor, generating images from each frame of the data, wherein the images represent a plurality of resolutions, detecting blobs in the images and tracking the object by associating the blobs with tracks of the object, detecting a pose of the object by classifying each blob as corresponding to one of a plurality of object shapes, and generating a gesture signal in response to the pose and the tracks and controlling a component coupled to the interface system with the gesture signal.

2. A system comprising a detection and tracking algorithm running on a processor that is coupled to a sensor, wherein the detection and tracking algorithm is coupled to a gestural interface, the detection and tracking algorithm receiving sensor data of an appendage of a body, generating from the sensor data a first image having a first resolution, detecting a plurality of blobs in the first image, associating the plurality of blobs with tracks of the appendage, generating from the sensor data a second image having a second resolution, and classifying, using the second image, each blob of the plurality of blobs as one of a plurality of hand shapes.

3. A method comprising:

a detection and tracking algorithm running on a processor that is coupled to a sensor, wherein the detection and tracking algorithm is coupled to a gestural interface;

receiving sensor data of an appendage of a body;

generating from the sensor data a first image having a first resolution;

detecting a plurality of blobs in the first image;

associating the plurality of blobs with tracks of the appendage;

generating from the sensor data a second image having a second resolution; and

classifying, using the second image, each blob of the plurality of blobs as one of a plurality of hand shapes.

4. A method comprising:

a detection and tracking algorithm running on a processor that is coupled to a sensor, wherein the detection and tracking algorithm is coupled to a gestural interface;

receiving data from a sensor corresponding to an object detected by the sensor;

generating images from each frame of the data, wherein the images represent a plurality of resolutions;

detecting blobs in the images and tracking the object by associating the blobs with tracks of the object;

detecting a pose of the object by classifying each blob as corresponding to one of a plurality of object shapes; and

controlling a gestural interface in response to the pose and the tracks.

5. The method of claim 4, wherein the detecting of the pose and the tracks is based on a three-dimensional structure of the object.

6. The method of claim 4, wherein the detecting of the pose and the tracks comprises real-time local segmentation and object detection using depth data of the sensor.

7. The method of claim 4, wherein the generating of the images comprises generating at least a first image having a first resolution and a second image having a second resolution.

8. The method of claim 7, wherein the detecting of the blobs comprises detecting the blobs in the first image.

9. The method of claim 8, wherein the detecting of the pose comprises detecting the pose in the second image.

10. The method of claim 4, wherein the object is a hand of a human subject, wherein the detecting of the pose and the tracking of the object comprises skeleton-free detecting.

11. The method of claim 10, comprising determining that the hand corresponds to extrema in terms of geodesic distance from a center of body mass of the human subject.

12. The method of claim 10, wherein the detecting of the pose and the tracking of the object comprises per-frame extrema detection.

13. The method of claim 12, wherein the detecting of the pose comprises matching the extrema detected to parts of the human subject.

14. The method of claim 12, comprising identifying extrema candidates by detecting directional peaks in a first depth image of the images.

15. The method of claim 14, comprising identifying potential hands as blobs that are spatially connected to the extrema candidates.

16. The method of claim 15, comprising excluding use of a pre-specified bounding box to limit processing volume.

17. The method of claim 14, wherein the data comprises data from a depth sensor.

18. The method of claim 17, comprising forming the first depth image by down-sampling the data of the depth sensor from an input resolution to a first resolution.

19. The method of claim 14, wherein the detecting of the directional peaks comprises identifying peak pixels that extend farther than their spatial neighbors in any of a plurality of cardinal directions.

20. The method of claim 19, wherein the detecting of the blobs comprises designating each peak pixel as a seed for a blob, and bounding the blob by a maximum hand size.

21. The method of claim 20, comprising establishing the maximum hand size as a size value plus a depth-dependent slack value that represents expected depth error.

22. The method of claim 21, wherein the size value is approximately 300 millimeters (mm).

23. The method of claim 21, wherein the depth error corresponds to a physical distance represented by a plurality of adjacent raw sensor readings.

24. The method of claim 20, comprising, for each blob, estimating a center of a potential hand by identifying a pixel that is farthest from a border of the blob.

25. The method of claim 24, comprising pruning each blob using a palm radius.

26. The method of claim 25, wherein the pruning includes hand pixels and excludes pixels corresponding to other parts of the human subject.

27. The method of claim 25, wherein the palm radius is approximately 200 mm.

28. The method of claim 25, comprising identifying extension pixels that extend the blob.

29. The method of claim 28, comprising searching an outer boundary of each blob and identifying the extension pixels.

30. The method of claim 28, wherein the extension pixels include pixels adjacent to the blob that have a similar depth as pixels of the blob.

31. The method of claim 28, comprising analyzing the extension pixels for a region that is small relative to a boundary length.

32. The method of claim 31, comprising pruning blobs having a disconnected extension region.

33. The method of claim 32, wherein, in a valid hand blob, the extension region corresponds to a wrist of the human subject.

34. The method of claim 28, wherein the tracking of the object comprises matching blobs in the first image with existing tracks of the hand.

35. The method of claim 34, comprising scoring each blob/track pair according to a minimum distance between a centroid of the blob and a trajectory of the track bounded by a current velocity.

36. The method of claim 35, comprising optimizing the associating between the blobs and the tracks by minimizing a total score across all matches.

37. The method of claim 36, wherein the minimizing of the total score uses a score threshold, wherein at least one blob/track is unmatched when a score of the blob/track pair exceeds the threshold.

38. The method of claim 37, wherein the score threshold is approximately 250 mm.

39. The method of claim 34, comprising comparing remaining unmatched blobs to the existing tracks.

40. The method of claim 39, comprising adding an unmatched blob to an existing track as a secondary matched blob when the unmatched blob is in close spatial proximity to the existing track.

41. The method of claim 40, wherein a plurality of blobs are associated with a single track.

42. The method of claim 39, comprising using any remaining unmatched blobs to seed new tracks.

43. The method of claim 39, comprising using any remaining unmatched blobs to prune old tracks.

44. The method of claim 39, wherein the detecting of the pose comprises, using a second depth image of the images, identifying pixels that correspond to the tracks of the hand.

45. The method of claim 44, comprising forming the second depth image by down-sampling the data of the depth sensor from an input resolution to a second resolution.

46. The method of claim 44, comprising identifying the pixels by seeding a connected component search at each pixel within a depth distance from a corresponding pixel of the first depth image, wherein the connected component comprises the blobs of the first depth image that are spatially connected to the extrema candidates.

47. The method of claim 46, comprising re-estimating the center of the hand using the identified pixels, wherein the re-estimating provides a three-dimensional position estimate having a relatively higher sensitivity.

48. The method of claim 44, comprising classifying each blob as one of the plurality of object shapes, wherein the plurality of object shapes include a plurality of hand shapes.

49. The method of claim 48, wherein the classifying uses randomized decision forests.

50. The method of claim 49, wherein each decision forest comprises a plurality of decision trees and a final classification of each blob is computed by merging results across the plurality of decision trees.

51. The method of claim 50, wherein the plurality of decision trees are randomized.

52. The method of claim 48, wherein the classifying as one of the plurality of hand shapes comprises use of a plurality of sets of image features.

53. The method of claim 52, wherein a first set of image features comprises global image statistics.

54. The method of claim 53, wherein the global image statistics comprise at least one of percentage of pixels covered by a blob contour, a number of fingertips detected, a mean angle from a centroid of a blob to the fingertips, and a mean angle of the fingertips.

55. The method of claim 54, comprising detecting fingertips from a contour of each blob by identifying regions of high positive curvature.

56. The method of claim 53, wherein a second set of image features comprises a number of pixels covered by every grid within a bounding box of a blob normalized by its total size.

57. The method of claim 56, comprising subsampling each blob to a pre-specified grid size.

58. The method of claim 56, wherein a third set of image features comprises a difference between a mean depth for each pair of individual cells of every grid within a bounding box of a blob.

59. The method of claim 58, wherein a fourth set of image features comprises a combination of the first set of image features, the second set of image features, and the third set of image features.

60. The method of claim 52, comprising, when an extension region is identified, estimating an orientation of the hand shape based on a vector connecting a center of the extension region to the centroid of the blob.

61. The method of claim 4, wherein the sensor comprises a depth sensor.

62. The method of claim 61, wherein the depth sensor is an infrared (IR) depth sensor that outputs data of a distance between components of the object and the sensor.

63. The method of claim 62, wherein the sensor comprises an infrared (IR) emitter that illuminates the object with infrared light beams.

64. The method of claim 61, wherein the sensor comprises a video camera.

65. The method of claim 64, wherein the video camera is a color camera that outputs multi-channel data.