Adaptive tracking system for spatial input devices

• US 9,740,922 B2
• Filed: 01/27/2015
• Issued: 08/22/2017
• Est. Priority Date: 04/24/2008
• Status: Active Grant

• Current Assignee: Oblong Industries, Inc. (Oblong, Inc.)
• Original Assignee: Oblong Industries, Inc. (Oblong, Inc.)
• Inventors: Csaszar, Ambrus, Kogan, Dima, Yarin, Paul
• Primary Examiner(s): Shankar, Vijay
• Application Number:
  US14/606,563
• Publication Number:
  US 20150371082A1
• Time in Patent Office: 938 Days
• Field of Search: 345156-165, 345173-178, 178 1801- 1809, 178 2001- 2004
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: G16C 10/00 Computational theoretical chemistry, i.e. ICT specially adapted for theoretical aspects of quantum chemistry, molecular mechanics, molecular dynamics or the like
• CPC Class Codes: H04N 23/661 Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 23/695 Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

An adaptive tracking system for spatial input devices provides real-time tracking of spatial input devices for human-computer interaction in a Spatial Operating Environment (SOE). The components of an SOE include gestural input/output; network-based data representation, transit, and interchange; and spatially conformed display mesh. The SOE comprises a workspace occupied by one or more users, a set of screens which provide the users with visual feedback, and a gestural control system which translates user motions into command inputs. Users perform gestures with body parts and/or physical pointing devices, and the system translates those gestures into actions such as pointing, dragging, selecting, or other direct manipulations. The tracking system provides the requisite data for creating an immersive environment by maintaining a model of the spatial relationships between users, screens, pointing devices, and other physical objects within the workspace.

Claims

1. A system comprising:

a plurality of sensors, wherein a location of the plurality of sensors defines a spatial operating environment (SOE) that includes a plurality of objects; and

a tracking component running on a processor, wherein the tracking component receives from each sensor of the plurality of sensors feature data corresponding to each object of the plurality of objects detected by the respective sensor, wherein the tracking component generates and maintains from the feature data a coherent model of relationships between the plurality of objects and the SOE.

2. The system of claim 1, wherein the coherent model includes spatial relationships between the plurality of objects.

3. The system of claim 2, wherein the coherent model includes at least one of location, orientation, and motion of the plurality of objects.

4. The system of claim 1, wherein the ATC generates coincidence between virtual space and physical space that includes the SOE.

5. The system of claim 1, wherein a sensor detects a pose comprising location and orientation of the object relative to the sensor, wherein the pose comprises a six-degree-of-freedom (DOF) pose.

6. The system of claim 1, wherein the plurality of objects include at least one of a body, an appendage of a body, a device, an article of clothing, a glove, a display device, a piece of furniture.

7. The system of claim 1, wherein an origin of the coherent model is defined relative to a particular sensor of the plurality of sensors.

8. The system of claim 1, wherein an output of the ATC controls at least one of a function of an application, a display component, and a remote component coupled to the processor.

9. The system of claim 1, wherein the tracking component automatically detects a gesture of at least one object of the plurality of objects.

10. The system of claim 9, wherein the tracking component identifies a gesture of at least one object of the plurality of objects.

11. The system of claim 9, comprising translating the gesture to a gesture signal, and controlling a component coupled to the processor in response to the gesture signal.

12. The system of claim 11, wherein the detecting comprises identifying the gesture, wherein the identifying includes identifying a pose and an orientation of a portion of the object.

13. The system of claim 12, wherein the translating comprises translating information of the gesture to a gesture notation, wherein the gesture notation represents a gesture vocabulary, and the gesture signal comprises communications of the gesture vocabulary.

14. The system of claim 13, wherein the gesture vocabulary represents in textual form at least one of instantaneous pose states of kinematic linkages of the object, an orientation of kinematic linkages of the object, and a combination of orientations of kinematic linkages of the object.

15. The system of claim 13, wherein the gesture vocabulary includes a string of characters that represent a state of kinematic linkages of the object.

16. The system of claim 11, wherein controlling the component comprises controlling a three-space object in six degrees of freedom simultaneously by mapping the gesture to the three-space object, wherein the plurality of objects includes the three-space object.

17. The system of claim 16, comprising presenting the three-space object on a display device.

18. The system of claim 16, comprising controlling movement of the three-space object by mapping a plurality of gestures to a plurality of object translations of the three-space object.

19. The system of claim 16, wherein the detecting comprises detecting when an extrapolated position of the object intersects virtual space, wherein the virtual space comprises space depicted on a display device, wherein controlling the component comprises controlling a virtual object in the virtual space when the extrapolated position intersects the virtual object.

20. The system of claim 1, comprising detecting an event of a source device, wherein the source device includes the tracking component running on the processor, generating at least one data sequence comprising device event data specifying the event and state information of the event, wherein the device event data and state information are type-specific data having a type corresponding to an application of the source device, and forming a data capsule to include the at least one data sequence, the data capsule having a data structure comprising an application-independent representation of the at least one data sequence.

21. The system of claim 20, wherein the generating of the at least one data sequence comprises:

generating a first respective data set that includes first respective device event data;

generating a second respective data set that includes second respective state information; and

forming a first data sequence to include the first respective data set and the second respective data set.

22. The system of claim 21, wherein the generating of the first respective data set includes forming the first respective data set to include identification data of the source device, the identification data including data identifying the source device.

23. The system of claim 21, wherein the generating of the at least one data sequence comprises:

generating a first respective data set that includes first respective device event data;

generating a second respective data set that includes second respective state information; and

forming a second data sequence to include the first respective data set and the second respective data set.

24. The system of claim 23, wherein the generating of the first respective data set includes generating a first respective data set offset, wherein the first respective data set offset points to the first respective data set of the second data sequence.

25. The system of claim 23, wherein the generating of the second respective data set includes generating a second respective data set offset, wherein the second respective data set offset points to the second respective data set of the second data sequence.

26. The system of claim 21, wherein the first respective data set is a description list, the description list including a description of the data.

27. The system of claim 20, wherein the device event data is a tagged byte-sequence representing typed data, wherein the device event data includes a type header and a type-specific data layout.

28. The system of claim 20, wherein the state information is a tagged byte-sequence representing typed data, wherein the state information includes a type header and a type-specific data layout.

29. The system of claim 20, comprising:

generating at least one offset; and

forming the data capsule to include the at least one offset.

30. The system of claim 29, comprising:

generating a first offset having a first variable length;

wherein the first offset points to the device event data of a first data sequence of the at least one data sequence.

31. The system of claim 29, comprising:

generating a second offset having a second variable length;

wherein the second offset points to the state information of a first data sequence of the at least one data sequence.

32. The system of claim 29, comprising:

forming a first code path through the data capsule using a first offset of the at least one offset;

forming a second code path through the data capsule using a second offset of the at least one offset;

wherein the first code path and the second code path are different paths.

33. The system of claim 32, wherein at least one of the first offset and the second offset include metadata, the metadata comprising context-specific metadata corresponding to a context of the application.

34. The system of claim 20, comprising:

generating a header that includes a length of the data capsule;

forming the data capsule to include the header.

35. The system of claim 20, comprising transferring the data capsule to a repository coupled to the processor.

36. The system of claim 35, comprising:

detecting a second event of a second source device;

searching the repository for data capsules corresponding to the second event.

37. The system of claim 36, comprising:

identifying a correspondence between the data capsule and the second event;

extracting the data capsule from the repository in response to the identifying; and

executing on behalf of the second source device a processing operation corresponding to the second event on behalf of the second source device in response to contents of the data capsule, wherein the source device corresponds to an application of a first type and the second source device corresponds to a second application of a second type.

38. The system of claim 35, wherein the repository is coupled to a plurality of applications, the repository including a plurality of data capsules corresponding to the plurality of applications, the repository providing access to the plurality of data capsules by the plurality of applications, wherein at least two applications of the plurality of applications are different applications.

39. The system of claim 35, wherein the repository provides state caching of a plurality of data capsules.

40. The system of claim 35, wherein the repository provides linear sequencing of a plurality of data capsules.

41. The system of claim 20, wherein the data structure is untyped.

42. The system of claim 20, wherein the data structure of the data capsule provides a platform-independent representation of the device event data and the state information, and platform-independent access to the device event data and the state information.

43. The system of claim 1, wherein the feature data is absolute three-space location data of an instantaneous state of the respective body at a point in time and space.

44. The system of claim 1, comprising a plurality of tags affixed to the plurality of objects, wherein the feature data is derived using the plurality of tags.

45. The system of claim 44, wherein each tag of the plurality of tags comprises at least one feature that is detected and localized by the plurality of sensors.

46. The system of claim 44, wherein each tag includes at least one of labeling information, identity information, and pose information.

47. The system of claim 44, wherein a projective image of a tag includes labeling, wherein the at least one feature comprises at least one marker, wherein the labeling relates at least one point in the projective image to at least one corresponding marker.

48. The system of claim 44, wherein a projective image of a tag includes identity, wherein the at least one feature comprises a plurality of markers on the tag, wherein the identity distinguishes a first tag of the plurality of tags from a second tag of the plurality of tags.

49. The system of claim 44, wherein a projective image of a tag includes pose information.

50. The system of claim 49, wherein the pose information includes translation information and rotation information.

51. The system of claim 50, wherein the pose information relates a position and orientation of a tag to a position and orientation of the SOE.

52. The system of claim 1, wherein each sensor corresponds to a sensing volume in the SOE, wherein each sensor estimates a pose of an object within the sensing volume.

53. The system of claim 52, wherein the pose comprises at least one of location and orientation of the object.

54. The system of claim 52, wherein the sensing volume of each sensor at least partially overlaps with the sensing volume of at least one other sensor of the plurality of sensors, wherein a combined sensing volume of the plurality of sensors is contiguous.

55. The system of claim 1, wherein the tracking component generates for each sensor of the plurality of sensors a pose model of a pose relative to the SOE.

56. The system of claim 55, wherein, when a plurality of sensors all detect a first object at an instant in time, the tracking component generates a spatial relationship between the plurality of sensors, wherein the tracking component updates the coherent model using the spatial relationship.

57. The system of claim 56, wherein the tracking component defines an origin of the coherent model relative to a particular object of the plurality of objects, wherein the particular tag has a fixed pose relative to the SOE.

58. The system of claim 56, wherein the tracking component defines an origin of the coherent model relative to a particular sensor of the plurality of sensors and a particular object of the plurality of objects, wherein the particular object has a fixed pose relative to the SOE.

59. The system of claim 56, wherein correct pose models are determined for each sensor.

60. The system of claim 59, wherein an object is tracked by a sensor at a plurality of points in time and a plurality of pose models are generated for the object, wherein a plurality of confidence metrics are generated for the plurality of pose models and the plurality of pose models are culled based on the plurality of confidence metrics to remove any inconsistent pose models.

61. The system of claim 59, wherein an object is tracked by a plurality of sensors at a plurality of points in time and a plurality of sets of pose models are developed for the object, wherein each set of pose models comprises a plurality of pose models corresponding to each point in time.

62. The system of claim 61, wherein a plurality of confidence metrics are generated for the plurality of pose models of each set of pose models, and the plurality of sets of pose models are culled based on the plurality of confidence metrics to remove any inconsistent pose models.

63. The system of claim 61, wherein an average hypothesis comprises an average of the plurality of pose models of each set of pose models, wherein the average hypothesis approximates a maximum likelihood estimate for a true pose of a corresponding object.

64. The system of claim 63, wherein the average hypothesis comprises at least one of a positional component and a rotational component.

65. The system of claim 63, wherein a smoothed hypothesis is generated through application of a correction factor to the average hypothesis.

66. The system of claim 65, wherein the smoothed hypothesis is generated when at least one additional sensor detects an object, wherein the at least one additional sensor has not previously detected the object.

67. The system of claim 65, wherein the smoothed hypothesis is generated when at least one sensor of the plurality of sensors ceases detecting an object, wherein the at least one additional sensor has previously detected the object.

68. The system of claim 65, wherein the smoothed hypothesis comprises at least one of a positional component and a rotational component.

69. The system of claim 1, wherein at least one sensor of the plurality of sensors measures in real-time object poses of at least one object of the plurality of objects.

70. The system of claim 69, wherein the tracking component automatically adapts to changes in the object poses.

71. The system of claim 70, wherein the tracking component generates a model of a pose and a physical size of the at least one object.

72. The system of claim 70, wherein the at least one sensor is affixed to at least one location on a periphery of the at least one object, wherein the at least one object is a display device.

73. The system of claim 72, wherein the tracking component automatically determines the at least one location.

74. The system of claim 72, wherein the at least one sensor measures in real-time display device poses, and the tracking component automatically adapts to changes in the display device poses.

75. The system of claim 1, wherein at least one tag of a plurality of tags is affixed to at least one object of the plurality of objects.

76. The system of claim 75, wherein the plurality of sensors measure in real-time object poses of the at least one object using information of the at least one tag.

77. The system of claim 76, wherein the tracking component automatically adapts to changes in the object poses.

78. The system of claim 77, wherein the tracking component generates a model of a pose and a physical size of the at least one object.

79. The system of claim 77, wherein the at least one tag is affixed to at least one location on a periphery of the at least one object, wherein the at least one object is a display device.

80. The system of claim 79, wherein the tracking component automatically determines the at least one location.

81. The system of claim 79, wherein the plurality of sensors measure in real-time display device poses using information of the at least one tag.

82. The system of claim 81, wherein the tracking component automatically adapts to changes in the display device poses.

83. The system of claim 1, wherein at least one sensor of the plurality of sensors measures in real-time object poses of at least one object of the plurality of objects, wherein the at least one object is a marked object.

84. The system of claim 83, wherein the marked object is marked using a tagged object, wherein the tagged object comprises a tag affixed to an object.

85. The system of claim 84, wherein the marked object is marked when the tagged object is placed in direct contact with at least one location on the at least one object.

86. The system of claim 85, wherein the at least one location comprises a plurality of locations on the marked object.

87. The system of claim 86, wherein the plurality of sensors measure poses of the tagged object relative to the marked object and the SOE, wherein the poses of the tagged object sensed at the plurality of locations represent poses of the marked object.

88. The system of claim 84, wherein the marked object is marked when the tagged object is pointed at a plurality of locations on the at least one object.

89. The system of claim 88, wherein the plurality of sensors measure poses of the tagged object relative to the marked object and the SOE, wherein the poses of the tagged object represent poses of the marked object at points in time that correspond to when the tagged object is pointed at the plurality of locations.

90. The system of claim 1, comprising a plurality of tags affixed to the plurality of objects, wherein a tag comprises a linear-partial-tag (LPT) that includes a plurality of collinear markers, wherein the plurality of collinear markers convey an identity of the tag.

91. The system of claim 90, wherein a tag comprises a plurality of LPTs, wherein each LPT includes a plurality of collinear markers.

92. The system of claim 91, wherein a tag comprises a first LPT positioned on a substrate adjacent to a second LPT, wherein the first LPT includes a first set of collinear markers and the second LPT includes a second set of collinear markers.

93. The system of claim 92, wherein the plurality of sensors comprise at least one camera, and the feature data comprises a projective image acquired by the at least one camera, wherein the projective image includes the tag.

94. The system of claim 93, comprising searching the projective image and identifying the first LPT in the projective image, and fitting a line to the first set of collinear markers of the first LPT.

95. The system of claim 94, comprising computing a cross ratio of the first set of collinear markers, wherein the cross ratio is a function of pairwise distances between the plurality of collinear markers of the first set of collinear markers, and comparing the cross ratio to a set of cross ratios that correspond to a set of known LPTs.

96. The system of claim 95, comprising searching the projective image and identifying the second LPT, and combining the first LPT and the second LPT into a tag candidate, and computing a set of pose hypotheses corresponding to the tag candidate, and computing a confidence metric that is a re-projection error of a pose of the set of pose hypotheses.

97. The system of claim 93, wherein the at least one camera collects correspondence data between image coordinates of the projective image and the plurality of collinear markers.

98. The system of claim 97, comprising a camera calibration application, wherein intrinsic parameters of the at least one camera are modeled using the camera calibration application, wherein the intrinsic parameters include at least one of focal ratio, optical center, skewness, and lens distortion.

99. The system of claim 98, wherein an input to the camera calibration application includes the correspondence data.