Detecting, representing, and interpreting three-space input: gestural continuum subsuming freespace, proximal, and surface-contact modes

• US 8,681,098 B2
• Filed: 05/04/2010
• Issued: 03/25/2014
• Est. Priority Date: 04/24/2008
• Status: Expired due to Fees

• Current Assignee: Oblong Industries, Inc. (Oblong, Inc.)
• Original Assignee: Oblong Industries, Inc. (Oblong, Inc.)
• Inventors: Underkoffler, John S., Kramer, Kwindla Hultman
• Primary Examiner(s): Shankar, Vijay
• Application Number:
  US12/773,605
• Publication Number:
  US 20100281440A1
• Time in Patent Office: 1,421 Days
• Field of Search: 345156-178, 178 1801- 1811, 178 2001- 2004
• US Class Current: 345/158
• CPC Class Codes: G06F 18/285 Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06V 10/245 by locating a pattern; Special marks for positioning
• CPC Class Codes: G06V 10/87 using selection of the recognition techniques, e.g. of a classifier in a multiple classifier system
• CPC Class Codes: G06V 10/96 Management of image or video recognition tasks
• CPC Class Codes: G06V 40/107 Static hand or arm

Abstract

Systems and methods for detecting, representing, and interpreting three-space input are described. Embodiments of the system, in the context of an SOE, process low-level data from a plurality of sources of spatial tracking data and analyze these semantically uncorrelated spatiotemporal data and generate high-level gestural events according to dynamically configurable implicit and explicit gesture descriptions. The events produced are suitable for consumption by interactive systems, and the embodiments provide one or more mechanisms for controlling and effecting event distribution to these consumers. The embodiments further provide to the consumers of its events a facility for transforming gestural events among arbitrary spatial and semantic frames of reference.

Claims

1. A system comprising:

a data funnel coupled to a processor, wherein the data funnel collates input data from a plurality of sources, wherein the input data is semantically uncorrelated three-space data of an instantaneous spatial and geometric state of an object in a frame of reference of the object, wherein the plurality of sources comprise disparate sources, wherein the data funnel conforms the input data into a stream of spatiotemporal data, wherein the spatiotemporal data of the stream is uniformly represented;

a gesture engine coupled to the data funnel, wherein the gesture engine generates gestural events from the spatiotemporal data using a plurality of gesture descriptions, wherein the gesture engine represents the gestural events in a protoevent comprising a data format that is application-neutral and fully articulated; and

a distributor coupled to the gesture engine, wherein the distributor provides access to the gestural events by at least one event consumer via corresponding protoevents in a spatial-semantic frame of reference of the at least one event consumer.

2. The system of claim 1, wherein the input data comprises unconstrained freespace gestural data of the object.

3. The system of claim 1, wherein the input data comprises proximal gestural data of the object when the object is at least one of within a proximate range relative to a surface and within a defined volume.

4. The system of claim 1, wherein the input data comprises hover gestural data of the object when the object is within a space immediately adjacent to a surface.

5. The system of claim 1, wherein the input data comprises surface-contact gestural data of the object when the object is in contact with a surface.

6. The system of claim 1, wherein the input data comprises a plurality of data streams.

7. The system of claim 6, wherein the data funnel temporally aligns the plurality of data streams.

8. The system of claim 6, wherein the data funnel spatially seams events from the plurality of data streams and generates a single synthetic event.

9. The system of claim 6, wherein the data funnel performs semantic aggregation including collecting relevant events resulting from preceding operations of the data funnel.

10. The system of claim 6, wherein the data funnel performs metainformation tagging.

11. The system of claim 6, wherein the input data is received from at least one of an optical motion-tracking system, a time-of-flight tracking system, an electric field sensing system, and a touch screen device.

12. The system of claim 1, wherein the input data is received from an optical motion-tracking system.

13. The system of claim 1, wherein the input data is received from a time-of-flight tracking system.

14. The system of claim 1, wherein the input data is received from a touch screen device.

15. The system of claim 1, wherein the input data is received from an electric field sensing system.

16. The system of claim 1, wherein the input data is received from a capacitive sensing system.

17. The system of claim 1, wherein the gesture engine receives the spatiotemporal data comprising three-space position of the object.

18. The system of claim 1, wherein the gesture engine receives the spatiotemporal data comprising three-space orientation of the object.

19. The system of claim 1, wherein the gesture engine receives the spatiotemporal data comprising motion of the object.

20. The system of claim 1, wherein the gesture engine receives the spatiotemporal data comprising bulk three-space position of at least one of a plurality of elements comprising the object and a plurality of elements coupled to the object.

21. The system of claim 1, wherein the gesture engine receives the spatiotemporal data comprising bulk three-space orientation of at least one of a plurality of elements comprising the object and a plurality of elements coupled to the object.

22. The system of claim 1, wherein the gesture engine receives the spatiotemporal data comprising bulk motion of at least one of a plurality of elements comprising the object and a plurality of elements coupled to the object.

23. The system of claim 1, wherein the gesture engine receives the spatiotemporal data comprising a semantic digest of a pose of a plurality of elements comprising the object.

24. The system of claim 1, wherein the gesture engine compares the spatiotemporal data to the gesture descriptions.

25. The system of claim 24, wherein the gesture engine generates the protoevent in response to a match between spatiotemporal data and a gesture description, wherein the protoevent includes a data format comprising a digest of matched spatiotemporal data, interpreted in a semantic context of matched gestural descriptions.

26. The system of claim 1, wherein the gesture engine comprises a plurality of recognizers, wherein each recognizer comprises a gesture description.

27. The system of claim 26, wherein the gesture engine performs a plurality of operations on the plurality of recognizers, wherein the plurality of operations comprise ranking recognizers.

28. The system of claim 26, wherein the gesture engine performs a plurality of operations on the plurality of recognizers, wherein the plurality of operations comprise adding recognizers.

29. The system of claim 26, wherein the gesture engine performs a plurality of operations on the plurality of recognizers, wherein the plurality of operations comprise removing recognizers.

30. The system of claim 26, wherein the gesture engine performs a plurality of operations on the plurality of recognizers, wherein the plurality of operations comprise modifying recognizers.

31. The system of claim 26, wherein the gesture engine performs a plurality of operations on the plurality of recognizers, wherein the plurality of operations comprise reconfiguring recognizers.

32. The system of claim 26, wherein a recognizer remains dormant prior to a match between spatiotemporal data and activation criteria of the recognizer.

33. The system of claim 32, wherein a recognizer becomes active when geometric and spatiotemporal aspects of the spatiotemporal data match the activation criteria.

34. The system of claim 33, wherein the recognizer remains active as long as the spatiotemporal data satisfy maintenance criteria of the recognizer.

35. The system of claim 33, wherein the recognizer becomes inactive when the spatiotemporal data fail to satisfy the maintenance criteria.

36. The system of claim 1, wherein the distributor deposits the protoevents in at least one repository for access by the at least one event consumer.

37. The system of claim 36, wherein the distributor comprises a list of the at least one event consumer.

38. The system of claim 37, wherein the distributor asynchronously transmits each protoevent generated by the gesture engine to each of the at least one event consumers.

39. The system of claim 37, wherein the distributor synchronously transmits each protoevent generated by the gesture engine to each of the at least one event consumers.

40. The system of claim 1, comprising a transformer coupled to a remote client device of the at least one consumer, wherein the transformer re-renders the gestural events in a spatial-semantic frame of reference of the at least one event consumer.

41. The system of claim 40, wherein the distributor includes the transformer.

42. The system of claim 1, wherein the gesture engine generates the protoevent by generating at least one data sequence comprising gestural event data specifying the gestural event and state information of the gestural event, and forming a data capsule to include the at least one data sequence, the data capsule having a data structure comprising an application-independent representation of the at least one data sequence.

43. The system of claim 42, wherein the generating of the at least one data sequence comprises:

generating a first respective data set that includes first respective gestural event data;

generating a second respective data set that includes second respective state information; and

forming a first data sequence to include the first respective data set and the second respective data set.

44. The system of claim 43, wherein the generating of the at least one data sequence comprises:

generating a first respective data set that includes first respective gestural event data;

generating a second respective data set that includes second respective state information; and

forming a second data sequence to include the first respective data set and the second respective data set.

45. The system of claim 44, wherein the generating of the first respective data set includes generating a first respective data set offset, wherein the first respective data set offset points to the first respective data set of the second data sequence.

46. The system of claim 44, wherein the generating of the second respective data set includes generating a second respective data set offset, wherein the second respective data set offset points to the second respective data set of the second data sequence.

47. The system of claim 43, wherein the first respective data set is a description list, the description list including a description of the data.

48. The system of claim 42, comprising:

generating at least one offset; and

forming the data capsule to include the at least one offset.

49. The system of claim 48, comprising:

generating a first offset having a first variable length;

wherein the first offset points to the gestural event data of a first data sequence of the at least one data sequence.

50. The system of claim 48, comprising:

generating a second offset having a second variable length;

wherein the second offset points to the state information of a first data sequence of the at least one data sequence.

51. The system of claim 48, comprising:

forming a first code path through the data capsule using a first offset of the at least one offset;

forming a second code path through the data capsule using a second offset of the at least one offset;

wherein the first code path and the second code path are different paths.

52. The system of claim 48, wherein at least one of the first offset and the second offset include metadata, the metadata comprising context-specific metadata.

53. The system of claim 1, wherein the at least one event consumer is at least one interactive system of a plurality of interactive systems, wherein the plurality of interactive systems comprise a plurality of frames of reference.

54. The system of claim 53, wherein the at least one event consumer consumes the protoevents using an application type specific to the at least one event consumer.

55. The system of claim 54, wherein the at least one event consumer comprises a first interactive system having a first frame of reference and a second interactive system having a second frame of reference.

56. The system of claim 55, wherein the first interactive system consumes the protoevents using a first application type and the second interactive system consumes the protoevents using a second application type.

57. The system of claim 1, wherein the object is a human hand.

58. The system of claim 1, wherein the object is at least one finger of a human hand.

59. The system of claim 1, wherein the object includes at least one human hand and at least one finger of a human hand.

60. A system comprising:

a data funnel coupled to a processor, wherein the data funnel collates input data from a plurality of sources and conforms the input data into a spatiotemporal data stream, wherein the input data is absolute three-space location data of an instantaneous state of a body at a point in time and space;

a gesture engine coupled to the data funnel, wherein the gesture engine generates gestural events from the spatiotemporal data stream using a plurality of gesture descriptions, wherein the gesture engine represents each gestural event in a protoevent comprising a data format that is application-neutral; and

a distributor coupled to the gesture engine, wherein the distributor provides access to the gestural events through access by a plurality of event consumers to a plurality of protoevents, wherein the access to the gestural events is in a spatial-semantic frame of reference of the plurality of event consumers.