Multi-modal gestural interface

• US 9,495,013 B2
• Filed: 03/25/2014
• Issued: 11/15/2016
• Est. Priority Date: 04/24/2008
• Status: Active Grant

• Current Assignee: Oblong Industries, Inc. (Oblong, Inc.)
• Original Assignee: Oblong Industries, Inc. (Oblong, Inc.)
• Inventors: Underkoffler, John S., Kramer, Kwindla Hultman
• Primary Examiner(s): Shankar, Vijay
• Application Number:
  US14/224,947
• Publication Number:
  US 20140298269A1
• Time in Patent Office: 966 Days
• Field of Search: 345156-178, 178 1801- 1809, 178 2001- 2004
• US Class Current: 1/1
• CPC Class Codes: G06F 18/285 Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06F 3/033 Pointing devices displaced or positioned by the user , e.g. mice, trackballs, pens or joysticks; Accessories therefor digitisers characterised by the transducing means G06F3/041
• CPC Class Codes: G06V 10/96 Management of image or video recognition tasks
• CPC Class Codes: G06V 40/107 Static hand or arm

Abstract

Systems and methods for detecting, representing, and interpreting three-space input are described. Embodiments of the system, in the context of an SOE, process low-level data from a plurality of sources of spatial tracking data and analyze these semantically uncorrelated spatiotemporal data and generate high-level gestural events according to dynamically configurable implicit and explicit gesture descriptions. The events produced are suitable for consumption by interactive systems, and the embodiments provide one or more mechanisms for controlling and effecting event distribution to these consumers. The embodiments further provide to the consumers of its events a facility for transforming gestural events among arbitrary spatial and semantic frames of reference.

Claims

1. A system comprising:

a data funnel coupled to a processor and configured to collate input data from a plurality of sources, wherein the input data is uncorrelated data of an object, wherein the data funnel is configured to conform the input data into a stream of spatiotemporal data;

a gesture engine configured to receive the spatiotemporal data from the data funnel, use a plurality of gesture descriptions to generate gestural events from the spatiotemporal data, and represent the gestural events in a protoevent comprising a data format that is application-neutral and fully articulated; and

a distributor configured to provide access to the gestural events by at least one event consumer via corresponding protoevents.

2. The system of claim 1, wherein the plurality of sources comprise disparate sources.

3. The system of claim 1, wherein the input data is semantically uncorrelated three-space data of an instantaneous spatial and geometric state of an object in a frame of reference of the object.

4. The system of claim 1, wherein the spatiotemporal data of the stream is uniformly represented.

5. The system of claim 1, wherein the data format of the protoevent is fully articulated.

6. The system of claim 1, wherein the corresponding protoevents are in a spatial-semantic frame of reference of the at least one event consumer.

7. The system of claim 1, wherein the at least one event consumer is at least one interactive system of a plurality of interactive systems, wherein the plurality of interactive systems comprise a plurality of frames of reference.

8. The system of claim 7, wherein the at least one event consumer consumes the protoevents using an application type specific to the at least one event consumer.

9. The system of claim 8, wherein the at least one event consumer comprises a first interactive system having a first frame of reference and a second interactive system having a second frame of reference.

10. The system of claim 9, wherein the first interactive system consumes the protoevents using a first application type and the second interactive system consumes the protoevents using a second application type.

11. The system of claim 1, wherein the object is a portion of a human hand.

12. The system of claim 1, wherein the input data comprises unconstrained freespace gestural data of the object.

13. The system of claim 1, wherein the input data comprises proximal gestural data of the object when the object is at least one of within a proximate range relative to a surface and within a defined volume.

14. The system of claim 1, wherein the input data comprises hover gestural data of the object when the object is within a space immediately adjacent to a surface.

15. The system of claim 1, wherein the input data comprises surface-contact gestural data of the object when the object is in contact with a surface.

16. The system of claim 1, wherein the input data comprises a plurality of data streams.

17. The system of claim 16, wherein the data funnel temporally aligns the plurality of data streams.

18. The system of claim 16, wherein the data funnel spatially seams events from the plurality of data streams and generates a single synthetic event.

19. The system of claim 16, wherein the data funnel performs semantic aggregation including collecting relevant events resulting from preceding operations of the data funnel.

20. The system of claim 16, wherein the data funnel performs metainformation tagging.

21. The system of claim 16, wherein the input data is received from at least one of an optical motion-tracking system, a time-of-flight tracking system, an electric field sensing system, and a touch screen device.

22. The system of claim 1, wherein the input data is received from an optical motion-tracking system.

23. The system of claim 1, wherein the input data is received from a time-of-flight tracking system.

24. The system of claim 1, wherein the input data is received from a touch screen device.

25. The system of claim 1, wherein the input data is received from an electric field sensing system.

26. The system of claim 1, wherein the input data is received from a capacitive sensing system.

27. The system of claim 1, wherein the gesture engine receives the spatiotemporal data comprising three-space position of the object.

28. The system of claim 1, wherein the gesture engine receives the spatiotemporal data comprising three-space orientation of the object.

29. The system of claim 1, wherein the gesture engine receives the spatiotemporal data comprising motion of the object.

30. The system of claim 1, wherein the gesture engine receives the spatiotemporal data comprising bulk three-space position of at least one of a plurality of elements comprising the object and a plurality of elements coupled to the object.

31. The system of claim 1, wherein the gesture engine receives the spatiotemporal data comprising bulk three-space orientation of at least one of a plurality of elements comprising the object and a plurality of elements coupled to the object.

32. The system of claim 1, wherein the gesture engine receives the spatiotemporal data comprising bulk motion of at least one of a plurality of elements comprising the object and a plurality of elements coupled to the object.

33. The system of claim 1, wherein the gesture engine receives the spatiotemporal data comprising a semantic digest of a pose of a plurality of elements comprising the object.

34. The system of claim 1, wherein the gesture engine compares the spatiotemporal data to the gesture descriptions.

35. The system of claim 34, wherein the gesture engine generates the protoevent in response to a match between spatiotemporal data and a gesture description, wherein the protoevent includes a data format comprising a digest of matched spatiotemporal data, interpreted in a semantic context of matched gestural descriptions.

36. The system of claim 1, wherein the gesture engine comprises a plurality of recognizers, wherein each recognizer comprises a gesture description.

37. The system of claim 36, wherein the gesture engine performs a plurality of operations on the plurality of recognizers, wherein the plurality of operations comprise ranking recognizers.

38. The system of claim 36, wherein the gesture engine performs a plurality of operations on the plurality of recognizers, wherein the plurality of operations comprise adding recognizers.

39. The system of claim 36, wherein the gesture engine performs a plurality of operations on the plurality of recognizers, wherein the plurality of operations comprise removing recognizers.

40. The system of claim 36, wherein the gesture engine performs a plurality of operations on the plurality of recognizers, wherein the plurality of operations comprise modifying recognizers.

41. The system of claim 36, wherein the gesture engine performs a plurality of operations on the plurality of recognizers, wherein the plurality of operations comprise reconfiguring recognizers.

42. The system of claim 36, wherein a recognizer remains dormant prior to a match between spatiotemporal data and activation criteria of the recognizer.

43. The system of claim 42, wherein a recognizer becomes active when geometric and spatiotemporal aspects of the spatiotemporal data match the activation criteria.

44. The system of claim 43, wherein the recognizer remains active as long as the spatiotemporal data satisfy maintenance criteria of the recognizer.

45. The system of claim 43, wherein the recognizer becomes inactive when the spatiotemporal data fail to satisfy the maintenance criteria.

46. The system of claim 1, wherein the distributor deposits the protoevents in at least one repository for access by the at least one event consumer.

47. The system of claim 46, wherein the distributor comprises a list of the at least one event consumer.

48. The system of claim 47, wherein the distributor asynchronously transmits each protoevent generated by the gesture engine to each of the at least one event consumers.

49. The system of claim 47, wherein the distributor synchronously transmits each protoevent generated by the gesture engine to each of the at least one event consumers.

50. The system of claim 1, comprising a transformer coupled to a remote client device of the at least one consumer, wherein the transformer re-renders the gestural events in a spatial-semantic frame of reference of the at least one event consumer.

51. The system of claim 50, wherein the distributor includes the transformer.

52. The system of claim 1, wherein the gesture engine generates the protoevent by generating at least one data sequence comprising gestural event data specifying the gestural event and state information of the gestural event, and forming a data capsule to include the at least one data sequence, the data capsule having a data structure comprising an application-independent representation of the at least one data sequence.

53. The system of claim 52, wherein the generating of the at least one data sequence comprises:

generating a first respective data set that includes first respective gestural event data;

generating a second respective data set that includes second respective state information; and

forming a first data sequence to include the first respective data set and the second respective data set.

54. The system of claim 53, wherein the generating of the at least one data sequence comprises:

generating a first respective data set that includes first respective gestural event data;

generating a second respective data set that includes second respective state information; and

forming a second data sequence to include the first respective data set and the second respective data set.

55. The system of claim 54, wherein the generating of the first respective data set includes generating a first respective data set offset, wherein the first respective data set offset points to the first respective data set of the second data sequence.

56. The system of claim 54, wherein the generating of the second respective data set includes generating a second respective data set offset, wherein the second respective data set offset points to the second respective data set of the second data sequence.

57. The system of claim 53, wherein the first respective data set is a description list, the description list including a description of the data.

58. The system of claim 52, comprising:

generating at least one offset; and

forming the data capsule to include the at least one offset.

59. The system of claim 58, comprising:

generating a first offset having a first variable length;

wherein the first offset points to the gestural event data of a first data sequence of the at least one data sequence.

60. The system of claim 58, comprising:

generating a second offset having a second variable length;

wherein the second offset points to the state information of a first data sequence of the at least one data sequence.

61. The system of claim 58, comprising:

forming a first code path through the data capsule using a first offset of the at least one offset;

forming a second code path through the data capsule using a second offset of the at least one offset;

wherein the first code path and the second code path are different paths.

62. The system of claim 58, wherein at least one of the first offset and the second offset include metadata, the metadata comprising context-specific metadata.

63. The system of claim 1, wherein the object is a human hand.

64. The system of claim 1, wherein the object is at least one finger of a human hand.

65. The system of claim 1, wherein the object includes at least one human hand and at least one finger of a human hand.