Detecting, representing, and interpreting three-space input: gestural continuum subsuming freespace, proximal, and surface-contact modes

• US 9,779,131 B2
• Filed: 05/13/2014
• Issued: 10/03/2017
• Est. Priority Date: 04/24/2008
• Status: Active Grant

• Current Assignee: Oblong Industries, Inc. (Oblong, Inc.)
• Original Assignee: Oblong Industries, Inc. (Oblong, Inc.)
• Inventors: Underkoffler, John S., Kramer, Kwindla Hultman
• Primary Examiner(s): Shankar, Vijay
• Application Number:
  US14/276,093
• Publication Number:
  US 20150100593A1
• Time in Patent Office: 1,239 Days
• Field of Search: 345156-178, 178 1801- 1809, 178 2001- 2004
• CPC Class Codes: G06F 16/244 Grouping and aggregation
• CPC Class Codes: G06F 2203/04108 Touchless 2D- digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface without distance measurement in the Z direction
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06F 3/041 Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
• CPC Class Codes: G06F 40/205 Parsing
• CPC Class Codes: G06V 10/245 by locating a pattern; Special marks for positioning
• CPC Class Codes: G06V 10/94 Hardware or software architectures specially adapted for image or video understanding
• CPC Class Codes: G06V 40/107 Static hand or arm

Abstract

Systems and methods for detecting, representing, and interpreting three-space input are described. Embodiments of the system, in the context of an SOE, process low-level data from a plurality of sources of spatial tracking data and analyze these semantically uncorrelated spatiotemporal data and generate high-level gestural events according to dynamically configurable implicit and explicit gesture descriptions. The events produced are suitable for consumption by interactive systems, and the embodiments provide one or more mechanisms for controlling and effecting event distribution to these consumers. The embodiments further provide to the consumers of its events a facility for transforming gestural events among arbitrary spatial and semantic frames of reference.

Claims

1. A method comprising:

collating input data from a plurality of sources, wherein the input data includes data of a state of an object in a frame of reference of the object;

conforming the input data into a stream of spatiotemporal data;

generating data events from the spatiotemporal data using a plurality of event descriptions;

representing the data events in a data format that is application-neutral; and

distributing the data events and providing access to the data events by at least one event consumer in a spatial-semantic frame of reference of the at least one event consumer.

2. The method of claim 1, wherein the input data comprises data generated by the object.

3. The method of claim 1, wherein the input data comprises proximal gestural data of the object when the object is at least one of within a proximate range relative to a surface and within a defined volume.

4. The method of claim 1, wherein the input data comprises hover gestural data of the object when the object is within a plane adjacent to a surface.

5. The method of claim 1, wherein the input data comprises surface-contact gestural data of the object when the object is in contact with a surface.

6. The method of claim 1, wherein the input data comprises a plurality of data streams.

7. The method of claim 6, comprising temporally aligning the plurality of data streams.

8. The method of claim 6, comprising spatially seaming the data events from the plurality of data streams and generating a single synthetic event.

9. The method of claim 6, comprising performing semantic aggregation including collecting relevant data events resulting from preceding operations.

10. The method of claim 6, comprising performing metainformation tagging.

11. The method of claim 6, comprising receiving the input data from at least one of an input/output device, a touchscreen device, a sensor, a detector, an imager, and a tracking system.

12. The method of claim 1, wherein the plurality of sources include an input/output device.

13. The method of claim 1, wherein the plurality of sources include a touchscreen device.

14. The method of claim 1, wherein the plurality of sources include a sensor.

15. The method of claim 1, wherein the plurality of sources include a detector.

16. The method of claim 1, wherein the plurality of sources include an imager.

17. The method of claim 1, wherein the plurality of sources include a tracking system.

18. The method of claim 1, comprising receiving the spatiotemporal data comprising position of the object.

19. The method of claim 1, comprising receiving the spatiotemporal data comprising orientation of the object.

20. The method of claim 1, comprising receiving the spatiotemporal data comprising motion of the object.

21. The method of claim 1, comprising receiving the spatiotemporal data comprising position of at least one of a plurality of elements comprising the object and a plurality of elements coupled to the object.

22. The method of claim 1, comprising receiving the spatiotemporal data comprising orientation of at least one of a plurality of elements comprising the object and a plurality of elements coupled to the object.

23. The method of claim 1, comprising receiving the spatiotemporal data comprising motion of at least one of a plurality of elements comprising the object and a plurality of elements coupled to the object.

24. The method of claim 1, comprising receiving the spatiotemporal data comprising data events of at least one of a plurality of elements comprising the object and a plurality of elements coupled to the object.

25. The method of claim 1, comprising comparing the spatiotemporal data to the plurality of event descriptions.

26. The method of claim 25, comprising representing the data events in a capsule generated in response to a match between spatiotemporal data and an event description, wherein the capsule includes a data format comprising a digest of matched spatiotemporal data, interpreted in a semantic context of matched event descriptions.

27. The method of claim 1, comprising executing a plurality of recognizers, wherein each recognizer comprises an event description.

28. The method of claim 27, comprising performing a plurality of operations on the plurality of recognizers, wherein the plurality of operations comprise ranking recognizers.

29. The method of claim 27, comprising performing a plurality of operations on the plurality of recognizers, wherein the plurality of operations comprise adding recognizers.

30. The method of claim 27, comprising performing a plurality of operations on the plurality of recognizers, wherein the plurality of operations comprise removing recognizers.

31. The method of claim 27, comprising performing a plurality of operations on the plurality of recognizers, wherein the plurality of operations comprise modifying recognizers.

32. The method of claim 27, comprising performing a plurality of operations on the plurality of recognizers, wherein the plurality of operations comprise reconfiguring recognizers.

33. The method of claim 27, wherein a recognizer remains dormant prior to a match between spatiotemporal data and activation criteria of the recognizer.

34. The method of claim 33, wherein a recognizer becomes active when parameters of the spatiotemporal data match the activation criteria.

35. The method of claim 34, wherein the recognizer remains active as long as the spatiotemporal data satisfy maintenance criteria of the recognizer.

36. The method of claim 34, wherein the recognizer becomes inactive when the spatiotemporal data fail to satisfy the maintenance criteria.

37. The method of claim 1, comprising representing the data events in capsules and depositing the capsules in at least one repository for access by the at least one event consumer.

38. The method of claim 37, comprising providing a list of the at least one event consumer.

39. The method of claim 38, comprising asynchronously transmitting each capsule generated to each of the at least one event consumers.

40. The method of claim 38, comprising synchronously transmitting each capsule generated to each of the at least one event consumers.

41. The method of claim 1, comprising transforming the data events among a plurality of spatial-semantic frames of reference corresponding to a plurality of event consumers.

42. The method of claim 1, comprising re-rendering the data events in a spatial-semantic frame of reference of the at least one event consumer.

43. The method of claim 1, comprising representing the data events in a capsule and generating the capsule by generating at least one data sequence comprising data representing the data event and state information of the data event, and forming the capsule to include the at least one data sequence, the capsule having a data structure comprising an application-independent representation of the at least one data sequence.

44. The method of claim 43, wherein the generating of the at least one data sequence comprises:

generating a first respective data set that includes first respective data representing the data event;

generating a second respective data set that includes second respective state information; and

forming a first data sequence to include the first respective data set and the second respective data set.

45. The method of claim 44, wherein the generating of the at least one data sequence comprises:

generating a first respective data set that includes first respective data representing the data event;

generating a second respective data set that includes second respective state information; and

forming a second data sequence to include the first respective data set and the second respective data set.

46. The method of claim 45, wherein the generating of the first respective data set includes generating a first respective data set offset, wherein the first respective data set offset points to the first respective data set of the second data sequence.

47. The method of claim 45, wherein the generating of the second respective data set includes generating a second respective data set offset, wherein the second respective data set offset points to the second respective data set of the second data sequence.

48. The method of claim 44, wherein the first respective data set is a description list, the description list including a description of the data.

49. The method of claim 43, comprising:

generating at least one offset; and

forming the capsule to include the at least one offset.

50. The method of claim 49, comprising:

generating a first offset having a first variable length;

wherein the first offset points to the data representing the data event of a first data sequence of the at least one data sequence.

51. The method of claim 49, comprising:

generating a second offset having a second variable length;

wherein the second offset points to the state information of a first data sequence of the at least one data sequence.

52. The method of claim 49, comprising:

forming a first code path through the capsule using a first offset of the at least one offset;

forming a second code path through the capsule using a second offset of the at least one offset;

wherein the first code path and the second code path are different paths.

53. The method of claim 49, wherein at least one of the first offset and the second offset include metadata, the metadata comprising context-specific metadata.

54. The method of claim 1, representing the data events in a capsule comprising the data format and distributing the data events and providing access to the data events via corresponding capsules, wherein the at least one event consumer is at least one interactive system of a plurality of interactive systems, wherein the plurality of interactive systems comprise a plurality of frames of reference.

55. The method of claim 54, wherein the at least one event consumer consumes the capsules using an application type specific to the at least one event consumer.

56. The method of claim 55, wherein the at least one event consumer comprises a first interactive system having a first frame of reference and a second interactive system having a second frame of reference.

57. The method of claim 56, wherein the first interactive system consumes the capsules using a first application type and the second interactive system consumes the capsules using a second application type.