Detecting, representing, and interpreting three-space input: gestural continuum subsuming freespace, proximal, and surface-contact modes

• US 10,521,021 B2
• Filed: 02/01/2019
• Issued: 12/31/2019
• Est. Priority Date: 04/24/2008
• Status: Active Grant

• Current Assignee: Oblong Industries, Inc. (Oblong, Inc.)
• Original Assignee: Oblong Industries, Inc. (Oblong, Inc.)
• Inventors: Underkoffler, John S., Kramer, Kwindla Hultman
• Primary Examiner(s): Shankar, Vijay
• Application Number:
  US16/265,798
• Publication Number:
  US 20190187801A1
• Time in Patent Office: 333 Days
• CPC Class Codes: G06F 16/244 Grouping and aggregation
• CPC Class Codes: G06F 2203/04108 Touchless 2D- digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface without distance measurement in the Z direction
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06F 3/041 Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
• CPC Class Codes: G06F 40/205 Parsing
• CPC Class Codes: G06V 10/245 by locating a pattern; Special marks for positioning
• CPC Class Codes: G06V 10/94 Hardware or software architectures specially adapted for image or video understanding
• CPC Class Codes: G06V 40/107 Static hand or arm

Abstract

Systems and methods for detecting, representing, and interpreting three-space input are described. Embodiments of the system, in the context of an SOE, process low-level data from a plurality of sources of spatial tracking data and analyze these semantically uncorrelated spatiotemporal data and generate high-level gestural events according to dynamically configurable implicit and explicit gesture descriptions. The events produced are suitable for consumption by interactive systems, and the embodiments provide one or more mechanisms for controlling and effecting event distribution to these consumers. The embodiments further provide to the consumers of its events a facility for transforming gestural events among arbitrary spatial and semantic frames of reference.

Claims

1. A system comprising:

a data funnel;

a gesture engine coupled to the data funnel;

a gestural event consumer;

wherein the data funnel is constructed to;

receive a plurality of tracking data streams from a plurality of types of object tracking devices streaming low-level tracking events for a same object in real time,

temporally align the received low-level tracking events in real-time to generate a temporally-aligned multi-source tracking data stream that identifies a tracking event provided by two or more of the object tracking devices, and

semantically aggregate the temporally aligned low-level tracking events of multi-source tracking data stream,

wherein the gesture engine is constructed to generate at least one gesture event from the semantically aggregated low-level tracking events, and distribute at least one gesture event to the gestural event consumer.

2. The system of claim 1, wherein the gestural event consumer is constructed to update display at a display device.

3. The system of claim 2, further comprising the display device.

4. The system of claim 1, wherein the gestural event consumer is constructed to update state of an application.

5. The system of claim 4, further comprising at least one processor that executes the application.

6. The system of claim 1, wherein the low-level tracking events include events generated by using at least two different types of sensors.

7. The system of claim 1, wherein the low-level tracking events include events generated by using at least an image sensor and a motion sensor.

8. The system of claim 1, wherein the low-level tracking events include events generated by using at least an image sensor and a sensor that is different from an image sensor.

9. The system of claim 1, wherein at least two of the plurality of types of object tracking devices are user input devices.

10. A method comprising:

receiving a plurality of tracking data streams from a plurality of types of object tracking devices streaming low-level tracking events for a same object in real time;

temporally aligning the received low-level tracking events in real-time to generate a temporally-aligned multi-source tracking data stream that identifies a tracking event provided by two or more of the object tracking devices;

semantically aggregating the temporally aligned low-level tracking events of the multi-source tracking data stream;

with a gesture engine, generating at least one gesture event from the semantically aggregated low-level tracking events; and

distributing the at least one gesture event to at least one gestural event consumer.

11. The method of claim 10, wherein the at least one gestural event consumer updates display at a display device.

12. The method of claim 10, wherein the gestural event consumer updates state of an application.

13. The method of claim 10, wherein the low-level tracking events include events generated by using at least two different types of sensors.

14. The method of claim 10, wherein the low-level tracking events include events generated by using at least an image sensor and a motion sensor.

15. The method of claim 10, wherein the low-level tracking events include events generated by using at least an image sensor and a sensor that is different from an image sensor.