Detecting, representing, and interpreting three-space input: gestural continuum subsuming freespace, proximal, and surface-contact modes

• US 10,235,412 B2
• Filed: 08/25/2017
• Issued: 03/19/2019
• Est. Priority Date: 04/24/2008
• Status: Active Grant

• Current Assignee: Oblong Industries, Inc. (Oblong, Inc.)
• Original Assignee: Oblong Industries, Inc. (Oblong, Inc.)
• Inventors: Underkoffler, John S., Kramer, Kwindla Hultman
• Primary Examiner(s): Shankar, Vijay
• Application Number:
  US15/686,529
• Publication Number:
  US 20170351734A1
• Time in Patent Office: 571 Days
• CPC Class Codes: G06F 16/244 Grouping and aggregation
• CPC Class Codes: G06F 2203/04108 Touchless 2D- digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface without distance measurement in the Z direction
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06F 3/041 Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
• CPC Class Codes: G06F 40/205 Parsing
• CPC Class Codes: G06V 10/245 by locating a pattern; Special marks for positioning
• CPC Class Codes: G06V 10/94 Hardware or software architectures specially adapted for image or video understanding
• CPC Class Codes: G06V 40/107 Static hand or arm

Abstract

Systems and methods for detecting, representing, and interpreting three-space input are described. Embodiments of the system, in the context of an SOE, process low-level data from a plurality of sources of spatial tracking data and analyze these semantically uncorrelated spatiotemporal data and generate high-level gestural events according to dynamically configurable implicit and explicit gesture descriptions. The events produced are suitable for consumption by interactive systems, and the embodiments provide one or more mechanisms for controlling and effecting event distribution to these consumers. The embodiments further provide to the consumers of its events a facility for transforming gestural events among arbitrary spatial and semantic frames of reference.

Claims

1. A system comprising:

a spatiotemporal first data source;

a spatiotemporal second data source, wherein the first and second data sources are different types of data sources;

a gestural control system; and

at least one gestural event consumer,

wherein the gestural control system is constructed to;

receive spatial input data for an object from the first data source and the second data source;

collect, temporally align, and spatially seam the spatial input data received from the first data source and the second data source to form a single conformed spatiotemporal data stream;

generate gestural event data by matching the spatiotemporal data stream with gesture-describing criteria; and

distribute the generated gestural event data to the at least one gestural event consumer,

wherein the first data source is an optical tracking device and the second data source is a motion tracking device that is different from an optical tracking device.

2. The system of claim 1, wherein the generated gestural event data has an application-neutral data format.

3. The system of claim 1, wherein at least one of the spatiotemporal data sources is constructed to track the object within a real-world coordinate frame.

4. The system of claim 1, wherein each spatiotemporal data source is constructed to track the object within a real-world coordinate frame.

5. The system of claim 1, wherein the first data source provides spatiotemporal events to the gestural control system at a first rate and the second data source provides spatiotemporal events to the gestural control system at a second rate that is different form the first rate.

6. The system of claim 1, wherein the first data source provides the gestural control system with at least one estimate of accuracy of spatial qualities represented by the spatial input data provided by the first data source to the gestural control system.

7. The system of claim 1, wherein the second data source provides the gestural control system with at least one estimate of accuracy of spatial qualities represented by the spatial input data provided by the second data source to the gestural control system.

8. The system of claim 1, wherein the first data source provides the gestural control system with at least one estimate of accuracy of temporal qualities represented by the spatial input data provided by the first data source to the gestural control system.

9. A method comprising:

a gestural control system;

receiving spatial input data for an object from a spatiotemporal first data source and a spatiotemporal second data source, wherein the first and second data sources are different types of data sources;

temporally aligning and spatially seaming the spatial input data received from the first data source and the second data source to form a single spatiotemporal data stream;

generating gestural event data by matching the spatiotemporal data stream with gesture-describing criteria; and

distributing the generated gestural event data to at least one gestural event consumer,

wherein the first data source is an optical tracking device and the second data source is a motion tracking device that is different from an optical tracking device.

10. The method of claim 9, wherein the generated gestural event data has an application-neutral data format.

11. The method of claim 9, wherein at least one of the spatiotemporal data sources is constructed to track the object within a real-world coordinate frame.

12. The method of claim 9, wherein each spatiotemporal data source is constructed to track the object within a real-world coordinate frame.

13. The method of claim 9, wherein the first data source provides spatiotemporal events to the gestural control system at a first rate and the second data source provides spatiotemporal events to the gestural control system at a second rate that is different form the first rate.

14. The method of claim 13, wherein the first data source provides the gestural control system with at least one estimate of accuracy of spatial qualities represented by the spatial input data provided by the first data source to the gestural control system.

15. The method of claim 13, wherein the second data source provides the gestural control system with at least one estimate of accuracy of spatial qualities represented by the spatial input data provided by the second data source to the gestural control system.

16. The method of claim 13, wherein the first data source provides the gestural control system with at least one estimate of accuracy of temporal qualities represented by the spatial input data provided by the first data source to the gestural control system.

17. The method of claim 13, wherein the second data source provides the gestural control system with at least one estimate of accuracy of temporal qualities represented by the spatial input data provided by the second data source to the gestural control system.