Systems and methods of gestural interaction in a pervasive computing environment

• US 10,782,657 B2
• Filed: 02/19/2015
• Issued: 09/22/2020
• Est. Priority Date: 05/27/2014
• Status: Active Grant

• Current Assignee: Lmi Liquidating Co., LLC
• Original Assignee: Ultrahaptics Ip Two Limited (Ultraleap Holdings Ltd.)
• Inventors: Gordon, Robert Samuel, Sills, Maxwell, Durdik, Paul Alan
• Primary Examiner(s): Rosen, Elizabeth H
• Application Number:
  US14/625,635
• Publication Number:
  US 20150346701A1
• Time in Patent Office: 2,042 Days
• CPC Class Codes: G05B 15/02 electric
• CPC Class Codes: G05B 2219/2642 Domotique, domestic, home control, automation, smart house
• CPC Class Codes: H04L 12/2803 Home automation networks
• CPC Class Codes: H04L 12/2809 indicating that an appliance service is present in a home automation network monitoring functionality H04L43/0817; discovery or management thereof, e.g. service location protocol [SLP] or web services, H04L67/51

Abstract

The technology disclosed relates to selecting among devices room to interact with. It also relates operating a smart phone with reduced power consumption. It further relates to gesturally interacting with devices that lack gestural responsiveness. The technology disclosed also relates to distinguishing control gestures from proximate non-control gestures in a pervasive three dimensional (3D) sensory space. The technology disclosed further relates to selecting among virtual interaction modalities to interact with.

Claims

1. A method of selecting among devices to interact with, the method including:

capturing, using a smart phone equipped with a motion sensory control device that detects gestures made by a hand moving freely in at least three dimensions in a three dimensional (3D) sensory space, a series of temporally sequential images of the hand as it moves;

automatically detecting, by the smart phone, multiple heterogeneous external devices that (i) are external to the smart phone and the motion sensory control device and (ii) accept motion control commands;

detecting, in images of the hand as captured, a first gesture made in the 3D sensory space;

interpreting the detected first gesture as selecting a first selected external device from the multiple heterogeneous external devices;

establishing a communication channel with which motion control commands are exchanged between the smart phone and the first selected external device;

detecting a paradigm-setting gesture that identifies one of the first selected external device and a second selected external device; and

processing a subsequent gesture in the 3D sensory space to send a motion control command to the one of the first and second selected external devices identified by the paradigm-setting gesture.

2. The method of claim 1, further including creating data for display by the smart phone that identifies the heterogeneous external devices and interpreting the subsequent gesture made in the 3D sensory space as selecting the one of the first and second selected external devices identified from the data.

3. The method of claim 1, further including detecting one or more subsequent gestures, processing the subsequent gestures, thereby reducing bandwidth needed to send motion control information, and sending the gestures via a communication channel to the one of the first and second selected external devices.

4. The method of claim 1, wherein the detected paradigm-setting gesture sets a control paradigm, and wherein the method further comprises using the control paradigm to process one or more subsequent gestures to send to the one of the first and second selected external devices.

5. The method of claim 4, further including controlling on-screen responsiveness of the one of the first and second selected external devices to one or more subsequent gestures based at least on the control paradigm.

6. The method of claim 1, further including detecting a gesture in the 3D sensory space and automatically selecting a particular external device selected from among the multiple heterogeneous external devices that accepts motion control commands based at least on biometric information determined from a hand of a user while performing the gesture.

7. The method of claim 6, wherein the biometric information is determined from at least one of vein patterns, palm prints, and fingerprints of the hand.

8. The method of claim 1, further including:

detecting, by the smart phone equipped with a motion sensory control device that detects gestures in a three dimensional (3D) sensory space, a second gesture in the 3D sensory space and interpreting the detected second gesture as selecting the second selected external device from the multiple heterogeneous external devices; and

establishing a second communication channel with which motion control commands are exchanged between the smart phone and the second selected external device.

9. The method of claim 8, further including creating data for display by the smart phone that identifies the first selected external device and the second selected external device and interpreting a gesture as selecting one of the communication channel and the second communication channel.

10. The method of claim 1, further including controlling on-screen responsiveness of the one of the first and second selected external devices to subsequent gestures, based at least on the paradigm-setting gesture.

11. A non-transitory machine readable storage medium storing instructions which when executed by a processor perform:

capturing, using a smart phone equipped with a motion sensory control device that detects gestures made by a hand moving freely in at least three dimensions in a three dimensional (3D) sensory space, a series of temporally sequential images of the hand as it moves;

automatically detecting, by the smart phone, multiple heterogeneous external devices that (i) are external to the smart phone and the motion sensory control device and (ii) accept motion control commands;

detecting, in images of the hand as captured, a first gesture in the 3D sensory space;

interpreting the first detected gesture as selecting a first selected external device from the multiple heterogeneous external devices;

establishing a communication channel with which motion control commands are exchanged between the smart phone and the first selected external device;

detecting, by the smart phone equipped with a motion sensory control device that detects gestures in a three dimensional (3D) sensory space, a second gesture in the 3D sensory space;

interpreting the detected second gesture as selecting a second selected external device from the multiple heterogeneous external devices;

establishing a second communication channel with which motion control commands are exchanged between the smart phone and the second selected external device; and

detecting a paradigm-setting gesture that identifies one of the first and second selected external devices, and processing a subsequent gesture in the 3D sensory space to send a motion control command to one of the first and second selected external devices identified by the paradigm-setting gesture.

12. A system, including:

a processor;

an interface with a motion sensory control device that detects gestures in a three dimensional (3D) sensory space; and

a memory storing instructions which when executed by a processor perform;

automatically detecting multiple heterogeneous external devices that (i) are external to the system and (ii) accept motion control commands;

detecting, using the motion sensory control device that detects gestures made by a hand moving freely in at least three dimensions in a three dimensional (3D) sensory space, a first gesture in the 3D sensory space by capturing a series of temporally sequential images of the hand as it moves;

interpreting the detected first gesture as selecting a first selected external device from the multiple heterogeneous external devices;

establishing a communication channel with which motion control commands are exchanged between the system and the first selected external device;

detecting, using the motion sensory control device that detects gestures made by a hand moving freely in at least three dimensions in a three dimensional (3D) sensory space, a second gesture in the 3D sensory space by capturing a series of temporally sequential images of the hand as it moves captured using cameras of the motion sensory control device;

interpreting the detected second gesture as selecting a second selected external device from the multiple heterogeneous external devices;

establishing a second communication channel with which motion control commands are exchanged between the system and the second selected external device; and

detecting a paradigm-setting gesture that identifies one of the first and second selected external devices, and processing a subsequent gesture in the 3D sensory space to send a motion control command to one of the first and second selected external devices identified by the paradigm-setting gesture.