Portable remote control device enabling three-dimensional user interaction with at least one appliance

• US 8,773,512 B1
• Filed: 06/29/2012
• Issued: 07/08/2014
• Est. Priority Date: 06/30/2011
• Status: Active Grant

• Current Assignee: KAYA Dynamics LLC
• Original Assignee: Aquifi, Inc.
• Inventors: Rafii, Abbas
• Primary Examiner(s): Vaughn, Jr., William C
• Assistant Examiner(s): PEREZ FUENTES, LUIS M
• Application Number:
  US13/507,446
• Time in Patent Office: 739 Days
• US Class Current: 348/47
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: G02B 5/124 plural reflecting elements forming part of a unitary plate or sheet
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/042 by opto-electronic means

Abstract

A portable remote control device enables user interaction with an appliance by detecting user gestures made in a hover zone, and converting the gestures to commands that are wirelessly transmitted to the appliance. The remote control device includes at least two cameras whose intersecting FOVs define a three-dimensional hover zone within which user interactions are imaged. Separately and collectively image data is analyzed to identify a relatively few user landmarks. Substantially unambiguous correspondence is established between the same landmark on each acquired image, and a three-dimensional reconstruction is made in a common coordinate system. Preferably cameras are modeled to have characteristics of pinhole cameras, enabling rectified epipolar geometric analysis to facilitate more rapid disambiguation among potential landmark points. As a result processing overhead and latency times are substantially reduced. Landmark identification and position information is convertible into commands that alter the appliance behavior as intended by the user'"'"'s gesture.

Claims

1. A method to enable at least one user interaction made in a three-dimensional hover zone to be recognized by a hand holdable remote control device that creates a detectable event, corresponding to said interaction, useable to remotely control an appliance, the method including the following steps:

(a) providing said hand holdable device system with an acquisition system that includes at least a first camera have a first FOV to acquire at least a number N of data points of two-dimensional image information of at least a portion of said user in said FOV, each said camera being attached to said remote control device;

wherein said at least first camera has a characteristic selected from a group consisting of (i) said camera is a time-of-flight (TOF) camera that acquires three-dimensional image information of at least a portion of said user, (ii) said first camera includes a structured light camera system, and (iii) said camera is a two-dimensional camera that acquires two-dimensional image information of at least a portion of said user;

(b) processing said at least two-dimensional image information acquired from each said camera at step (a) so as to represent an imaged portion of said user using a set of landmark points sufficient to recognize a gesture made by said user, said processing discarding at least about 90% of said number N data points;

said landmark points being three-dimensional if said acquisition system includes at least one of (i) a TOF camera, (ii) at least two two-dimensional cameras, and (iii) a structure light system; and

said landmark points being two-dimensional if said acquisition system includes only one two-dimensional camera;

wherein said processing has at least one characteristic selected from a group consisting of (i) processing occurs within said remote control device, (ii) processing occurs externally to said remote control device, and (iii) at least a portion of said processing occurs within said remote control device and at least a portion of said processing occurs externally to said remote control device;

(c) translating each user gesture identified in step (b) to a command to control said appliance, wherein said translating has a characteristic selected from a group consisting of (i) translating occurs within said remote control device, (ii) translating occurs externally to said remote control device, and (iii) at least a portion of said translating occurs within said remote control device and at least a portion of said translating occurs externally to said remote control device; and

(d) transmitting each said user gesture translated in step (c) to said appliance to control at least one aspect of said appliance, wherein said transmitting has a characteristic selected from a group consisting of (i) transmitting occurs within said remote control device, (ii) transmitting occurs externally to said remote control device, and (iii) at least a portion of said transmitting occurs within said remote control device and at least a portion of said transmitting occurs externally to said remote control device.

2. The method of claim 1, wherein at step (a) said acquisition system includes a first two-dimensional camera having a first FOV and a second two-dimensional camera having a second FOV, each of said first and said second camera attached to said remote control device such that said three-dimensional hover zone is defined by an intersection of said first FOV and said second FOV.

3. The method of claim 2, wherein:

at step (a) said first camera has a first pixel sensor array with a first resolution, and said second camera has a second pixel sensor array with a second resolution, said first camera and said second camera being disposed on said remote control device;

at step (b) said processing extracts from said number N of data points a lesser number of three-dimensional landmarks, wherein magnitude of said three-dimensional landmarks has a characteristic selected from a group consisting of (i) said magnitude ≦

100, (ii) said magnitude ≦

50, and said magnitude ≦

12; and

wherein said second pixel sensor array has at least one characteristic selected from a group consisting of (i) said second resolution is substantially equal to said first resolution, (ii) said second resolution differs from said first resolution, (iii) wavelength sensitivity of said second pixel sensor array is substantially identical to wavelength sensitivity of said first pixel sensory array, (iv) wavelength sensitivity of said second pixel sensor array differs from wavelength sensitivity of said first pixel sensory array, (v) said second pixel sensor array is sensitive to RGB wavelengths, (vi) said second pixel sensor array is sensitive to gray scale wavelengths, (vii) said second pixel sensor array is sensitive to IR wavelengths, and (viii) wavelength sensitivity of said first pixel sensor array and wavelength of said second pixel sensor array are sensitive to IR wavelengths.

4. The method of claim 3, wherein processing at step (b) has at least one characteristic selected from a group consisting of (i) processing reduces acquired image information to ≦

1% N, (ii) processing reduces acquired information to ≦

0.1% N, and (iii) processing reduces acquired information to ≦

0.01% N.

5. The method of claim 2, wherein processing at step (b) includes at least one processing step having a characteristic selected from (i) said processing includes edge detection, (ii) said processing includes object modeling, (iii) said processing includes image rectification, (iv) said processing includes epipolar geometric analysis, and (v) said processing includes machine learning.

6. The method of claim 1, wherein said landmark has at least one characteristic selected from a group consisting of (i) said landmark is approximate centroid of said user'"'"'s head, (ii) said landmark is approximate centroid of said user'"'"'s hand, (iii), said landmark is approximate centroid of said user'"'"'s finger, (iv) said landmark is approximate location of said user'"'"'s fingertip, and (v) said landmark is approximate location of said user'"'"'s shoulder joint.

7. The method of claim 1, wherein:

said appliance includes an appliance display screen; and

said handholdable device includes a device display screen that replicates at least in part an image rendered by said appliance on said appliance display screen.

8. The method of claim 1, wherein said appliance has a characteristic selected from a group consisting of (i) said appliance includes a smart TV, (ii) said appliance includes a computer system, (iii) said appliance includes a set top box, (iv) said appliance includes a game console, (v) said appliance includes a kiosk, (vi) said appliance includes a signage unit, (vii) said appliance includes a lighting system, (viii) said appliance includes a locking system, and (ix) said appliance is remotely switchable between at least two states.

9. The method of claim 1, wherein said three-dimensional hover zone has at least one characteristic selected from a group consisting of (i) said zone includes a user-facing surface of said hand holdable remote control as well as a three-dimensional region extending outwardly from said user-facing surface of said hand holdable remote control device;

wherein said handholdable remote control device enables at least one user interaction having a characteristic selected from a group consisting of (i) said interaction includes a user gesture that physically contacts said user-facing surface of said hand holdable remote control device, (ii) said interaction includes a user gesture made in a region of said three-dimensional hover zone without physically touching said user-facing surface of said hand holdable remote control device, and (iii) said interaction includes user movement in said hover zone, said movement terminating at said user-facing surface of said hand holdable remote control device.

10. The method of claim 2, wherein at least said first two-dimensional camera includes an in-camera controller-processor;

and

wherein said means for processing can access said in-camera controller-processor to carry out at least one preliminary processing step towards reducing said number N of data points.

11. A handholdable remote control device to enable at least one user interaction, made in a three-dimensional hover zone, with an appliance, said interaction creating a detectable event useable by said appliance, where at least a portion of said user is representable by at least one three dimensional landmark, the handholdable remote control device including:

an acquisition system comprising at least a first two-dimensional camera having a first FOV and a first resolution, and a second two-dimensional camera having a second FOV and a second resolution, said first and second camera disposed on said remote control device such an intersection of said first FOV and second FOV defines said three-dimensional hover zone, said acquisition system acquiring at least a number N of data points of two-dimensional image information from at least a portion of said user in said three-dimensional hover zone;

means for processing said at least two-dimensional image information acquired from said acquisition system so as to represent an imaged portion of said user using a set of three-dimensional landmark points sufficient to recognize a gesture made by said user, said means for processing discarding at least about 90% of said number N data points;

wherein said means for processing has at least one characteristic selected from a group consisting of (i) said means for processing is disposed within said handholdable remote control device, and (ii) at least a portion of said means for processing is disposed external to said handholdable remote control device;

means for translating each user gesture recognized by said means for processing to a command to control said appliance, said means for translating disposed within said handholdable remote control device;

wherein said means for translating has a characteristic selected from a group consisting of (i) said means for translating is disposed within said handholdable remote control device, (ii) translating occurs externally to said remote control device, and (iii) at least a portion of said translating occurs within said remote control device and at least a portion of said translating occurs externally to said remote control device; and

a transmitting system to transmit each said user gesture translated by said means for translating to said appliance to control at least one aspect of said appliance, wherein said transmitting system has a characteristic selected from a group consisting of (i) said transmitting system is disposed within said handholdable remote control device, (ii) at least a portion of said transmitting system is disposed externally to said handholdable remote control device, and (iii) at least a portion of said transmitting system is disposed within said handholdable remote control device and at least a portion of said transmitting system is disposed externally to said handholdable remote control device.

12. The handholdable device of claim 11, wherein:

said first camera has a first pixel sensor array with a first resolution, and said second camera has a second pixel sensor array with a second resolution;

wherein said second pixel sensor array has at least one characteristic selected from a group consisting of (i) said second resolution is substantially equal to said first resolution, (ii) said second resolution differs from said first resolution, (iii) wavelength sensitivity of said second pixel sensor array is substantially identical to wavelength sensitivity of said first pixel sensory array, (iv) wavelength sensitivity of said second pixel sensor array differs from wavelength sensitivity of said first pixel sensory array, (v) said second pixel sensor array is sensitive to RGB wavelengths, (vi) said second pixel sensor array is sensitive to gray scale wavelengths, (vii) said second pixel sensor array is sensitive to IR wavelengths, and (viii) wavelength sensitivity of said first pixel sensor array and wavelength of said second pixel sensor array are sensitive to IR wavelengths.

13. The handholdable remote control device of claim 11, wherein:

said means for processing processes said at least two-dimensional image information acquired from each said camera so as to represent an imaged portion of said user using a set of landmark points sufficient to recognize a gesture made by said user, said means for processing discarding at least about 90% of said number N data points.

14. The handholdable remote control device of claim 11, wherein:

said means for processing processes said at least two-dimensional image information acquired from each said camera so as to represent an imaged portion of said user using a set of landmark points sufficient to recognize a gesture made by said user;

wherein said means for processing has at least one characteristic selected from a group consisting of (i) processing reduces acquired image information to ≦

1% N, (ii) processing reduces acquired information to ≦

0.1% N, and (iii) processing reduces acquired information to ≦

0.01% N.

15. The handholdable remote control device of claim 11, wherein:

said means for processing carries out at least one procedure selected from a group consisting of (i) processing includes edge detection, (ii) processing includes object modeling, (iii) processing includes image rectification, (iv) processing includes epipolar geometric analysis, and (v) processing includes machine learning.

16. The handholdable remote control device of claim 11, wherein said landmark has at least one characteristic selected from a group consisting of (i) said landmark is approximate centroid of said user'"'"'s head, (ii) said landmark is approximate centroid of said user'"'"'s hand, (iii), said landmark is approximate centroid of said user'"'"'s finger, (iv) said landmark is approximate location of said user'"'"'s fingertip, and (v) said landmark is approximate location of said user'"'"'s shoulder joint.

17. The handholdable remote control device of claim 11, wherein said appliance includes an appliance display screen;

and

said handholdable remote control device includes a device display screen that replicates at least in part an image rendered by said appliance on said appliance display screen.

18. The handholdable remote control device of claim 11, wherein said appliance has a characteristic selected from a group consisting of (i) said appliance includes a smart TV, (ii) said appliance includes a computer system, (iii) said appliance includes a set top box, (iv) said appliance includes a game console, (v) said appliance includes a kiosk, (vi) said appliance includes a signage unit, (vii) said appliance includes a lighting system, (viii) said appliance includes a locking system, and (ix) said appliance is remotely switchable between at least two states.

19. The handholdable remote control device of claim 11, wherein said three-dimensional hover zone has at least one characteristic selected from a group consisting of (i) said zone includes a user-facing surface of said hand handholdable remote control as well as a three-dimensional region extending outwardly from said user-facing surface of said handholdable remote control device;

wherein said handholdable remote control device enables at least one user interaction having a characteristic selected from a group consisting of (i) said interaction includes a user gesture that physically contacts said user-facing surface of said hand holdable remote control device, (ii) said interaction includes a user gesture made in a region of said three-dimensional hover zone without physically touching said user-facing surface of said handholdable remote control device, and (iii) said interaction includes user movement in said hover zone, said movement terminating at said user-facing surface of said handholdable remote control device.

20. The handholdable remote control device of claim 11, further including a feedback system to advise said user that a gesture has been recognized and subsequent control of said appliance has been successfully carried out.