Human interface electronic device

• US 8,971,565 B2
• Filed: 05/28/2009
• Issued: 03/03/2015
• Est. Priority Date: 05/29/2008
• Status: Active Grant

• Current Assignee: Hie-D Technologies, LLC
• Original Assignee: Hie-D Technologies, LLC
• Inventors: Anzola, Carlos A.
• Primary Examiner(s): Akhavannik, Hadi
• Application Number:
  US12/473,890
• Publication Number:
  US 20090296991A1
• Time in Patent Office: 2,105 Days
• Field of Search: 382/100, 382/103, 382/104, 382/107, 345156-158, 345/204, 345/419, 345/633, 715/799, 715/863, 713/400, 348/154, 348/164, 348/E05.09, 356/27, 705/14.58, 455/456.1, 455/457, 463 30- 32, 463 36- 37
• US Class Current: 382/100
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]

Abstract

According to one general aspect, an apparatus configured to facilitate user gesture-based input may include an optical sensor, a plurality of infrared sensors, and processor. In various embodiments, the optical sensor may be configured to capture image data, within the visual spectrum. In some embodiments, the plurality of infrared sensors configured capture positional information regarding gestures made by a user. In one embodiment, the processor may be configured to receive the image data captured by the optical sensor and the positional information captured by the plurality of infrared sensors. In some embodiments, the processor may be configured to process the image data captured by the optical sensor. In various embodiments, the processor may be configured to determine, for each infrared sensor, a movement vector including coordinates that represents movement associated with the respective infrared sensor. In one embodiment, the apparatus may be configured to provide signals relating to the processed image data and the plurality of movement vectors to a host apparatus configured to be controlled, at least in part, by gestures made by the user.

Claims

1. An apparatus configured to facilitate user gesture-based input comprising:

an optical sensor configured to capture image data, within the visual spectrum;

a plurality of infrared sensors configured to capture positional information regarding gestures made by a user, positioned symmetrically on either side of the optical sensor each with an inclination configured to limit a field of view of the plurality of infrared sensors to the right and left sides of the user;

a processor configured to;

receive the image data captured by the optical sensor and the positional information captured by the plurality of infrared sensors,

process the image data captured by the optical sensor, and

determine, for each infrared sensor, a movement vector indicating a change in at least one of X, Y, and Z coordinates that represents movement associated with the respective infrared sensor; and

wherein the apparatus is configured to provide signals relating to the processed image data and the plurality of movement vectors to a host apparatus configured to be controlled, at least in part, by gestures made by the user.

2. The apparatus of claim 1, wherein the plurality of infrared sensors includes a plurality of sub-groups, each sub-group including at least one infrared sensor;

and

wherein each sub-group is configured to capture positional information regarding gestures made by a respective portion of the user.

3. The apparatus of claim 1, wherein the plurality of infrared sensors includes:

a first portion of the plurality of infrared sensors and a second portion of the plurality of infrared sensors,

wherein the first portion of the plurality of infrared sensors are positioned to capture positional information regarding gestures made by a first portion of the user, and

wherein the second portion of the plurality of infrared sensors are positioned to capture positional information regarding gestures made by a second portion of the user.

4. The apparatus of claim 1, wherein the optical sensor is configured to capture image data if an ambient light level is above a minimum ambient light level, wherein the minimum ambient light level is a value between 0 and 0.5 lux, inclusive.

5. The apparatus of claim 1, wherein the processor is configured to, for each infrared sensor:

triangulate a coordinate or at least one point of reflection based upon the positional information captured by the plurality of infrared sensors.

6. The apparatus of claim 1, wherein the processor is configured to produce:

a plurality of digital signals representing the movement vector associated with each respective infrared sensor;

an analog synchronized horizontal-vertical signal associated with the optical sensor;

a digital serial data signal associated with the optical sensor; and

a digital clock signal configured to synchronize at least the analog synchronized horizontal-vertical signal.

7. The apparatus of claim 1, wherein the optical sensor includes a charge-coupled device, and is configured to produce a visual angle including dimensions of substantially 75 degrees horizontal and substantially 55 degrees vertical;

wherein a first portion of the plurality of infrared sensors are positioned at a first side of the optical sensor and set-back by at least 15 degrees from a front of the apparatus;

wherein a second portion of the plurality of infrared sensors are positioned at a second side of the optical sensor and set-back by at least 15 degrees from the front of the apparatus;

wherein the processor is configured to produce a plurality of signals relating to the processed image data and the plurality of movement vectors; and

wherein the apparatus further includes;

a multiplexer configured to time division multiplex between the plurality of signals relating to the processed image data and the plurality of movement vectors, and

a host apparatus interface configured to communicate the multiplexed signals to the host apparatus.

21. The apparatus of claim 1, wherein the plurality of infrared sensors statically positioned.

8. A method to facilitate user gesture-based input comprising:

capturing, via an optical sensor, image data, within the visual spectrum;

capturing, via a plurality of positional sensors, positional information regarding gestures made by the user, a first portion and a second portion of the plurality of positional sensors positioned symmetrically on either side of the optical sensor each with an inclination configured to limit a field of view of the plurality of positional sensors to the right and left sides of the user;

processing the image data captured by the optical sensor;

determining, for each positional sensor, a movement vector indicating a change in at least one of X, Y, and Z coordinates that represents movement associated with the respective infrared sensor; and

providing signals relating to the processed image data and the plurality of movement vectors to a host apparatus configured to be controlled, at least in part, by gestures made by the user.

9. The method of claim 8, wherein capturing image data includes capturing image data if an ambient light level is above a minimum ambient light level, wherein the minimum ambient light level is a value between 0 and 0.5 lux, inclusive.

10. The method of claim 8, wherein capturing, via a plurality of positional sensors, includes:

capturing positional information regarding gestures made by a first portion of the user via a first portion of the plurality of positional sensors, and

capturing positional information regarding gestures made by a second portion of the user via a second portion of the plurality of positional sensors.

11. The method of claim 8, wherein determining, for each positional sensor, a movement vector includes:

triangulating a coordinate or at least one point of reflection based upon the positional information captured by the plurality of positional sensors.

12. The method of claim 8, wherein providing signals includes producing at least one of the following:

a plurality of digital signals representing the movement vector associated with each respective infrared sensor;

an analog synchronized horizontal-vertical signal associated with the optical sensor;

a digital serial data signal associated with the optical sensor; and

a digital clock signal configured to synchronize at least the analog synchronized horizontal-vertical signal.

13. The method of claim 8, wherein capturing image data includes:

capturing image data via a charge-coupled device, and

producing a visual angle including dimensions of substantially 75 degrees horizontal and substantially 55 degrees vertical;

wherein capturing positional information includes;

capturing positional information from a first group of positional sensors positioned on a first side of the optical sensor and set-back by at least 15 degrees from a front of the optical sensor, and

capturing positional information from a second group of positional sensors positioned on a second side of the optical sensor and set-back by at least 15 degrees from the front of the optical sensor; and

wherein providing includes;

time division multiplexing between the signals relating to the processed image data and the plurality of movement vectors, and

transmitting the multiplexed signals to the host apparatus.

14. A computer program product for facilitating user gesture-based input, the computer program product being tangibly embodied on a computer-readable medium and including executable code that, when executed, is configured to cause a host apparatus to

receive, from a sensor apparatus, a plurality of signals relating to visual image data captured by an optical sensor of the sensor apparatus;

receive, from the sensor apparatus, a plurality of movement vectors captured by a plurality infrared sensors of the sensor apparatus and produced in response to gestures made by a user the plurality of movement vectors indicating a change in X, Y, and Z coordinates associated with the gestures, a first portion and a second portion of the plurality of infrared sensors positioned symmetrically on either side of the optical sensor each with an inclination configured to limit a field of view of the plurality of infrared sensors to the right and left sides of the user;

based upon the received plurality of signals, identify at least one constellation of reference points, wherein each constellation is associated with a respective body part of the user;

track the movement of the body part(s) of the user, based at least in part upon the respective identified constellation(s); and

convert the movement of the respective body part(s) of the user into user interface commands.

15. The computer program product of claim 14, wherein the executable code configured to cause identifying at least one constellation of reference points also includes:

identifying the user;

determining the distance of the user from the sensor apparatus;

based at least in part upon the distance of the user from the sensor apparatus, determining a number of constellations of reference points to identify.

16. The computer program product of claim 14, wherein the executable code configured to cause identifying at least one constellation of reference points also includes:

identifying a constellation of reference points associated with a face of the user which includes;

identifying a reference point associated with a nose of the user,

identifying a pair of reference points associated with the eyes of the user, and

identifying at least one reference point associated with a mouth of the user.

17. The computer program product of claim 14, wherein the executable code configured to cause identifying at least one constellation of reference points also includes:

modeling a skin color of the user; and

limiting the identification of pixels, received via the plurality of received signals relating to visual image data, to substantially focus on pixels substantially colored with the skin color of the user.

18. The computer program product of claim 14, wherein the executable code configured to cause tracking the movement of the body part of the user also includes:

determining a deformation within a respective constellation of reference points;

measuring the gradient of the deformation; and

based upon the gradient of the deformation, determining a movement of the respective body part of the user.

19. The computer program product of claim 14, wherein the executable code configured to cause tracking the movement of the body part of the user also includes:

mapping the user'"'"'s surroundings;

registering static objects within the user'"'"'s surroundings;

periodically receiving, from a sensor apparatus, a plurality of signals relating to visual image data and a plurality of movement vectors produced in response to gestures made by a user; and

based upon the received signals, remapping the user'"'"'s surroundings to determine non-static objects.

20. The computer program product of claim 14, wherein the executable code configured to cause tracking the movement of the body part of the user also includes:

determining the speed of the movement of the body part of the user based upon a temporal derivative and a special derivative of a change in each pixel, received via the plurality of received signals relating to visual image data.

22. An apparatus configured to facilitate user gesture-based input comprising:

an optical sensor including a charge-coupled device, and is configured to;

capture image data within the visual spectrum, and

produce a visual angle having horizontal and vertical axis;

a plurality of infrared sensors including a configured to capture positional information regarding gestures made by a user, wherein

a first portion of the plurality of infrared sensors are positioned at a first side of the optical sensor and set-back from a front of the apparatus, and

a second portion of the plurality of infrared sensors are positioned at a second side of the optical sensor and set-back from the front of the apparatus;

a processor configured to;

receive the image data captured by the optical sensor and the positional information captured by the plurality of infrared sensors,

process the image data captured by the optical sensor, and

determine, for each infrared sensor, a movement vector indicating a change in at least one of X, Y, and Z coordinates that represents movement associated with the respective infrared sensor,

a multiplexer configured to time division multiplex between the plurality of signals relating to the processed image data and the plurality of movement vectors; and

a host apparatus interface configured to communicate the multiplexed signals to the host apparatus, wherein the apparatus is configured to provide signals relating to the processed image data and the plurality of movement vectors to the host apparatus configured to be controlled based on gestures made by the user.