Systems and methods for controlling device operation according to hand gestures

• US 9,423,879 B2
• Filed: 06/27/2014
• Issued: 08/23/2016
• Est. Priority Date: 06/28/2013
• Status: Active Grant

• Current Assignee: Chia Ming Chen
• Original Assignee: Chia Ming Chen
• Inventors: Chen, Chia Ming
• Primary Examiner(s): Olson, Jason
• Assistant Examiner(s): SUBEDI, DEEPROSE D
• Application Number:
  US14/318,019
• Publication Number:
  US 20150002391A1
• Time in Patent Office: 788 Days
• Field of Search: 345/156, 348/47, 348/48
• US Class Current: 1/1
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: H04N 23/125 Colour sequential image capture, e.g. using a colour wheel
• CPC Class Codes: H04N 23/56 provided with illuminating means
• CPC Class Codes: H04N 23/61 based on recognised objects
• CPC Class Codes: H04N 23/63 by using electronic viewfinders

Abstract

Provided is a non-contact sensing device that comprises a sensor comprising a plurality of function key sensors. A function key sensor of the plurality of function key sensors has a field of view. The function key sensor is constructed and arranged to detect a hand gesture at the field of view and to generate a function key control signal in response to detecting the hand gesture at the field of view. A processor processes the function key control signal from the function key sensor and outputs a command to a remote apparatus in response to the processed control signal.

Claims

1. A non-contact sensing device, comprising:

a sensor comprising a plurality of function key sensors having a plurality of non-overlapping field of views (FOVs), each FOV corresponding to a different function key;

a function key sensor of the plurality of function key sensors constructed and arranged to detect a presence of a body part at in its field of view, and to generate a function key control signal in response to detecting the body part at the field of view, wherein the body part of a user activates a function key to which the FOV of the function key sensor corresponds without the user physically touching the function key;

a processor that receives and processes the function key control signal and outputs a command according to an assigned task of the function key sensor to a remote apparatus under control of the non-contact sensing device in response to the processed function key control signal, wherein the processor further stores data of the function key for identifying the function key and associating the function key with the assigned task of the function key sensor; and

a control spot generator that illuminates the non-overlapping FOVs corresponding to the function keys to visually distinguish the different function keys.

2. The non-contact sensing device of claim 1, further comprising one or more cameras that recognize a hand gesture, wherein the command is generated from a combination of a function key control signal corresponding to the function key sensor and the recognized hand gesture.

3. The non-contact sensing device of claim 1, wherein the sensor is a staring sensor comprising a detector array that includes a combination of the function key sensors and non-function key sensors, the staring sensor generating all image pixels of the detector array simultaneously.

4. The non-contact sensing device of claim 1, wherein the sensor is a scanning sensor that scans a portion of a field of view at a time.

5. The non-contact sensing device of claim 1, wherein the sensor includes a scan mirror that scans all function key sensors and only the non-function key sensors in the path of the scan to shorten the data acquisition time.

6. The non-contact sensing device of claim 1, wherein the sensor is constructed and arranged as an emissive mode sensor comprising a thermal sensor that collects thermal radiation emitted from the hand or other body part.

7. The non-contact sensing device of claim 1, wherein the sensor is constructed and arranged as a reflective mode sensor comprising a color sensor that collects color light reflected from a hand or other body part.

8. The non-contact sensing device of claim 1, wherein the control spot generator generates a control spot that is aligned with the field of view of the function key sensor, and wherein the function key sensor detects a hand or body part within the control spot.

9. The non-contact sensing device of claim 1, further comprising a beamsplitter positioned between the sensor and the control spot generator, wherein light beams outputted from the control spot generator directed at the beamsplitter coincide with the field of views of the function key sensors.

10. The non-contact sensing device of claim 1, wherein the function key corresponding to the function key sensor distinguished from other function key sensors is identified by positioning a ground truth target at the control spot among a plurality of control spots and collecting by the non-contact sensing device an image of the ground truth target, wherein a pixel or group of pixels at the sensor having a highest detector output is identified as the current function key.

11. The non-contact sensing device of claim 1, wherein the control spot generator comprises a white light emitting diode (LED), a control spot generator plate having a plurality of color filters, and a lens, wherein color light is generated from the color filters when the white LED illuminates, and wherein a plurality of control spots are generated.

12. The non-contact sensing device of claim 11, wherein each color control spot is aligned with a field of view of a function key sensor.

13. The non-contact sensing device of claim 1, wherein the control spot generator comprises a plurality of color LEDs, light pipes, a light pipe mounting plate, and a lens, wherein the color LEDs are placed at the input ends of light pipes and the output ends of light pipes are placed at the focal plane of the lens, thereby generating control spots of different colors that each illuminate a field of view of a different function key sensor, and wherein the light pipe plate holds the light pipes together at the focal plane of the lens.

14. The non-contact sensing device of claim 1, wherein the remote apparatus comprises a plurality of devices, and wherein the processor generates a device number for a device of the plurality of devices, each device number corresponding to a different hand gesture at a designated control spot, thereby allowing a user to choose what device to operate.

15. The non-contact sensing device of claim 1, wherein a hand gesture at a chosen control spot corresponding to a function key triggers a tracking mode in which a camera tracks hand gesture motion commands for controlling one or multiple apparatuses, wherein in the tracking mode all function keys except the chosen one become inactive, and the processor will not process signals from these function keys, and wherein when exiting the tracking mode, a stop hand gesture can be inserted into the chosen control spot, wherein the other function keys become active again.

16. The non-contact sensing device of claim 1, wherein the sensor is a color sensor comprising color filters on a rotating wheel in front of the sensor, wherein an image of a scene is taken for each color filter, and wherein function key pixels of color images are processed by the processor to determine if a skin color spectrum is detected.

17. The non-contact sensing device of claim 1, wherein the sensor is a color sensor comprising a color camera, and wherein function key pixels of color images are processed to determine if a skin color spectrum is detected.

18. The non-contact sensing device of claim 1, wherein the processor distinguishes the function key sensor from other function key sensors of the plurality of sensors from the function key positions stored in the processor during a function key identification calibration process.

19. The non-contact sensing device of claim 1 further comprising a head mounted display collocated with the sensor, the display providing visual information regarding an operation of the remote apparatus.

20. The non-contact sensing device of claim 19, wherein the remote apparatus is a drone, and wherein the non-contact sensing device is constructed and arranged to control the operation of the drone, and wherein the head mounted display displays a combination of flight information, onboard camera images, and other information regarding the operation of the drone.

21. The non-contact sensing device of claim 1, a camera captures image data corresponding to the hand gesture and the processor converts the captured image data into a cursor command signal that controls a cursor at a display.