MULTI-SENSOR BASED USER INTERFACE

• US 20170060254A1
• Filed: 03/03/2016
• Published: 03/02/2017
• Est. Priority Date: 03/03/2015
• Status: Active Grant

• Current Assignee: NVIDIA Corporation
• Original Assignee: NVIDIA Corporation
• Inventors: MOLCHANOV, Pavlo, GUPTA, Shalini, KIM, Kihwan, PULLI, Kari
• Granted Patent: US 10,168,785 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: B60R 11/04 Mounting of cameras operative during drive; Arrangement of controls thereof relative to the vehicle
• CPC Class Codes: G01S 13/42 Simultaneous measurement of distance and other co-ordinates indirect measurement G01S13/46
• CPC Class Codes: G01S 13/4454 phase comparisons monopulse, i.e. comparing the echo signals received by an interferometric antenna arrangement
• CPC Class Codes: G01S 13/584 adapted for simultaneous range and velocity measurements
• CPC Class Codes: G01S 7/0233 Avoidance by phase multiplex
• CPC Class Codes: G01S 7/0234 Avoidance by code multiplex
• CPC Class Codes: G01S 7/352 Receivers
• CPC Class Codes: G01S 7/415 Identification of targets based on measurements of movement associated with the target
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 18/25 Fusion techniques
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 3/082 modifying the architecture, e.g. adding, deleting or silencing nodes or connections
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

An apparatus and method for gesture detection and recognition. The apparatus includes a processing element, a radar sensor, a depth sensor, and an optical sensor. The radar sensor, the depth sensor, and the optical sensor are coupled to the processing element, and the radar sensor, the depth sensor, and the optical sensor are configured for short range gesture detection and recognition. The processing element is further configured to detect and recognize a hand gesture based on data acquired with the radar sensor, the depth sensor, and the optical sensor.

Claims

1. An apparatus for gesture detection and recognition, the apparatus comprising:

a processing element;

a radar sensor;

a depth sensor;

an optical sensor, wherein the radar sensor, the depth sensor, and the optical sensor are coupled to the processing element, and wherein the radar sensor, the depth sensor, and the optical sensor are configured for short range gesture detection and the processing element is configured to detect a hand gesture based on data acquired with the radar sensor, the depth sensor, and the optical sensor.

2. The apparatus as described in claim 1, wherein the processing element is configured to operate the radar sensor in an always-on mode during a period in which the depth sensor and the optical sensor are powered down.

3. The apparatus as described in claim 1, wherein the processing element is configured to activate the depth sensor and the optical sensor based on motion detection of the radar sensor being above a threshold.

4. The apparatus as described in claim 1, wherein the radar sensor, the depth sensor, and the optical sensor are a portion of a user interface device of a vehicle.

5. The apparatus as described in claim 1, wherein a portion of the processing element is configured to function as a deep neural network (DNN).

6. The apparatus as described in claim 5, wherein the DNN comprises two 3D convolutional layers and two fully-connected layers.

7. The apparatus as described in claim 1, wherein the radar sensor, the depth sensor, and the optical sensor are configured for gesture detection and recognition under low light conditions.

8. The apparatus as described in claim 1, wherein the processing element is a graphics processing unit (GPU).

9. The apparatus as described in claim 1, wherein the radar sensor, the depth sensor, and the optical sensor are configured for gesture detection and recognition within a range of one meter.

10. The apparatus as described in claim 1, wherein the hand gesture is a dynamic hand gesture and wherein further the processing element is configured to automatically determine the type of hand gesture performed and a command associated with the dynamic hand gesture.

11. A system for hand gesture detection, the system comprising:

a processor;

a first sensor;

a second sensor;

a third sensor, wherein the first sensor, the second sensor, and third sensor are coupled to the processor, and wherein the first sensor, the second sensor, and third sensor are configured for short range gesture detection and recognition and wherein further the processor is configured to detect and recognize a hand gesture based on data acquired with first sensor, the second sensor, and third sensor.

12. The system as described in claim 11, wherein the first sensor comprises a radar.

13. The system as described in claim 12, wherein the second sensor is a depth sensor.

14. The system as described in claim 13, wherein the third sensor is an optical sensor.

15. The system as described in claim 11, wherein the first sensor, the second sensor, and third sensor are a portion of a user interface device for use in a vehicle.

16. The system as described in claim 11, wherein a portion of the processor is configured to function as a deep neural network (DNN).

17. The system as described in claim 11, wherein the processor is a graphics processing unit (GPU).

18. A mobile apparatus comprising:

a processing element;

a radar sensor;

a depth sensor;

an optical sensor, wherein the radar sensor, the depth sensor, and the optical sensor are coupled to the processing element, and wherein the radar sensor, the depth sensor, and the optical sensor are configured for short range gesture detection and recognition and wherein further the processing element is configured to detect and recognize a hand gesture of a driver based on data received from the radar sensor, the depth sensor, and the optical sensor, and wherein the processing element is configured to automatically determine the type of gesture performed and a command associated with the hand gesture.

19. The mobile apparatus as described in claim 18, wherein the processing element is configured to function as a neural network.

20. The mobile apparatus as described in claim 18, wherein the processing element is a graphics processing unit (GPU).