Detecting user interactions with a computing system of a vehicle

• US 10,209,832 B2
• Filed: 07/25/2016
• Issued: 02/19/2019
• Est. Priority Date: 07/25/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Iyer, Rakesh Narayan, Onorato, Joseph
• Primary Examiner(s): Edun, Muhammad N
• Application Number:
  US15/218,873
• Publication Number:
  US 20180024695A1
• Time in Patent Office: 939 Days
• CPC Class Codes: B60K 2360/141 Activation of instrument input devices by approaching fingers or pens
• CPC Class Codes: B60K 2360/1438 Touch screens
• CPC Class Codes: B60K 35/00 Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
• CPC Class Codes: B60K 35/10 Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/03547 Touch pads, in which fingers can move on a surface
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: H04N 5/33 Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20

Abstract

A computing system of a vehicle for controlling a vehicle or systems within a vehicle. The computing system comprises a presence-sensitive panel within the vehicle, an infrared camera configured to capture images in a three-dimensional space within the vehicle, at least one processor; and at least one storage device that stores instructions. When the instructions are executed, they case the at least one processor to: receive, from the presence-sensitive panel, a first indication of input, receive, from the infrared camera, a second indication of input, and determine, based on at least one of the first indication of input or the second indication of input, an operation to be performed.

Claims

1. A computing system of a vehicle comprising:

a presence-sensitive panel positioned substantially horizontally on a central console of the vehicle;

an infrared camera configured to capture images in a three-dimensional space within the vehicle, the infrared camera being positioned under the presence-sensitive panel and being oriented to capture images in the three-dimensional space above the presence-sensitive panel;

at least one processor; and

at least one storage device that stores instructions that, when executed, cause the at least one processor to;

receive, from the presence-sensitive panel, a first indication of input,

receive, from the infrared camera, a second indication of input, and

determine, based on at least one of the first indication of input or the second indication of input, an operation to be performed.

2. The computing system of claim 1,

wherein the presence-sensitive panel is positioned between the infrared camera and the three-dimensional space and is transparent to infrared.

3. The computing system of claim 1,

wherein the presence-sensitive panel is configured to detect two-dimensional gesture inputs performed within a threshold distance of the presence-sensitive panel;

and

wherein the infrared camera is configured to detect three-dimensional gesture inputs performed beyond the threshold distance of an input surface of the presence-sensitive panel and in the three-dimensional space.

4. The computing system of claim 1, wherein determining the operation to be performed is based on both the first indication of input and the second indication of input.

5. The computing system of claim 1, wherein determining the operation to be performed is based on the first indication of input.

6. The computing system of claim 5, wherein the operation to be performed is a first operation to be performed, and wherein the instructions, when executed, further cause the at least one processor to:

responsive to determining the first operation to be performed, output, for display, information about the first operation to be performed;

determine, based on the second indication of input, a second operation to be performed; and

responsive to determining the second operation to be performed, output, for display, information about the second operation to be performed.

7. The computing system of claim 5, further comprising a display screen, wherein the operation to be performed is a first operation to be performed, and wherein the instructions, when executed, further cause the at least one processor to:

responsive to determining the first operation to be performed, display on the display screen information about the first operation to be performed;

determine, based on the second indication of input, a second operation to be performed; and

responsive to determining the second operation to be performed, display on the display screen information about the second operation to be performed.

8. The computing system of claim 5, further comprising a display screen, and wherein the instructions, when executed, further cause the at least one processor to:

responsive to determining the operation to be performed and prior to receiving the second indication of input, display on the display screen information requesting confirmation to perform the operation;

determine, based on the second indication of input, that the operation should be performed; and

perform the operation.

9. The computing system of claim 5,

wherein the three-dimensional space has high ambient light conditions, and

wherein determining the operation to be performed is based on the second indication of input and includes determining gestures from images captured by the infrared camera in the high ambient light conditions.

10. The computing system of claim 5,

wherein the three-dimensional space has low ambient light conditions, and

wherein determining the operation to be performed is based on the second indication of input and includes determining gestures from images captured by the infrared camera in the low ambient light conditions.

11. The computing system of claim 5,

wherein the three-dimensional space includes structure in a field of view of the infrared camera, and

wherein determining the operation to be performed is based on the second indication of input and includes discerning gestures from the structure in the field of view.

12. A method comprising:

receiving, by a computing system of a vehicle, from a presence-sensitive panel positioned substantially horizontally on a central console of the vehicle, a first indication of input;

receiving, by the computing system, from an infrared camera configured to capture images in a three-dimensional space within the vehicle, a second indication of input, wherein the infrared camera is positioned under the presence-sensitive panel and is oriented to capture images in the three-dimensional space above the presence-sensitive panel; and

determining, based on at least one of the first indication of input or the second indication of input, an operation to be performed.

13. The method of claim 12,

wherein the presence-sensitive panel is positioned between the infrared camera and the three-dimensional space and is transparent to infrared.

14. The method of claim 12,

wherein the presence-sensitive panel is configured to detect two-dimensional gesture inputs performed within a threshold distance of the presence-sensitive panel;

and

wherein the infrared camera is configured to detect three-dimensional gesture inputs performed beyond the threshold distance of an input surface of the presence-sensitive panel and in the three-dimensional space.

15. The method of claim 12, wherein determining the operation to be performed is based on both the first indication of input and the second indication of input.

16. The method of claim 12, wherein determining the operation to be performed is based on the first indication of input.

17. The method of claim 16, wherein the operation to be performed is a first operation to be performed, the method further comprising:

responsive to determining the first operation to be performed, outputting, for display, information about the first operation to be performed;

determining, based on the second indication of input, a second operation to be performed; and

responsive to determining the second operation to be performed, outputting, for display, information about the second operation to be performed.

18. A non-transitory computer-readable storage medium comprising instructions that, when executed by a computing system of a vehicle, cause at least one processor of the computing system to:

receive, from a presence-sensitive panel positioned substantially horizontally on a central console of the vehicle, a first indication of input;

receive, from an infrared camera configured to capture images in a three-dimensional space within the vehicle, a second indication of input, wherein the infrared camera is positioned under the presence-sensitive panel and is oriented to capture images in the three-dimensional space above the presence-sensitive panel; and

determine, based on at least one of the first indication of input or the second indication of input, an operation to be performed.