System and method for monitoring 3D space in front of an output unit for the control of the output unit

• US 10,031,624 B2
• Filed: 12/05/2016
• Issued: 07/24/2018
• Est. Priority Date: 12/18/2015
• Status: Active Grant

• Current Assignee: Aptiv Technologies AG
• Original Assignee: Delphi Technologies, Inc. (BorgWarner Incorporated)
• Inventors: Barth, Alexander, Bando, Piotr, Hollemann, Roland
• Primary Examiner(s): Dharia, Prabodh M
• Application Number:
  US15/368,741
• Publication Number:
  US 20170192617A1
• Time in Patent Office: 596 Days
• Field of Search: 345156-184, 345441, 715765, 715769, 715856, 715711, 715773
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0416 Control or interface arrangements specially adapted for digitisers
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06V 40/107 Static hand or arm
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00

Abstract

A system for monitoring 3D space in front of an output unit for the control of the output unit comprises a 3D imaging unit, a processing and control unit coupled to the 3D imaging unit and an output unit comprising a display area and coupled to the processing and control unit. The 3D imaging unit is configured to monitor an interaction zone in front of the display area of the output unit. The processing and control unit is configured to predict where a finger of a hand that approaches the display area of the output unit and is monitored by the 3D imaging unit will touch the display area and to control the output unit to modify content displayed on the display area based on the predicted touch position.

Claims

1. A system for monitoring 3D space in front of an output unit for the control of the output unit, said system comprising:

a 3D imaging unit;

a processing and control unit coupled to the 3D imaging unit; and

an output unit comprising a display area, said output unit coupled to the processing and control unit, wherein

the 3D imaging unit is configured to monitor an interaction zone in front of the display area of the output unit,

the processing and control unit is configured to predict where a finger of a hand that approaches the display area of the output unit and is monitored by the three-dimensional imaging unit (3D imaging unit) will touch the display area, and to control the output unit to modify content displayed on the display area based on the predicted touch position,

the processing and control unit is configured to localize a finger-tip and a finger base of the finger and to fit a line through the finger-tip and the finger base,

the processing and control unit is configured to compute an intersection point of the line through the finger-tip and the finger base with the display area,

the processing and control unit comprises a two-dimensional map (2D map) representing the display area and comprising a plurality of cells, and the processing and control unit is configured to compute the intersection point of the line with the display area in consecutive time steps, wherein for each of the time steps at least one of the cells is identified corresponding to the computed intersection point and a numeric value assigned to the at least one identified cell is incremented, and

the processing and control unit is configured to find peaks in the 2D map for each of the time steps in order to identify the predicted touch position of the finger.

2. A system as claimed in claim 1, wherein the output unit is configured to display a plurality of elements on the display area, wherein at least one of the elements is highlighted and/or enlarged if the processing and control unit predicts that the finger of the hand will touch the at least one of the elements.

3. A system as claimed in claim 1, wherein the processing and control unit is configured to identify a hand with an exposed index finger in an image received from the 3D imaging unit.

4. A system as claimed in claim 1, wherein the processing and control unit is configured to compute an intersection line of a plane through the line and a camera origin of the 3D imaging unit with the display area, wherein, the predicted touch position of the finger lies on the intersection line (59).

5. A system as claimed in claim 1, wherein the processing and control unit is configured to compute a distance between the finger-tip and the predicted touch position, a velocity of the finger-tip, and the time when the finger-tip will touch the display area.

6. A system as claimed in claim 1, wherein the processing and control unit is configured to increment numeric values assigned to cells located within a certain distance around the at least one identified cell for each of the time steps, wherein the distance depends on an uncertainty of the computed intersection point.

7. A vehicle comprising a system as claimed claim 1, wherein the output unit comprises a human machine interface.

8. A method for monitoring three-dimensional space (3D space) in front of an output unit for the control of the output unit, said method comprising:

monitoring an interaction zone in front of a display area of the output unit;

predicting where a finger of a hand that approaches the display area of the output unit and is monitored will touch the display area; and

controlling the output unit to modify content displayed on the display area based on the predicted touch position, wherein

a finger-tip and a finger base of the finger are localized and a line is fitted through the finger-tip and the finger base,

an intersection point of the line through the finger-tip and the finger base with the display area is computed,

a two-dimensional map (2D map) representing the display area and comprising a plurality of cells is provided, and the intersection point of the line with the display area is computed in consecutive time steps, wherein for each of the time steps at least one of the cells is identified corresponding to the computed intersection point and a numeric value assigned to the at least one identified cell is incremented, and

peaks are identified in the 2D map for each of the time steps in order to identify the predicted touch position of the finger.

9. The method as claimed in claim 8, wherein the output unit displays a plurality of elements on the display area and at least one of the elements is highlighted and/or enlarged if it is predicted that the finger of the hand, will touch the at least one of the elements.

10. The method as claimed in claim 8, wherein an intersection line of a plane through the line and a camera origin of the 3D imaging unit with the display area is computed, wherein, in particular, the predicted touch position of the finger lies on the intersection line.