System and method for predicting a touch position of a pointer on a touch-enabled unit or determining a pointing direction in 3D space

• US 10,620,752 B2
• Filed: 06/08/2018
• Issued: 04/14/2020
• Est. Priority Date: 07/03/2017
• Status: Active Grant

• Current Assignee: Aptiv Technologies AG
• Original Assignee: Delphi Technologies LLC (BorgWarner Incorporated)
• Inventors: Barth, Alexander
• Primary Examiner(s): Lee, Nicholas J
• Application Number:
  US16/003,257
• Publication Number:
  US 20190004667A1
• Time in Patent Office: 676 Days
• Field of Search: None
• CPC Class Codes: G06F 2203/04108 Touchless 2D- digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface without distance measurement in the Z direction
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0418 for error correction or compensation, e.g. based on parallax, calibration or alignment
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

A system for predicting a touch position of a pointer on a touch-enabled unit includes a 3D imaging unit, a processing unit coupled to the 3D imaging unit, and a touch-enabled unit coupled to the processing unit. The 3D imaging unit is configured to monitor an interaction zone in front of the touch-enabled unit. The processing unit includes a prediction module that is configured to predict where a pointer that approaches the touch-enabled unit and is monitored by the 3D imaging unit will touch the touch-enabled unit, and a calibration module that is configured to generate at least one calibration parameter by comparing the predicted touch position to the actual touch position of the pointer detected by the touch-enabled unit and to transfer the at least one calibration parameter to the prediction module in order to calibrate the prediction module.

Claims

1. A system for predicting a touch position of a pointer on a touch-enabled unit, said system comprising:

a 3D imaging unit;

a processing unit coupled to the 3D imaging unit;

a touch-enabled unit coupled to the processing unit, wherein the 3D imaging unit is configured to monitor an interaction zone in front of the touch-enabled unit, and the processing unit comprises a prediction module that is configured to predict where a pointer that approaches the touch-enabled unit and is monitored by the 3D imaging unit will touch the touch-enabled unit; and

a calibration module that is configured to;

generate at least one calibration parameter by comparing the predicted touch position to the actual touch position of the pointer detected by the touch-enabled unit,

use a plurality of pairs for the generation of the at least one calibration parameter, wherein each of the pairs contains a predicted touch position and a corresponding actual touch position, and

transfer the at least one calibration parameter to the prediction module to calibrate the prediction module.

2. The system as claimed in claim 1, wherein the prediction module is configured to provide coordinates of the predicted touch position in a coordinate system associated with the 3D imaging unit and the touch-enabled unit is configured to provide coordinates of the actual touch position in a coordinate system associated with the touch-enabled unit, and wherein the at least one calibration parameter comprises translating and/or rotating and/or scaling parameters for at least one of the coordinate systems such that a deviation between the coordinates of the predicted touch position in the coordinate system associated with the 3D imaging unit and the coordinates of the actual touch position in the coordinate system associated with the touch-enabled unit is minimized.

3. The system as claimed in claim 1, wherein a 3D image of a scene captured by the 3D image unit comprises a 2D intensity image of the scene and a depth map of the scene, and wherein the at least one calibration parameter comprises an offset for the values of the depth map.

4. The system as claimed in claim 1, wherein the calibration module is configured to use only those pairs of the plurality of pairs for the generation of the at least one calibration parameter that contain a predicted touch position that was predicted when the distance of the pointer from the touch-enabled unit was in a predetermined distance range, and/or

wherein the calibration module is configured to use only those pairs of the plurality of pairs for the generation of the at least one calibration parameter that contain a predicted touch position that was predicted during a predetermined time interval before the pointer touched the touch-enabled unit.

5. The system as claimed in claim 1, wherein the calibration module is configured to weight each pair with a respective uncertainty factor before using the pairs for the generation of the at least one calibration parameter, wherein the uncertainty factor depends on the distance of the pointer from the touch-enabled unit when the prediction module predicted the touch position of the respective pair.

6. The system as claimed in claim 1, wherein the calibration module is configured to request a user to touch at least one predetermined position on the touch-enabled unit with the pointer, wherein the prediction module is configured to predict where the pointer that approaches the touch-enabled unit to touch the at least one predetermined position will touch the touch-enabled unit, and wherein the calibration module is configured to generate the at least one calibration parameter by comparing the predicted touch position to the at least one predetermined position.

7. The system as claimed in claim 1, comprising a user identification module that is configured to identify a user using the system, wherein the calibration module is configured to generate the at least one calibration parameter for a respective user and to store the at least one calibration parameter in a memory, wherein the at least one calibration parameter is transferred to the prediction module from the memory if the user identification module identifies the respective user using the system.

8. A system for determining a pointing direction of a pointer in 3D space, said system comprising:

a 3D imaging unit, and

a processing unit coupled to the 3D imaging unit, wherein the processing unit comprises a determination module that is configured to determine a pointing direction of a pointer of a user that is monitored by the 3D imaging unit when the user has the intention to point to an object, and a calibration module that is configured to generate at least one calibration parameter by comparing the determined pointing direction to the position of the object and to transfer the at least one calibration parameter to the determination module in order to calibrate the determination module, wherein the determination module is configured to determine the pointing direction of the pointer pointing to the same object for a plurality of times and to determine a peak in a distribution of the determined pointing directions, and the calibration module is configured to generate the at least one calibration parameter by comparing the pointing direction corresponding to the peak in the distribution of the determined pointing directions to the position of the object.

9. The system as claimed in claim 8, further comprising an input unit for inputting a command by the user, wherein the calibration module is configured to assume that the user has the intention to point to the object if the command input by the user in the input unit is related to the object.

10. The system as claimed in claim 8, wherein the calibration module is configured to request the user to point with the pointer to a predetermined object, wherein the determination module is configured to determine the pointing direction of the pointer when pointing to the predetermined object and wherein the calibration module is configured to generate the at least one calibration parameter by comparing the determined pointing direction to the position of the predetermined object.

11. The system as claimed in claim 8, wherein the 3D imaging unit is configured to capture consecutive images of a scene, wherein at least a first one of the images comprises a 2D intensity image and a depth map, and at least a second one of the images comprise only a 2D intensity image, but not a depth map, wherein the scene comprises a display showing an element and the determination module is configured to determine the coordinates of the element in a coordinate system associated with attached to the 3D imaging unit and the calibration module is configured to generate at least one further calibration parameter for calibrating the coordinate system associated with attached to the 3D imaging unit by comparing the coordinates of the element in the coordinate system associated with attached to the 3D imaging unit to predetermined coordinates of the element in a fixed coordinate system.

12. A system for determining a pointing direction of a pointer in 3D space, said system comprising:

a 3D imaging unit, and

a processing unit coupled to the 3D imaging unit, wherein the processing unit comprises a determination module that is configured to determine a pointing direction of a pointer of a user that is monitored by the 3D imaging unit when the user has the intention to point to an object, and a calibration module that is configured to generate at least one calibration parameter by comparing the determined pointing direction to the position of the object and to transfer the at least one calibration parameter to the determination module in order to calibrate the determination module;

wherein the 3D imaging unit is configured to capture consecutive images of a scene, wherein at least a first one of the images comprises a 2D intensity image and a depth map, and at least a second one of the images comprise only a 2D intensity image, but not a depth map, wherein the scene comprises a display showing an element and the determination module is configured to determine the coordinates of the element in a coordinate system associated with the 3D imaging unit and the calibration module is configured to generate at least one further calibration parameter for calibrating the coordinate system associated with the 3D imaging unit by comparing the coordinates of the element in the coordinate system associated with the 3D imaging unit (to predetermined coordinates of the element in a fixed coordinate system.

13. The system as claimed in claim 12, wherein the determination module is configured to determine the pointing direction of the pointer pointing to the same object for a plurality of times and to determine a peak in the distribution of the determined pointing directions, and the calibration module is configured to generate the at least one calibration parameter by comparing the pointing direction corresponding to the peak in the distribution of the determined pointing directions to the position of the object.

14. A system for predicting a touch position of a pointer on a touch-enabled unit, said system comprising:

a 3D imaging unit, wherein a 3D image of a scene captured by the 3D image unit comprises a 2D intensity image of the scene and a depth map of the scene;

a processing unit coupled to the 3D imaging unit;

a touch-enabled unit coupled to the processing unit, wherein the 3D imaging unit is configured to monitor an interaction zone in front of the touch-enabled unit, and the processing unit comprises a prediction module that is configured to predict where a pointer that approaches the touch-enabled unit and is monitored by the 3D imaging unit will touch the touch-enabled unit; and

a calibration module that is configured to generate at least one calibration parameter by comparing the predicted touch position to the actual touch position of the pointer detected by the touch-enabled unit and to transfer the at least one calibration parameter to the prediction module to calibrate the prediction module, and wherein the at least one calibration parameter comprises an offset for the values of the depth map.

15. The system as claimed in claim 14, wherein the calibration module is configured to use a plurality of pairs for the generation of the at least one calibration parameter, wherein each of the pairs contains a predicted touch position and a corresponding actual touch position.

16. The system as claimed in claim 14, wherein the prediction module is configured to provide coordinates of the predicted touch position in a coordinate system of the 3D imaging unit and the touch-enabled unit is configured to provide coordinates of the actual touch position in a coordinate system of the touch-enabled unit, and wherein the at least one calibration parameter comprises translating and/or rotating and/or scaling parameters for at least one of the coordinate systems attached to the 3D imaging unit and attached to the touch-enabled unit such that a deviation between the coordinates of the predicted touch position in the coordinate system of the 3D imaging unit and the coordinates of the actual touch position in the coordinate system of the touch-enabled unit is minimized.

17. A system for predicting a touch position of a pointer on a touch-enabled unit, said system comprising:

a 3D imaging unit;

a processing unit coupled to the 3D imaging unit;

a touch-enabled unit coupled to the processing unit, wherein the 3D imaging unit is configured to monitor an interaction zone in front of the touch-enabled unit, and the processing unit comprises a prediction module that is configured to predict where a pointer that approaches the touch-enabled unit and is monitored by the 3D imaging unit will touch the touch-enabled unit; and

a calibration module that is configured to generate at least one calibration parameter by comparing the predicted touch position to the actual touch position of the pointer detected by the touch-enabled unit and to transfer the at least one calibration parameter to the prediction module to calibrate the prediction module;

wherein the prediction module is configured to provide coordinates of the predicted touch position in a coordinate system of the 3D imaging unit and the touch-enabled unit is configured to provide coordinates of the actual touch position in a coordinate system of the touch-enabled unit, and wherein the at least one calibration parameter comprises translating and/or rotating and/or scaling parameters for at least one of the coordinate systems such that a deviation between the coordinates of the predicted touch position in the coordinate system of the 3D imaging unit and the coordinates of the actual touch position in the coordinate system of the touch-enabled unit is minimized.

18. The system as claimed in claim 17, wherein a 3D image of a scene captured by the 3D image unit comprises a 2D intensity image of the scene and a depth map of the scene, and wherein the at least one calibration parameter comprises an offset for the values of the depth map.

19. The system as claimed in claim 17, wherein the calibration module is configured to use a plurality of pairs for the generation of the at least one calibration parameter, wherein each of the pairs contains a predicted touch position and a corresponding actual touch position.

20. The system as claimed in claim 19, wherein the calibration module is configured to use only those pairs of the plurality of pairs for the generation of the at least one calibration parameter that contain a predicted touch position that was predicted according to at least one control parameter, wherein the at least one control parameter comprises the distance of the pointer from the touch-enabled unit being within a predetermined distance range or the predicted touch position being predicted during a predetermined time interval before the pointer touched the touch-enabled unit.