Multi-modal touch screen emulator

• US 9,201,500 B2
• Filed: 09/28/2012
• Issued: 12/01/2015
• Est. Priority Date: 09/28/2012
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Srinivasan, Sadagopan, Illikkal, Rameshkumar, Iyer, Ravishankar
• Primary Examiner(s): NGUYEN, JENNIFER T
• Application Number:
  US13/631,309
• Publication Number:
  US 20140092014A1
• Time in Patent Office: 1,159 Days
• Field of Search: 345156-158, 345/173, 715/781, 715/863
• US Class Current: 1/1
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042

Abstract

Systems and methods may provide for capturing a user input by emulating a touch screen mechanism. In one example, the method may include identifying a point of interest on a front facing display of the device based on gaze information associated with a user of the device, identifying a hand action based on gesture information associated with the user of the device, and initiating a device action with respect to the front facing display based on the point of interest and the hand action.

Claims

1. A method comprising:

receiving a first image from a first front facing camera of a device having a front facing display;

identifying one or more eyes of a user of the device based on the first image;

determining a gaze location of the one or more eyes on the front facing display;

determining a change in the gaze location;

receiving a second image from a second front facing camera of the device;

identifying a point of interest on the front facing display based on gaze information, wherein the gaze information includes the gaze location and the change in the gaze location;

identifying a hand of the user based on the second image;

identifying one or more fingertips associated with the hand of the user based on the second image;

determining a movement of one or more of the hand and the one or more fingertips;

identifying a hand action based on gesture information, wherein the gesture information is to include the movement and the movement is one or more of a one-dimensional (1D) movement, a two-dimensional (2D) movement and a three-dimensional (3D) movement; and

initiating a device action with respect to the front facing display based on the point of interest and the hand action, wherein the device action is to emulate a multi-modal touchscreen function with respect to the front facing display that concurrently tracks and synchronizes the gaze information and the gesture information.

2. The method of claim 1, further including classifying the movement as the device action, wherein the device action includes one or more of a single click operation, a double click operation, a pinch operation and a cursor movement operation.

3. The method of claim 1, wherein initiating the device action includes sending a request to a device driver, wherein the device driver includes one or more of a mouse driver, a touchpad driver and a trackball driver.

4. The method of claim 1, further including:

synchronizing the gaze information with the gesture information; and

logging one or more of the gaze information and the gesture information.

5. A computer readable storage medium comprising a set of instructions which, if executed by a processor, cause a device to:

identify one or more eyes of a user of a device having a front facing display;

determine a gaze location of the one or more eyes on the front facing display;

identify a point of interest on the front facing display based on gaze information associated with the user of the device;

determine a change in the gaze location, wherein the gaze information is to include the gaze location and a change in the gaze location;

identify a hand action based on gesture information associated with the user of the device;

initiate a device action with respect to the front facing display based on the point of interest and the hand action, wherein the device action is to emulate a multi-modal touchscreen function with respect to the front facing display based on the first image and the second image, and wherein the multi-modal touchscreen function is to concurrently track and synchronize the gaze information and the gesture information;

receive image data from a front facing camera configuration of the device;

identify the gaze information and the gesture information based on the image data;

receive a first image from a first camera of the camera configuration; and

receive a second image from a second camera of the camera configuration, wherein the image data is to include the first image and the second image, the gaze information is to be determined based on the first image, and the gesture information is to be determined based on the second image.

6. The medium of claim 5, wherein the instructions, if executed, cause the device to:

identify a hand of the user;

identify one or more fingertips associated with the hand of the user; and

determine a movement of one or more of the hand and the one or more fingertips, wherein the gesture information is to include the movement and the movement is to be one or more of a one-dimensional (1D) movement, a two-dimensional (2D) movement and a three-dimensional (3D) movement.

7. The medium of claim 6, wherein the instructions, if executed, cause the device to classify the movement as the device action, wherein the device action is to include one or more of a single click operation, a double click operation, a pinch operation and a cursor movement operation.

8. The medium of claim 5, wherein the instructions, if executed, cause the device to send a request to a device driver to initiate the device action, wherein the device driver is to include one or more of a mouse driver, a touchpad driver and a trackball driver.

9. The medium of claim 5, wherein the instructions, if executed, cause the device to:

synchronize the gaze information with the gesture information; and

log one or more of the gaze information and the gesture information.

10. An apparatus comprising:

a gaze module to identify a point of interest on a front facing display of a device based on gaze information associated with a user of the device, including;

identifying one or more eyes of the user,

determining a gaze location of the one or more eyes on the front facing display, and

determining a change in the gaze location, wherein the gaze information is to include the gaze location and the change in the gaze location;

a gesture module to identify a hand action based on gesture information associated with the user of the device, including;

identifying a hand of the user,

identifying one or more fingertips associated with the hand of the user, and

determining a movement of one or more of the hand and the one or more fingertips, wherein the gesture information is to include the movement and the movement is to be one or more of a one-dimensional (1D) movement, a two-dimensional (2D) movement and a three-dimensional (3D) movement;

an integration module to initiate a device action with respect to the front facing display based on the point of interest and the hand action, wherein the device action is to emulate a multi-modal touchscreen function with respect to the front facing display based on the first image and the second image, and wherein the multi-modal touchscreen function is to concurrently track and synchronize the gaze information and the gesture information;

a camera interface to receive image data from a front facing camera configuration of the device, wherein the gaze module is to identify the gaze information based on the image data and the gesture module is to identify the gesture information based on the image data,

wherein the camera interface is to;

receive a first image from a first camera of the camera configuration, and

receive a second image from a second camera of the camera configuration, wherein the image data is to include the first image and the second image, the gaze module is to determine the gaze information based on the first image, and the gesture module is to determine the gesture information based on the second image.

11. The apparatus of claim 10, wherein the integration module is to classify the movement as the device action, wherein the device action is to include one or more of a single click operation, a double click operation, a pinch operation and a cursor movement operation.

12. The apparatus of claim 10, wherein the integration module is to send a request to a device driver to initiate the device action, wherein the device driver is to include one or more of a mouse driver, a touchpad driver and a trackball driver.

13. The apparatus of claim 10, wherein the integration module is to:

synchronize the gaze information with the gesture information, and

log one or more of the gaze information and the gesture information.

14. A device comprising:

a front facing display; and

a multi-modal touch screen emulator including;

a gaze module to identify a point of interest on the front facing display based on gaze information associated with a user of the device, including;

identifying one or more eyes of the user,

determining a gaze location of the one or more eyes on the front facing display, and

determining a change in the gaze location, wherein the gaze information is to include the gaze location and the change in the gaze location,

a gesture module to identify a hand action based on gesture information associated with a user of the device, and

an integration module to initiate a device action with respect to the front facing display based on the point of interest and the hand action, wherein the device action is to emulate a multi-modal touchscreen function with respect to the front facing display based on the first image and the second image, and wherein the multi-modal touchscreen function is to concurrently track and synchronize the gaze information and the gesture information; and

a front facing camera configuration,

wherein the multi-modal touch screen emulator further includes a camera interface to receive image data from the front facing camera configuration, the gaze module is to identify the gaze information based on the image data, and the gesture module is to identify the gesture information based on the image data, and

wherein the camera configuration includes a first camera and a second camera, and wherein the camera interface is to,

receive a first image from the first camera of the camera configuration, and

receive a second image from the second camera of the camera configuration, wherein the image data is to include the first image and the second image, the gaze module is to determine the gaze information based on the first image, and the gesture module is to determine the gesture information based on the second image.

15. The device of claim 14, wherein the gesture module is to,

identify a hand of the user,

identify one or more fingertips associated with the hand of the user, and

determine a movement of one or more of the hand and the one or more fingertips, wherein the gesture information is to include the movement and the movement is to be one or more of a one-dimensional (1D) movement, a two-dimensional (2D) movement and a three-dimensional (3D) movement.

16. The device of claim 15, wherein the integration module is to classify the movement as the device action, wherein the device action is to include one or more of a single click operation, a double click operation, a pinch operation and a cursor movement operation.

17. The device of claim 14, wherein the integration module is to send a request to a device driver to initiate the device action, wherein the device driver is to include one or more of a mouse driver, a touchpad driver and a trackball driver.

18. The device of claim 14, wherein the integration module is to:

synchronize the gaze information with the gesture information, and

log one or more of the gaze information and the gesture information.

19. A method comprising:

identifying one or more eyes of a user of a device having a front facing display;

determining a gaze location of the one or more eyes on the front facing display;

identifying a point of interest on the front facing display based on gaze information associated with the user of the device;

determining a change in the gaze location, wherein the gaze information is to include the gaze location and the change in the gaze location;

identifying a hand action based on gesture information associated with the user of the device;

initiating a device action with respect to the front facing display based on the point of interest and the hand action, wherein the device action is to emulate a multi-modal touchscreen function with respect to the front facing display based on the first image and the second image, and wherein the multi-modal touchscreen function is to concurrently track and synchronize the gaze information and the gesture information;

receiving image data from a front facing camera configuration of the device;

identifying the gaze information and the gesture information based on the image data;

receiving a first image from a first camera of the camera configuration; and

receiving a second image from a second camera of the camera configuration, wherein the image data is to include the first image and the second image, the gaze information is to be determined based on the first image, and the gesture information is to be determined based on the second image.

20. The method of claim 19, further including:

identifying a hand of the user;

identifying one or more fingertips associated with the hand of the user; and

determining a movement of one or more of the hand and the one or more fingertips, wherein the gesture information is to include the movement and the movement is to be one or more of a one-dimensional (1D) movement, a two-dimensional (2D) movement and a three-dimensional (3D) movement.

21. The method of claim 20, further including classifying the movement as the device action, wherein the device action is to include one or more of a single click operation, a double click operation, a pinch operation and a cursor movement operation.

22. The method of claim 19, further including sending a request to a device driver to initiate the device action, wherein the device driver is to include one or more of a mouse driver, a touchpad driver and a trackball driver.

23. The method of claim 19, further including:

synchronizing the gaze information with the gesture information; and

logging one or more of the gaze information and the gesture information.