Detecting hand-eye coordination in real time by combining camera eye tracking and wearable sensing

• US 10,296,786 B2
• Filed: 02/15/2017
• Issued: 05/21/2019
• Est. Priority Date: 02/15/2017
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Banerjee, Dipyaman, Dey, Kuntal
• Primary Examiner(s): Bernardi, Brenda C
• Application Number:
  US15/433,282
• Publication Number:
  US 20180232890A1
• Time in Patent Office: 825 Days
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/033 Pointing devices displaced or positioned by the user , e.g. mice, trackballs, pens or joysticks; Accessories therefor digitisers characterised by the transducing means G06F3/041
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 7/0016 involving temporal comparison
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

Methods, systems, and computer program products for detecting hand-eye coordination in real time by combining camera eye tracking and wearable sensing are provided herein. A computer-implemented method includes determining an eye gaze path of an individual during a given period of time by measuring (i) any eye movements made by the individual during the given period of time and (ii) any head movements made by the individual during the given period of time; measuring any hand movements made by the individual during the given period of time; performing a spatio-temporal analysis of the determined eye gaze path and the measured hand movements to compute a hand-eye coordination value attributable to the individual; and outputting the computed hand-eye coordination value to at least one user.

Claims

1. A computer-implemented method, comprising:

determining an eye gaze path of an individual during a given period of time, wherein said determining the gaze path comprises measuring, via captured video data, (i) any eye movements made by the individual during the given period of time and (ii) any head movements made by the individual during the given period of time;

measuring, via data derived from one or more wearable sensors, any hand movements made by the individual during the given period of time;

performing a spatio-temporal analysis of the determined eye gaze path and the measured hand movements to compute a hand-eye coordination value attributable to the individual, wherein the spatio-temporal analysis comprises;

comparing (i) temporal values associated with movement along the determined eye gaze path and (ii) temporal values associated with the measured hand movement;

defining a given projected plane as an optical projection plane in the three-dimensional space, wherein said defining comprises maintaining one dimension as a constant and varying the other two dimensions; and

comparing (i) one or more spatial trajectories associated with the determined eye gaze path on the given projected plane and (ii) one or more spatial trajectories associated with the measured hand movements on the given projected plane, wherein said comparing spatial trajectories comprises measuring one or more deviations, with respect to a set of one or more reference points, between (a) the one or more spatial trajectories associated with the measured hand movements on the given projected plane and (b) one or more gaze saccades within the determined eye gaze path; and

outputting the computed hand-eye coordination value to at least one user;

wherein the steps are carried out by at least one computing device.

2. The computer-implemented method of claim 1, wherein the captured video data comprise video data captured by at least one monocular video camera.

3. The computer-implemented method of claim 1, wherein said determining an eye gaze path comprises determining one or more time-stamped gaze patterns of the individual during the given period of time.

4. The computer-implemented method of claim 1, wherein said determining an eye gaze path comprises:

determining multiple gaze points during the given period of time; and

creating the eye gaze path based on the multiple gaze points.

5. The computer-implemented method of claim 1, wherein the data derived from one or more wearable sensors comprise motion data.

6. The computer-implemented method of claim 1, wherein the data derived from one or more wearable sensors comprise orientation data.

7. The computer-implemented method of claim 1, wherein the one or more wearable sensors comprises an accelerometer.

8. The computer-implemented method of claim 1, wherein the one or more wearable sensors comprises a gyroscope.

9. The computer-implemented method of claim 1, wherein the one or more wearable sensors comprises a magnetometer.

10. The computer-implemented method of claim 1, wherein the one or more wearable sensors comprise one or more sensors resident on a smart watch worn by the individual.

11. The computer-implemented method of claim 1, wherein said spatio-temporal analysis comprises measuring one or more hand fixation zones and one or more eye fixation zones at one or more related timestamps.

12. The computer-implemented method of claim 1, wherein said comparing temporal values comprises measuring a delay between one or more of the measured hand movements and one or more movements along the determined eye gaze path.

13. A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a device to cause the device to:

determine an eye gaze path of an individual during a given period of time, wherein said determining the gaze path comprises measuring, via captured video data, (i) any eye movements made by the individual during the given period of time and (ii) any head movements made by the individual during the given period of time;

measure, via data derived from one or more wearable sensors, any hand movements made by the individual during the given period of time;

perform a spatio-temporal analysis of the determined eye gaze path and the measured hand movements to compute a hand-eye coordination value attributable to the individual, wherein the spatio-temporal analysis comprises;

comparing (i) temporal values associated with movement along the determined eye gaze path and (ii) temporal values associated with the measured hand movement;

defining a given projected plane as an optical projection plane in the three-dimensional space, wherein said defining comprises maintaining one dimension as a constant and varying the other two dimensions; and

comparing (i) one or more spatial trajectories associated with the determined eye gaze path on the given projected plane and (ii) one or more spatial trajectories associated with the measured hand movements on the given projected plane, wherein said comparing spatial trajectories comprises measuring one or more deviations, with respect to a set of one or more reference points, between (a) the one or more spatial trajectories associated with the measured hand movements on the given projected plane and (b) one or more gaze saccades within the determined eye gaze path; and

output the computed hand-eye coordination value to at least one user.

14. The computer program product of claim 13, wherein said determining an eye gaze path comprises determining one or more time-stamped gaze patterns of the individual during the given period of time.

15. The computer program product of claim 13, wherein said determining an eye gaze path comprises:

determining multiple gaze points during the given period of time; and

creating the eye gaze path based on the multiple gaze points.

16. The computer program product of claim 13, wherein said comparing temporal values comprises measuring a delay between one or more of the measured hand movements and one or more movements along the determined eye gaze path.

17. A system comprising:

a memory; and

at least one processor operably coupled to the memory and configured for;

determining an eye gaze path of an individual during a given period of time, wherein said determining the gaze path comprises measuring, via captured video data, (i) any eye movements made by the individual during the given period of time and (ii) any head movements made by the individual during the given period of time;

measuring, via data derived from one or more wearable sensors, any hand movements made by the individual during the given period of time;

performing a spatio-temporal analysis of the determined eye gaze path and the measured hand movements to compute a hand-eye coordination value attributable to the individual, wherein the spatio-temporal analysis comprises;

comparing (i) temporal values associated with movement along the determined eye gaze path and (ii) temporal values associated with the measured hand movement;

defining a given projected plane as an optical projection plane in the three-dimensional space, wherein said defining comprises maintaining one dimension as a constant and varying the other two dimensions; and

comparing (i) one or more spatial trajectories associated with the determined eye gaze path on the given projected plane and (ii) one or more spatial trajectories associated with the measured hand movements on the given projected plane, wherein said comparing spatial trajectories comprises measuring one or more deviations, with respect to a set of one or more reference points, between (a) the one or more spatial trajectories associated with the measured hand movements on the given projected plane and (b) one or more gaze saccades within the determined eye gaze path; and

outputting the computed hand-eye coordination value to at least one user.

18. A computer-implemented method, comprising:

providing, to a user, a task which requires hand-eye coordination by the user;

commencing the task at a first time value and ending the task at a second time value;

determining an eye gaze path of the user during a temporal period that includes the first time value and the second time value, wherein said determining the gaze path comprises measuring, via captured video data, (i) any eye movements made by the user during the temporal period and (ii) any head movements made by the user during the temporal period;

measuring, via data derived from one or more wearable sensors, any hand movements made by the user during the temporal period;

performing a spatio-temporal analysis of the determined eye gaze path and the measured hand movements to compute a hand-eye coordination value for the user, with respect to the task, wherein the spatio-tempora analysis comprises;

comparing (i) temporal values associated with movement along the determined eye gaze path and (ii) temporal values associated with the measured hand movement;

defining a given projected plane as an optical projection plane in the three-dimensional space, wherein said defining comprises maintaining one dimension as a constant and varying the other two dimensions; and

comparing (i) one or more spatial trajectories associated with the determined eye gaze path on the given projected plane and (ii) one or more spatial trajectories associated with the measured hand movements on the given projected plane, wherein said comparing spatial trajectories comprises measuring one or more deviations, with respect to a set of one or more reference points, between (a) the one or more spatial trajectories associated with the measured hand movements on the given projected plane and (b) one or more gaze saccades within the determined eye gaze path; and

outputting the computed hand-eye coordination value to the user;

wherein the steps are carried out by at least one computing device.