HEAD AND EYE TRACKING

• US 20160066781A1
• Filed: 04/10/2014
• Published: 03/10/2016
• Est. Priority Date: 04/10/2013
• Status: Active Grant

• Current Assignee: Auckland UniServices Limited (Auckland University)
• Original Assignee: Auckland UniServices Limited (Auckland University)
• Inventors: Thompson, Ben, Turuwhenua, Jason
• Granted Patent: US 10,506,924 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: A61B 3/113 for determining or recording eye movement
• CPC Class Codes: A61B 3/145 by video means
• CPC Class Codes: A61B 5/0077 Devices for viewing the surface of the body, e.g. camera, magnifying lens
• CPC Class Codes: A61B 5/1114 Tracking parts of the body
• CPC Class Codes: A61B 5/1128 using image analysis A61B5/1127 takes precedence
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/30041 Eye; Retina; Ophthalmic
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 7/0012 Biomedical image inspection
• CPC Class Codes: G06T 7/269 using gradient-based methods

Abstract

Embodiments of the invention relate to a method of extracting eye velocity information from a video footage having a plurality of frames, comprising detecting at least part of an eye in at least two frames of the video footage, applying an optical flow algorithm to the at least two frames of the video footage to extract pixel velocity information, and determining a statistical measure from the pixel velocity information within the detected at least part of the eye. Other embodiments of the invention relate to a method of extracting head image trajectory information from a video footage having a plurality of frames, comprising detecting at least part of a facial region of the head image in at least two frames of the video footage, determining a measure of the movement of the at least part of a facial region between the at least two frames, and determining a transformation map from the measure of the movement.

Claims

1. A method of extracting eye velocity information from a video footage having a plurality of frames, comprising:

detecting at least part of an eye in at least two frames of the video footage,

applying an optical flow algorithm to the at least two frames of the video footage to extract pixel velocity information, and

determining a statistical measure from the pixel velocity information within the detected at least part of the eye.

4. The method, system or device as claimed in any one of claims 1 to 3, further comprising one or more of the steps of (in any order):

determining the velocity magnitude and direction of at least part of an eye from the velocity information,

determining any velocity maxima and minima from the velocity information,

comparing any maxima or minima to a lower velocity threshold and discarding any maxima or minima below the threshold,

comparing any maxima or minima to an upper velocity threshold and discarding any maxima or minima above the threshold,

comparing the inter-frame distance between adjacent maxima or minima to a frame threshold and discarding any maxima or minima below the frame threshold,

determining any sole maxima or sole minima in the velocity information and discarding sole maxima or sole minima,

averaging the maxima or minima velocity magnitudes, and/or

normalising the maxima or minima velocity magnitudes.

5. The method, system or device as claimed in any one of claims 1 to 4, wherein the statistical measure of the pixel velocity information is an average pixel velocity.

6. The method, system or device as claimed in any one of claims 1 to 5, wherein the at least part of the eye is a region of contrast discontinuity of the eye.

7. The method, system or device as claimed in any one of claims 1 to 6, wherein the at least part of the eye is a limbus portion of the eye.

8. The method, system or device as claimed in any one of claims 1 to 7, further comprising generating a list of pixel velocity averages for each consecutive frame in the video footage.

9. The method, system or device as claimed in any one of claims 1 to 8, further comprising comparing the determined velocity information to healthy velocity information to make a determination of the health of the eye and/or visual pathway.

10. The method, system or device as claimed in any one of claims 1 to 9, further comprising comparing the determined velocity information to healthy velocity information to make a determination of the presence and/or strength of OKN.

11. The method, system or device as claimed in any one of claims 1 to 10, wherein the limbus portion of the eye is determined by an edge detection algorithm.

12. The method, system or device as claimed in claim 11, wherein edge detection is performed by application of a Prewitt operator.

13. The method, system or device as claimed in any one of claims 1 to 12, wherein the optical flow algorithm is a Lucas-Kanade optical flow algorithm.

14. The method, system or device as claimed in any one of claims 1 to 13, wherein the optical flow algorithm is applied to determine pixel velocity information between consecutive frames of a length of video footage corresponding to several reset events.

15. The method, system or device as claimed in any one of claims 1 to 14, further comprising a visual stimulus arranged in front of the eye, the stimulus operable to elicit optokinetic nystagmus.

16. The method, system or device as claimed in claim 15, further comprising comparing the average velocity information to known optokinetic nystagmus velocity information to make a determination of the health of an eye in response to the stimulus.

17. The method, system or device as claimed in one of claims 1 to 16, wherein the video footage is obtained from a subject watching a stimulus.

18. The method, system or device as claimed in claim 17, wherein the stimulus is a screen.

19. The method, system or device as claimed in claim 18, wherein the camera and screen are contained within an enclosure.

20. The method, system or device as claimed in claim 18, wherein the camera, screen and controller are integrated within an enclosure.

21. The method, system or device as claimed in claim 19 or 20, wherein the enclosure is any one or more of a smart phone, tablet or portable computation device.

22. The method, system or device as claimed in any one of claims 1 to 21, wherein detecting at least part of an eye does not include detection of a feature of the eye.

25. The method of claim 23 or claim 24, wherein the method further comprises the method of extracting eye velocity information from the recording of the eye according to claim 1.

26. The method, system or device as claimed in any one of claims 1 to 25, wherein the statistical measure is applied to determine information relating to one or more of a person'"'"'s behaviour, a person'"'"'s response to advertising, security purposes, or consumer attention.

27. The method, system or device as claimed in any one of claims 1 to 26, further comprising extracting head image trajectory information from the video footage by a method comprising:

detecting at least part of a facial region of the head in at least two frames of the video footage,

determining a measure of the movement of the at least part of a facial region between the at least two frames, and

determining a transformation map from the measure of the movement.

28. The method, system or device as claimed in claim 27, further comprising:

determining the inverse of the transformation map,

applying inverse of the transformation to each frame in the video footage such the facial region within the frame is held substantially constant.

29. The method, system or device as claimed in any one of claim 27 or 28, further comprising detecting the facial region from within a frame of the video footage using the Viola-Jones algorithm.

30. The method, system or device as claimed in any one of claims 27 to 29, further comprising identifying one or more particular portions of the facial region within the frame using a Harris corner detection algorithm.

31. The method, system or device as claimed in any one of claims 27 to 30, further comprising determining the measure of the movement of the facial region or one or more particular portions of a facial region between consecutive video frames using a Kanade Lucas Tomasi (KLT) point tracker.

32. The method, system or device as claimed in any one of claims 27 to 31, wherein the measure of the movement comprises one or more of rotation, scale and/or translation movement.

33. The method, system or device as claimed in any one of claims 27 to 32, wherein a similarity transformation map is created from the movement information.

34. The method, system or device as claimed in claim 33, further comprising applying the similarity transformation map to offset each frame in the video footage such that the facial region within the frame is held substantially constant with respect to each frame.

35. The method, system or device as claimed in any one of claims 27 to 34, further comprising applying an inverse similarity transformation to crop each frame in the video footage to generate a new frame such that the facial region within the frame is held substantially constant with respect to each frame.

36. The method, system or device as claimed in any one of claims 27 to 35, wherein the head trajectory information is used to improve the eye velocity information.

37. The method, system or device as claimed in any one of claims 27 to 35, wherein head trajectory information is offset against the eye tracking information to substantially remove eye velocity information caused by head movement.

51. The method, system or device as claimed in any one of claims 1 to 35 in combination with the method, system or device of any one of claims 40 to 50.

2. An eye tracking system comprising:

a camera arranged to capture footage of an eye,

a controller configured to receive the footage of the eye and perform the steps of;

detecting at least part of an eye in at least two frames of the footage,

applying an optical flow algorithm to the footage to thereby determine pixel velocity information between at least two frames of the footage, and

determining a statistical measure from the pixel velocity information within the detected at least part of the eye.

3. A controller device programmed to perform the steps of:

detecting at least part of an eye in at least two frames of the footage,

applying an optical flow algorithm to the footage to thereby determine pixel velocity information between at least two frames of the footage, and

determining a statistical measure of the pixel velocity information within the detected at least part of the eye.

23. A method of improving the attention of a subject for eye tracking comprising:

operating at least one display to show a video of an video functional to gain the attention of a viewer,

operating at least one display to show an OKN stimulus video, and

recording an eye watching the OKN stimulus video with a camera.

24. The method of claim 23, wherein the video functional to gain the attention of a viewer is an animated video.

38. A method of processing video footage having a plurality of frames to determine the presence of OKN, the method comprising:

extracting head trajectory information by a method comprising;

detecting at least part of a facial region of the head image in at least two frames of the video footage,

determining a measure of the movement of the at least part of a facial region between the at least two frames, and

determining a transformation map from the measure of the movement, and

extracting eye velocity information by a method comprising;

detecting at least part of an eye in at least two frames of the video footage,

applying an optical flow algorithm to the at least two frames of the video footage to extract pixel velocity information, and

determining a statistical measure from the pixel velocity information within the detected at least part of the eye.

39. A method as claimed in claim 38 wherein the head trajectory information is offset against the eye tracking information to substantially remove eye velocity information caused by head movement.

40. A method of extracting head image trajectory information from a video footage having a plurality of frames, comprising:

detecting at least part of a facial region of the head image in at least two frames of the video footage,

determining a measure of the movement of the at least part of a facial region between the at least two frames, and

determining a transformation map from the measure of the movement.

43. The method, system or device as claimed in any one of claims 40 to 42, wherein the method or steps further comprise:

determining the inverse of the transformation map,

applying inverse of the transformation to each frame in the video footage such the facial region within the frame is held substantially constant.

44. The method, system or device as claimed in any one of claims 40 to 43, further comprising detecting the facial region from within a frame of the video footage using the Viola-Jones algorithm.

45. The method, system or device as claimed in any one of claims 40 to 44, further comprising identifying one or more particular portions of the facial region within the frame using a Harris corner detection algorithm.

46. The method, system or device as claimed in any one of claims 40 to 45, further comprising determining the measure of the movement of the facial region or one or more particular portions of a facial region between consecutive video frames using a Kanade Lucas Tomasi (KLT) point tracker.

47. The method, system or device as claimed in any one of claims 40 to 46, wherein the measure of the movement comprises one or more of rotation, scale and/or translation movement.

48. The method, system or device as claimed in any one of claims 40 to 47, wherein a similarity transformation map is created from the movement information.

50. The method, system or device as claimed in any one of claims 40 to 49, further comprising applying an inverse similarity transformation to crop each frame in the video footage to generate a new frame such that the facial region within the frame is held substantially constant with respect to each frame.

41. A head tracking system for extracting head image trajectory information comprising:

a camera arranged to capture footage of a head and a controller configured to receive the footage of the head and perform the steps of;

detecting at least part of a facial region of the head in at least two frames of the video footage,

determining a measure of the movement of the at least part of a facial region between the at least two frames, and

determining a transformation map from the measure of the movement.

42. A controller device configured to perform the steps of:

detecting at least part of a facial region of the head in at least two frames of the video footage,

determining a measure of the movement of the at least part of a facial region between the at least two frames, and

determining a transformation map from the measure of the movement.

49. The method, system or device as claimed in claim 49, further comprising applying the similarity transformation map to offset each frame in the video footage such that the facial region within the frame is held substantially constant with respect to each frame.