Systems and methods for tracking motion and gesture of heads and eyes

• US 9,785,249 B1
• Filed: 12/21/2016
• Issued: 10/10/2017
• Est. Priority Date: 12/06/2016
• Status: Active Grant

• Current Assignee: Vuelosophy, Inc.
• Original Assignee: Vuelosophy, Inc.
• Inventors: Nie, Xiaochun
• Primary Examiner(s): Sheng, Tom
• Application Number:
  US15/387,024
• Time in Patent Office: 293 Days
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883

Abstract

Apparatus, systems and methods configured for tracking head and eye movement are disclosed. In one example, an apparatus comprises an imaging sensor, a spatial orientation sensor and a computing device comprising a processor and a memory communicatively coupled with the processor. The imaging sensor is configured for capturing and recording images of an eye of a user when the user stares at a gaze point and rotates a head of the user about a first axis and about a second axis that is orthogonal to the first axis. The spatial orientation sensor is coupled to the imaging sensor such that the imaging sensor has a fixed position and a fixed orientation relative to the spatial orientation sensor and is configured for generating outputs during the rotations of the head of the user, wherein each of the outputs corresponds to one of the images of the eye.

Claims

1. An apparatus configured for tracking head movement, comprising:

a spatial orientation sensor configured for;

generating a first output of orientation of the spatial orientation sensor during a first rotation of a head of a user about a first axis, and

generating a second output of orientation of the spatial orientation sensor during a second rotation of the head of the user about a second axis that is orthogonal to the first axis; and

a computing device comprising a processor and a memory communicatively coupled with the processor, wherein the processor is configured for;

determining an orientation relationship between the head and the spatial orientation sensor based on the first and second outputs.

2. The apparatus of claim 1, wherein the computing device comprises a working mode coordinator configured for determining a working mode for the computing device based on an input of the user, wherein the working mode is at least one of:

head calibration, head tracking, and gesture determination.

3. The apparatus of claim 1, wherein the computing device comprises a head calibrator configured for:

obtaining the first output of the spatial orientation sensor;

calculating a first rotation vector aligned with the first axis based on the first output;

obtaining the second output of the spatial orientation sensor;

calculating a second rotation vector aligned with the second axis based on the second output; and

calculating one or more parameters representing an orientation relationship between the head and the spatial orientation sensor based on the first and second rotation vectors.

4. The apparatus of claim 1, wherein the computing device comprises a head tracker configured for:

obtaining an initial output of the spatial orientation sensor when the head is at an initial orientation;

obtaining a final output of the spatial orientation sensor when the head is at a final orientation; and

detecting a head movement from the initial orientation to the final orientation, based on the initial output of the spatial orientation sensor, the final output of the spatial orientation sensor, and the orientation relationship between the head and the spatial orientation sensor.

5. The apparatus of claim 4, wherein the head tracker is further configured for:

configuring the final output to be a new initial output of the spatial orientation sensor; and

configuring the final orientation to be a new initial orientation of the head, such that future detection of a head movement is relative to the new initial orientation.

6. The apparatus of claim 4, wherein the computing device comprises a gesture determiner configured for:

determining a head gesture based on the detected head movement, wherein the head gesture is head nod when the head moves up and down about a third axis, wherein the head gesture is head shake when the head moves left and right about a fourth axis that is orthogonal to the third axis, and wherein the head gesture is head bobble when the head moves about a fifth axis that is orthogonal to both the third axis and the fourth axis.

7. The apparatus of claim 6, wherein the computing device comprises an instruction generator configured for generating an instruction based on the determined head gesture, wherein the instruction corresponds to an input of the user to a machine.

8. The apparatus of claim 1, wherein the spatial orientation sensor is the only spatial orientation sensor in the apparatus.

9. An apparatus configured for tracking eye movement, comprising:

an imaging sensor configured for capturing and recording images of an eye of a user when the user stares at a gaze point and performs a sequence of eye motions relative to a head of the user;

a spatial orientation sensor, coupled to the imaging sensor such that the imaging sensor has a fixed position and a fixed orientation relative to the spatial orientation sensor, configured for;

generating outputs of orientation of the spatial orientation sensor during the sequence of eye motions, wherein each of the outputs corresponds to one of the images of the eye; and

a computing device comprising a processor and a memory communicatively coupled with the processor, wherein the processor is configured for;

determining a position relationship between an eyeball center of the eye and the imaging sensor based on;

the images of the eye, the outputs of the spatial orientation sensor, and the fixed orientation of the imaging sensor relative to the spatial orientation sensor.

10. The apparatus of claim 9, wherein the computing device comprises a working mode coordinator configured for determining a working mode for the computing device based on an input of the user, wherein the working mode is at least one of:

eye calibration, eye tracking, and gesture determination.

11. The apparatus of claim 9, wherein the apparatus is mounted on a fixed platform and the computing device is further configured for:

configuring two markers in front of the imaging sensor;

configuring three evenly spaced horizontal lines on a screen of the imaging sensor, wherein a middle line of the three lines is in middle of the screen;

obtaining a first output of the spatial orientation sensor when the imaging sensor is oriented to an orientation such that an image of the two markers sits in middle of a top line of the three lines;

obtaining a second output of the spatial orientation sensor when the imaging sensor is oriented to an orientation such that an image of the two markers sits in middle of a bottom line of the three lines;

obtaining a third output of the spatial orientation sensor when the imaging sensor is oriented to an orientation such that an image of the two markers sits on left side of the middle line of the three lines;

obtaining a fourth output of the spatial orientation sensor when the imaging sensor is oriented to an orientation such that an image of the two markers sits on right side of the middle line of the three lines; and

calculating the fixed orientation of the imaging sensor relative to the spatial orientation sensor and a focal length of the imaging sensor based on the first, second, third, and fourth outputs.

12. The apparatus of claim 9, wherein the computing device comprises an eye calibrator configured for:

instructing the user to stare at the gaze point and perform the sequence of eye motions;

obtaining a sequence of data pairs each of which includes an image of the eye and a corresponding output of the spatial orientation sensor;

obtaining coordinates of an eyeball center of the eye in each image; and

calculating coordinates of the eyeball center relative to the imaging sensor.

13. The apparatus of claim 12, wherein the eye calibrator is further configured for calculating coordinates of eyeball centers of both eyes of the user relative to the imaging sensor based on images of both eyes captured by the imaging sensor.

14. The apparatus of claim 9, wherein the computing device comprises an eye tracker configured for:

calculating a gaze vector pointing through an eyeball center of the eye on a head to an object the user is looking at, relative to the imaging sensor;

calculating the gaze vector relative to the head based on the gaze vector relative to the imaging sensor;

calculating a final orientation of the eye relative to the head based on the gaze vector relative to the head;

detecting an eye movement of the eye from a predetermined initial orientation to the final orientation of the eye relative to the head, based on;

the predetermined initial orientation of the eye relative to the head and the final orientation of the eye relative to the head; and

calculating a final gaze vector relative to the world based on the eye movement of the eye and an orientation relationship between the head and the spatial orientation sensor.

15. The apparatus of claim 14, wherein the eye tracker is further configured for:

configuring the final orientation of the eye to be a new initial orientation of the eye relative to the head, such that future detection of an eye movement is based on the new initial orientation of the eye relative to the head.

16. The apparatus of claim 14, wherein the computing device comprises a gesture determiner configured for:

determining an eye gesture based on the detected eye movement, wherein the eye gesture is a stare condition when the gaze vector relative to the world is limited in a predetermined small range over a period of time.

17. The apparatus of claim 16, wherein the computing device comprises an instruction generator configured for generating an instruction based on the determined eye gesture, wherein the instruction corresponds to an input of the user to a machine.

18. The apparatus of claim 14, wherein the eye tracker is further configured for detecting eye movements of both eyes of the user, wherein a gaze vector corresponding to each of the eyes moves from a predetermined initial orientation to a final orientation relative to the world.

19. The apparatus of claim 9, wherein the apparatus comprises no more imaging sensor other than the imaging sensor and no more spatial orientation sensor other than the spatial orientation sensor.

20. An apparatus configured for tracking head and eye movement, comprising:

an imaging sensor configured for capturing and recording images of an eye of a user when the user stares at a gaze point and rotates a head of the user about a first axis and about a second axis that is orthogonal to the first axis;

a spatial orientation sensor, coupled to the imaging sensor such that the imaging sensor has a fixed position and a fixed orientation relative to the spatial orientation sensor, configured for;

generating outputs of orientation of the spatial orientation sensor during rotations of the head of the user, wherein each of the outputs corresponds to one of the images of the eye; and

a computing device comprising a processor and a memory communicatively coupled with the processor, wherein the processor is configured for;

determining an orientation relationship between the head and the spatial orientation sensor based on the outputs, and

determining a position relationship between the center of an eyeball of the eye and the imaging sensor based on;

the images of the eye, the outputs of the spatial orientation sensor, and the fixed orientation of the imaging sensor relative to the spatial orientation sensor.

21. The apparatus of claim 20, wherein the computing device comprises:

a head tracker configured for;

obtaining an initial output of the spatial orientation sensor when the head is at an initial orientation,

obtaining a final output of the spatial orientation sensor when the head is at a final orientation, and

detecting a head movement from the initial orientation to the final orientation, based on the initial output of the spatial orientation sensor, the final output of the spatial orientation sensor, and the orientation relationship between the head and the spatial orientation sensor; and

an eye tracker configured for;

calculating a gaze vector pointing through the center of the eyeball to an object the user is looking at, relative to the imaging sensor,

calculating a gaze vector relative to the head based on the gaze vector relative to the imaging sensor,

calculating a final orientation of the eye relative to the head based on the gaze vector relative to the head,

detecting an eye movement of the eye from a predetermined initial orientation to the final orientation of the eye relative to the head, based on;

the predetermined initial orientation of the eye relative to the head and the final orientation of the eye relative to the head, and

calculating a final gaze vector relative to the world based on the eye movement of the eye and an orientation relationship between the head and the spatial orientation sensor.

22. The apparatus of claim 21, wherein the head movement and the eye movement are detected at the same time.

23. The apparatus of claim 21, wherein the head movement and the eye movement are detected independently from each other.

24. The apparatus of claim 21, wherein:

the eye tracker is further configured for configuring the final orientation of the eye to be a new initial orientation of the eye relative to the head, such that future detection of an eye movement is based on the new initial orientation of the eye relative to the head; and

the head tracker is further configured for;

configuring the final output to be a new initial output of the spatial orientation sensor; and

configuring the final orientation to be a new initial orientation of the head, such that future detection of a head movement is relative to the new initial orientation of the head.

25. The apparatus of claim 21, wherein the computing device comprises a gesture determiner configured for:

determining a head gesture based on the detected head movement, wherein the head gesture is head nod when the head moves up and down about a third axis, wherein the head gesture is head shake when the head moves left and right about a fourth axis that is orthogonal to the third axis, and wherein the head gesture is head bobble when the head moves about a fifth axis that is orthogonal to both the third axis and the fourth axis;

determining an eye gesture based on the detected eye movement, wherein the eye gesture is a stare condition when the gaze vector is limited in a predetermined small range over a period of time; and

determining a gesture combination based on the eye gesture and the head gesture.

26. The apparatus of claim 25, wherein the computing device comprises an instruction generator configured for generating an instruction based on the determined gesture combination, wherein the instruction corresponds to an input of the user to a machine.

27. The apparatus of claim 20, wherein the apparatus comprises no more imaging sensor other than the imaging sensor and no more spatial orientation sensor other than the spatial orientation sensor.