Method and system for correcting gaze offset

• US 9,141,875 B2
• Filed: 04/26/2011
• Issued: 09/22/2015
• Est. Priority Date: 05/26/2010
• Status: Expired due to Fees

• Current Assignee: Ramot at Tel Aviv University Limited (Tel Aviv University)
• Original Assignee: Ramot at Tel Aviv University Limited (Tel Aviv University)
• Inventors: Wolf, Lior, Freund, Ziv
• Primary Examiner(s): Anwah, Olisa
• Application Number:
  US13/699,751
• Publication Number:
  US 20130070046A1
• Time in Patent Office: 1,610 Days
• Field of Search: 348/14.16, 348/14.01, 348/14.12, 348/14.14, 382/103, 382/275, 345/8, 345/156
• US Class Current: 1/1
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction
• CPC Class Codes: G06V 40/197 Matching; Classification
• CPC Class Codes: H04N 7/144 camera and display on the same optical axis, e.g. optically multiplexing the camera and display for eye to eye contact
• CPC Class Codes: H04N 7/15 Conference systems

Abstract

A method of correcting gaze offset in an image of at least one individual having eyes is disclosed. The method comprises: processing the image to extract location of at least one eye over the image, processing the image to replace imagery data associated with each location of each eye with replacement data thereby providing a corrected image, and transmitting the corrected image to a display device. The replacement data are preferably previously-recorded imagery data which respectively correspond to the same eye but a different gaze.

Claims

1. A method of correcting gaze offset in an image of at least one individual having eyes, comprising:

processing the image so as to extract location of at least one eye over the image;

processing the image to replace imagery data associated with each location of each eye with replacement data, said replacement data being previously-recorded imagery data which respectively correspond to the same eye but a different gaze, wherein said replacing is based, at least in part, on eye parameters defining a shape of said eye and being computed using a non-specific database of eye images, thereby providing a corrected image; and

transmitting said corrected image to a display device.

2. The method of claim 1, further comprising, prior to said extraction of said locations, capturing at least one image of each of the at least one individual at one or more predetermined gazes, thereby providing said previously-recorded data.

3. The method according to claim 1, further comprising processing said previously-recorded data so as to compute eye parameters wherein said replacing is based on said computed parameters.

4. The method of claim 3, wherein said computation of said parameters is bootstrapped using said non-specific database of eye images.

5. The method according to claim 3, further comprising applying a morphological transformation to said previously-recorded data according to said parameters.

6. The method according to claim 4, further comprising varying said parameters so as to optimize a predetermined objective function describing matching between said replacement data and images in said non-specific database.

7. The method according to claim 3, further comprising varying said parameters so as to optimize a predetermined objective function describing matching between said replacement data and said imagery data associated with said location of said eye.

8. The method according to claim 1, further comprising, for each eye, determining whether an eyelid of said eye is close or open, wherein said imagery data is replaced with said replacement data only if said eyelid is open.

9. The method according to claim 1, further comprising, for each eye, determining a characteristic gaze direction of said eye, wherein said imagery data is replaced with said replacement data only if said gaze direction is other than a predetermined gaze direction.

10. The method according to claim 1, further comprising, processing the image to replace imagery data associated with each location of each eye with additional replacement data to provide at least one additional corrected image such that each corrected image is characterized by a different gaze, and transmitting each corrected image to a different display device.

11. The method according to claim 10, further comprising for each eye, determining a characteristic gaze direction of said eye, wherein said at least one additional corrected image is generated responsively to said determined characteristic gaze direction.

22. The method according to claim 1, wherein said extraction of said locations comprises, for each eye:

detecting eye corners, defining a region-of-interest based on said corners and scaling said region-of-interest to predetermined dimension.

23. The method according to claim 1, wherein said previously-recorded imagery data is stored in a computer-accessible specific database of imagery data, and wherein said replacement of said imagery data comprises searching over said specific database for said previously-recorded imagery data.

24. The method of claim 23, wherein said searching comprises comparing imagery data of the image with imagery data entries in said specific database, determining a similarity level responsively to said comparison, and selecting said previously-recorded imagery data based on said similarity level.

25. The method according to claim 1, wherein said image is a frame of a video stream.

26. The method of claim 25, wherein said replacement data are selected from a specific database according to an optimization procedure which comprises a first objective function for reducing differences between said replacement data and replacement data of a previous frame, and a second objective function for reducing differences between said replacement data and replacement data of a frame immediately following an eyelid opening of the respective eye.

12. A computer software product, comprising a non-transitory computer-readable medium in which program instructions are stored, which instructions, when read by a processer, cause the processer to process the image so as to extract locations of at least one eye over the image, to provide a corrected image by replacing imagery data associated with each location of each eye with replacement data being previously-recorded imagery data respectively corresponding to the same eye but a different gaze, said replacing being based, at least in part, on eye parameters defining a shape of said eye and being computed using a non-specific database of eye images, and to transmit said corrected image to a display device.

13. An imaging system, comprising:

(a) a camera for capturing an image of at least one individual having eyes; and

(b) a processer configured for;

processing the image so as to extract locations of at least one eye over the image, processing the image to replace imagery data associated with each location of each eye with replacement data being previously-recorded imagery data respectively corresponding to the same eye but a different gaze, wherein said replacing is based, at least in part, on eye parameters defining a shape of said eye and being computed using a non-specific database of eye images, thereby providing a corrected image; and

transmitting said corrected image to a display device.

14. The system of claim 13, wherein said previously-recorded imagery data are processed by the processor so as to compute eye parameters wherein said imagery data are replaced based on said computed parameters.

15. The system of claim 13, wherein said computation of said parameters is bootstrapped using an annotated non-specific database of eye images.

16. The system according to claim 14, wherein said previously-recorded data are transformed by said processor using a morphological transformation according to said parameters.

17. The system according to claim 14, wherein said parameters are varied by said processor so as to optimize a predetermined objective function describing matching between said replaced data and said replacement data.

18. The system according to claim 13, wherein the image is processed by said processor to determine whether an eyelid of said eye is close or open, and wherein said imagery data is replaced with said replacement data only if said eyelid is open.

19. The system according to claim 13, wherein the image is processed by said processor to determine a characteristic gaze direction of said eye wherein said imagery data is replaced with said replacement data only if said characteristic gaze direction is other than a predetermined gaze direction.

20. The system according to claim 13, wherein the image is processed by said processor to replace imagery data associated with each location of each eye with additional replacement data to provide at least one additional corrected image, such that each corrected image is characterized by a different gaze, and transmitting each corrected image to a different display device.

21. The system according to claim 20, wherein the image is processed by said processor to determine a characteristic gaze direction of said eye, wherein said at least one additional corrected image is generated responsively to said determined characteristic gaze direction.