Eye tracking enabled smart closed captioning

• US 10,447,960 B2
• Filed: 02/13/2017
• Issued: 10/15/2019
• Est. Priority Date: 03/25/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Wilairat, Weerapan, Thukral, Vaibhav
• Primary Examiner(s): Zhao, Daquan
• Application Number:
  US15/430,705
• Publication Number:
  US 20170155868A1
• Time in Patent Office: 974 Days
• Field of Search: 386244, 386239, 386278, 386280, 386282
• CPC Class Codes: G06F 2203/04804 Transparency, e.g. transparent or translucent windows
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/44218 Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45
• CPC Class Codes: H04N 21/47 End-user applications
• CPC Class Codes: H04N 21/4884 for displaying subtitles
• CPC Class Codes: H04N 5/85 on discs or drums

Abstract

Systems and methods for controlling closed captioning using an eye tracking device are provided. The system for controlling closed captioning may comprise a display device, a closed captioning controller configured to display closed captioning text for a media item during playback on the display device, and an eye tracking device configured to detect a location of a user'"'"'s gaze relative to the display device and send the location to the closed captioning controller. The closed captioning controller may be configured to recognize a predetermined gaze pattern of the user'"'"'s gaze and, upon detecting the predetermined gaze pattern, partially or completely deemphasize the display of the closed captioning text.

Claims

1. A system for controlling closed captioning, comprising:

a display device;

a closed captioning controller; and

an eye tracking device configured to detect a location of a user'"'"'s gaze relative to the display device, and send the location to the closed captioning controller;

wherein the closed captioning controller is configured to recognize a predetermined gaze pattern of the user'"'"'s gaze and, upon recognizing the predetermined gaze pattern, partially or completely deemphasize a display of closed captioning text;

wherein the closed captioning controller is further configured to;

detect the user'"'"'s gaze upon a region in which a character is displayed on the display device;

detect an eye gaze gesture in which the user'"'"'s gaze is detected as moving from the region in which the character is displayed to a region away from the character that is outside a predetermined closed captioning display region in which the closed captioning text is displayed; and

in response to detecting the eye gaze gesture, display closed captioning text corresponding to words spoken by the character.

2. The system of claim 1, wherein, to recognize the predetermined gaze pattern, the closed captioning controller is further configured to determine whether or not the location of the user'"'"'s gaze is within the predetermined closed captioning display region, and if the user'"'"'s gaze is not within the predetermined closed captioning display region for longer than a predetermined period of time, deemphasize the display of the closed captioning text in the predetermined closed captioning display region.

3. The system of claim 2, wherein, following deemphasizing of the closed captioning text, the closed captioning controller is further configured to reemphasize the display of the closed captioning text in the predetermined closed captioning display region if the user'"'"'s gaze is within the predetermined closed captioning display region for longer than the predetermined period of time.

4. The system of claim 2, wherein, to recognize the predetermined gaze pattern, the closed captioning controller is further configured to detect, based on information including a direction and rate of change of the user'"'"'s gaze, that the user'"'"'s eye gaze is traveling within the predetermined closed captioning display region in a reading direction at a speed that is within a reading speed range.

5. The system of claim 4, wherein the closed captioning controller is further configured to deemphasize the closed captioning text by decreasing an opacity of the closed captioning text in the predetermined closed captioning display region if the user'"'"'s gaze is within the predetermined closed captioning display region but the speed of the user'"'"'s gaze is detected to be outside the reading speed range.

6. The system of claim 4, wherein the closed captioning controller is further configured to monitor the speed of the user'"'"'s gaze within the predetermined closed captioning display region and display auxiliary information regarding a word or phrase of the closed captioning text if the controller detects that the speed of the user'"'"'s gaze slows down below a predetermined slow reading speed threshold or pauses for at least a predetermined dwell time on the word or a word in the phrase in the closed captioning text.

7. The system of claim 6, wherein the closed captioning controller is further configured to:

prior to displaying the auxiliary information, alter at least one of a size or font of the word or phrase in the closed captioning text if the controller detects that the speed of the user'"'"'s gaze slows down below the predetermined slow reading speed threshold or pauses for at least the predetermined dwell time on the word or the word in the phrase in the closed captioning text.

8. The system of claim 2, wherein the closed captioning controller is further configured to:

monitor a distance between the user and the display device;

increase a size of the closed captioning text if the distance increases; and

decrease the size of the closed captioning text if the distance decreases.

9. The system of claim 2, wherein the closed captioning controller is further configured to:

define a plurality of adjacent subregions including a prior caption subregion and a current caption subregion within the predetermined closed captioning display region;

display a current caption of the closed captioning text in the current caption subregion;

upon detecting the user'"'"'s gaze within the prior caption subregion, display a previous caption of the closed captioning text and deemphasize the current caption in the current caption subregion; and

upon detecting the user'"'"'s gaze within the current caption subregion, deemphasize the prior caption in the prior caption subregion and reemphasize the current caption in the current caption subregion.

10. The system of claim 1, wherein the region away from the character is below the character.

11. A method for controlling closed captioning, comprising:

detecting a location of a user'"'"'s gaze relative to a display device;

recognizing a predetermined gaze pattern of the user'"'"'s gaze;

upon recognizing the predetermined gaze pattern, partially or completely deemphasizing a display of closed captioning text;

detecting the user'"'"'s gaze upon a region in which a character is displayed on the display;

detecting an eye gaze gesture in which the user'"'"'s gaze is detected as moving from the region in which the character is displayed to a region away from the character that is outside a predetermined closed captioning display region in which the closed captioning text is displayed; and

in response to detecting the eye gaze gesture, displaying closed captioning text corresponding to words spoken by the character.

12. The method of claim 11, further comprising:

wherein recognizing the predetermined gaze pattern includes determining whether or not the location of the user'"'"'s gaze is within the predetermined closed captioning display region; and

wherein partially or completely deemphasizing the display of the closed captioning text includes, if the user'"'"'s gaze is not within the predetermined closed captioning display region for longer than a predetermined period of time, deemphasizing the display of the closed captioning text in the predetermined closed captioning display region.

13. The method of claim 12, further comprising:

reemphasizing the display of the closed captioning text in the predetermined closed captioning display region if the user'"'"'s gaze is within the predetermined closed captioning display region for longer than the predetermined period of time.

14. The method of claim 12, further comprising:

detecting, based on information including a direction and rate of change of the user'"'"'s gaze, that the user'"'"'s eye gaze is traveling within the predetermined closed captioning display region in a reading direction at a speed that is within a reading speed range.

15. The method of claim 14, further comprising:

deemphasizing the closed captioning text by decreasing an opacity of the closed captioning text in the predetermined closed captioning display region if the user'"'"'s gaze is within the predetermined closed captioning display region but the speed of the user'"'"'s gaze is detected to be outside the reading speed range.

16. The method of claim 14, further comprising:

monitoring the speed of the user'"'"'s gaze within the predetermined closed captioning display region and displaying auxiliary information regarding a word or phrase of the closed captioning text if it is detected that the speed of the user'"'"'s gaze slows down below a predetermined slow reading speed threshold or pauses for at least a predetermined dwell time on the word or a word in the phrase in the closed captioning text; and

prior to displaying the auxiliary information, altering at least one of a size or font of the word or phrase in the closed captioning text if it is detected that the speed of the user'"'"'s gaze slows down below the predetermined slow reading speed threshold or pauses for at least the predetermined dwell time on the word or the word in the phrase in the closed captioning text.

17. The method of claim 12, further comprising:

monitoring a distance between the user and the display device;

increasing a size of the closed captioning text if the distance increases; and

decrease the size of the closed captioning text if the distance decreases.

18. The method of claim 12, further comprising:

defining a plurality of adjacent subregions including a prior caption subregion and a current caption subregion within the predetermined closed captioning display region;

displaying a current caption of the closed captioning text in the current caption region;

upon detecting the user'"'"'s gaze within the prior caption subregion, displaying a previous caption of the closed captioning text in the prior caption subregion and deemphasizing the current caption in the current caption region; and

upon detecting the user'"'"'s gaze within the current caption subregion, deemphasizing the prior caption in the prior caption region and reemphasizing the current caption in the current caption region.

19. The method of claim 11, wherein the region away from the character is below the character.

20. A method for controlling closed captioning, comprising:

detecting a location of a user'"'"'s gaze relative to a display device;

detecting the user'"'"'s gaze upon a region in which a character is displayed on the display device;

detecting an eye gaze gesture in which the user'"'"'s gaze is detected as moving from the region in which the character is displayed to a region away from the character that is outside a predetermined closed captioning display region in which closed captioning text is displayed; and

in response to detecting the eye gaze gesture, displaying closed captioning text corresponding to words spoken by the character in the predetermined closed captioning display region on the display device.