Method for controlling the displaying of text for aiding reading on a display device, and apparatus adapted for carrying out the method, computer program, and computer readable storage medium

• US 10,452,136 B2
• Filed: 10/02/2015
• Issued: 10/22/2019
• Est. Priority Date: 10/13/2014
• Status: Expired due to Fees

• Current Assignee: Thomson Licensing (Vantiva SA)
• Original Assignee: Thomson Licensing (Vantiva SA)
• Inventors: Schaeferjohann, Volker
• Primary Examiner(s): Azongha, Sardis F
• Application Number:
  US15/518,476
• Publication Number:
  US 20170228018A1
• Time in Patent Office: 1,481 Days
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/0483 Interaction with page-structured environments, e.g. book metaphor
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G06V 40/197 Matching; Classification

Abstract

This invention aims to support the reader in the task of reading text on a display device (21). In each case an area in the text to which the gaze is directed (view areas/focus area of the eyes) is highlighted, respectively. For this purpose the gaze direction is determined (101 to 108) in the reading device (21) and the highlighted region is controlled with gaze direction control (109). When a whole sentence is highlighted, the highlighting is stopped and starts anew with the beginning of the next sentence being read. The highlighted area moves with this technique with the view. The reader'"'"'s eye/brain is thus released from the complicated task of line tracking and can better concentrate on word recognition.

Claims

1. A method for controlling the displaying of text for aiding reading on a display device, comprising the steps of:

displaying the text on a display screen;

determining, using image processing, a view direction of a user while reading text on the display screen by capturing an image of the user'"'"'s face by a camera, the image processing including segmenting a display screen area into a plurality of segments and determining, using head position processing, to which segment the user'"'"'s gaze is directed, the head position processing including extracting a contour in the captured image and comparing the extracted contour in the captured image with archived contours which have been extracted and archived during a training phase, a best matching archived contour image determines to which segment the user'"'"'s gaze is directed to;

highlighting text in a segment of the display screen which corresponds to the determined view direction; and

tracking the user'"'"'s view direction by successively determining the view direction of the user while reading text on the display screen and successively highlighting text in the corresponding segments of the display screen, wherein the highlighting of text during view direction tracking is started at the beginning of a sentence when the user'"'"'s view is directed to the corresponding segment of the display screen, wherein successively more and more sections of said sentence are highlighted such that when the user'"'"'s view is directed to the end of said sentence the whole sentence is highlighted on the display screen, and wherein the training phase includes;

asking the user to look at certain segments of the display screen;

capturing a corresponding image; and

extracting and archiving the contour corresponding to the head position in the captured image.

2. The method according to claim 1, wherein, when the user'"'"'s view is directed to the beginning of a following sentence in the text on the display screen, the highlighting of the previous sentence is canceled and highlighting of the following sentence is started by highlighting the first section of the following sentence.

3. The method according to claim 1, wherein a front of the highlighted text area precedes the determined view direction and a successive gaze direction by one or more segments of the display screen.

4. The method according to claim 1, wherein the view direction tracking is performed successively from time to time or periodically with a certain time distance and wherein a highlighting region advances with a determined speed in an interval between two successive view direction tracking steps.

5. The method according to claim 4, wherein the determined speed with which the highlight region advances in the interval between two successive view direction tracking steps is derived from a determined speed in one or more of the previous intervals.

6. The method according to claim 1, wherein the head position processing includes a step of “

T”

-symbol extraction in the captured image and a step of comparing the extracted “

T”

-symbol with archived “

T”

-symbols which have been extracted and archived during said training phase, wherein a best matching archived “

T”

-symbol image determines to which segment the user'"'"'s gaze is directed to.

7. The method according to claim 6, wherein for “

T”

-symbol extraction a first line is drawn connecting determined centers of the user'"'"'s eyes in the captured image and a second line is drawn along a back of the user'"'"'s nose in the captured image.

8. The method according to claim 1, wherein in addition to said head position processing an eye position processing step is performed and a step of comparing an extracted eye position with archived eye positions is performed for increasing the accuracy of gaze direction determination.

9. The method according to claim 8, wherein for eye position processing an additional training phase is absolved with the user in which the user is asked to look straight at a dedicated segment of the display screen and to look at a plurality of segments at a border of the display screen without moving the user'"'"'s head, wherein in each case a corresponding image is captured of which in each case a corresponding eye position is extracted and archived.

10. The method according to claim 8, wherein the extracted eye position is compared with the archived eye positions and wherein a best matching archived eye position determines correction coordinates by which the view direction determination based on head position detection is to be corrected to result in a more accurate gaze direction determination.

11. The method according to claim 8, wherein an interpolation is carried out between a best matching archived eye position and a straight looking eye position from an additional training phase, for determining the correction coordinates for the extracted eye position.

12. A non-transitory computer readable storage medium having computer readable programming code stored thereon to perform a method, the method comprising:

displaying the text on a display screen;

determining, using image processing, a view direction of a user while reading text on the display screen by capturing an image of the user'"'"'s face by a camera, the image processing including segmenting a display screen area into a plurality of segments and determining, using head position processing, to which segment the user'"'"'s gaze is directed, the head position processing including extracting a contour in the captured image and comparing the extracted contour in the captured image with archived contours which have been extracted and archived during a training phase, a best matching archived contour image determines to which segment the user'"'"'s gaze is directed to;

highlighting text in a segment of the display screen which corresponds to the determined view direction;

and tracking the user'"'"'s view direction by successively determining the view direction of the user while reading text on the display screen and successively highlighting text in the corresponding segments of the display screen, wherein the highlighting of text during view direction tracking is started at the beginning of a sentence when the user'"'"'s view is directed to the corresponding segment of the display screen, wherein successively more and more sections of said sentence are highlighted such that when the user'"'"'s view is directed to the end of said sentence the whole sentence is highlighted on the display screen, and wherein the training phase includes;

asking the user to look at certain segments of the display screen;

capturing a corresponding image;

and extracting and archiving the contour corresponding to the head position in the captured image.