METHOD FOR CONTROLLING THE DISPLAYING OF TEXT FOR AIDING READING ON A DISPLAY DEVICE, AND APPARATUS ADAPTED FOR CARRYING OUT THE METHOD, COMPUTER PROGRAM, AND COMPUTER READABLE STORAGE MEDIUM

• US 20170228018A1
• Filed: 10/02/2015
• Published: 08/10/2017
• Est. Priority Date: 10/13/2014
• Status: Active Grant

• Current Assignee: Thomson Licensing (Vantiva SA)
• Original Assignee: Thomson Licensing SA (Vantiva SA)
• Inventors: SCHAEFERJOHANN, Volker
• Granted Patent: US 10,452,136 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/0483 Interaction with page-structured environments, e.g. book metaphor
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G06V 40/197 Matching; Classification

Abstract

This invention aims to support the reader in the task of reading text on a display device (21). In each case an area in the text to which the gaze is directed (view areas/focus area of the eyes) is highlighted, respectively. For this purpose the gaze direction is determined (101 to 108) in the reading device (21) and the highlighted region is controlled with gaze direction control (109). When a whole sentence is highlighted, the highlighting is stopped and starts anew with the beginning of the next sentence being read. The highlighted area moves with this technique with the view. The reader'"'"'s eye/brain is thus released from the complicated task of line tracking and can better concentrate on word recognition.

Claims

1. A method for controlling the displaying of text for aiding reading on a display device, comprising the steps of displaying the text on a display screen, determining the view direction of a user while reading text on the display screen, highlighting text in a segment of the display screen which corresponds to the determined view direction, and tracking the user'"'"'s view direction by successively determining the view direction of the user while reading text on the display screen and successively highlighting text in the corresponding segments of the display screen, wherein the highlighting of text during view direction tracking is started at the beginning of a sentence when the user'"'"'s view is directed to the corresponding segment of the display screen and wherein successively more and more sections of said sentence are highlighted such that when the user'"'"'s view is directed to the end of said sentence the whole sentence is highlighted on the display screen.

2. The method according to claim 1, wherein, when the user'"'"'s view is directed to the beginning of a following sentence in the text on screen, the highlighting of the previous sentence is cancelled and the highlighting of the following sentence is started by highlighting the first section of the following sentence.

3. The method according to claim 1, wherein the front of the highlighted text area precedes the determined view direction and a successive gaze direction by one or more segments of the display screen.

4. The method according to claim 1, wherein the view direction tracking is performed successively from time to time or periodically with a certain time distance and wherein the highlighting region advances with a determined speed in the interval between two successive view direction tracking steps.

5. The method according to claim 4, wherein the speed with which the highlight region advances in the interval between two successive view direction tracking steps is derived from the determined speed in one or more of the previous intervals.

6. The method according to claim 1, wherein for view direction determination an image is captured from the user'"'"'s face with a camera, and the view direction determination is made with the help of a step of dedicated image processing.

7. The method according to claim 6, wherein in the step of dedicated image processing the display screen area is segmented into a plurality of segments and with the help of a dedicated step of head position processing it is determined to which segment the user'"'"'s gaze, is directed to, wherein during a training phase with the user in which the user is asked to look at certain segments of the display screen and where in each case a corresponding image is captured of which in each case the corresponding head position is extracted and archived, and wherein the best matching archived head position determines to which segment the user'"'"'s gaze is directed to.

8. The method according to claim 7, wherein the head position processing includes a step of contour extraction in the captured image and a step of comparing the extracted contour with archived contours which have been extracted and archived during said training phase, wherein the best matching archived contour image determines to which segment the user'"'"'s gaze is directed to.

9. The method according to claim 8, wherein the contour images are subdivided into at least two parts (L, R) and a comparison operation is executed respectively for each part and wherein it is verified that the view direction determination with the at least two parts results in the same segment determination before highlighting text on the display screen.

10. The method according to claim 7, wherein the head position processing includes a step of “

T”

-symbol extraction in the captured image and a step of comparing the extracted “

T”

-symbol with archived “

T”

-symbols which have been extracted and archived during said training phase, wherein the best matching archived “

T”

-symbol image determines to which segment the user'"'"'s gaze is directed to.

11. The method according to claim 10, wherein for “

T”

-symbol extraction a first line is drawn connecting the determined centers of the user'"'"'s eyes in the captured image and a second line is drawn along the back of the user'"'"'s nose in the captured image.

12. The method according to claim 7, wherein in addition to said head position processing an eye position processing step is performed and a step of comparing an extracted eye position with archived eye positions is performed for increasing the accuracy of gaze direction determination.

13. The method according to claim 12, wherein for eye position processing an additional training phase is absolved with the user in which the user is asked to look straight at a dedicated segment of the screen and to look at a plurality of segments at the border of the display screen without moving the user'"'"'s head, wherein in each case a corresponding image is captured of which in each case a corresponding eye position is extracted and archived.

14. The method according to claim 12, wherein the extracted eye position is compared with the archived eye positions and wherein the best matching archived eye position determines correction coordinates by which the view direction determination based on head position detection is to be corrected to result in a more accurate gaze direction determination.

15. The method according to claim 12, wherein an interpolation is carried out between a best matching archived eye position and the straight looking eye position from the additional training phase, for determining the correction coordinates for the extracted eye position.

16. An apparatus adapted for the use of the method according to claim 1, comprising a central processing unit (CPU), random access memory (MEM) and a non-volatile memory (HDD), a display device or an interface (IF) for a display device, wherein said archived head positions, contours, “

T”

-symbols or eye positions are recorded in the non-volatile memory (HDD), and the central processing unit (CPU) is configured to perform the tasks of comparing an extracted head position, contour, “

T”

-symbol or eye position with said archived head position, contour, “

T”

-symbol or eye position, and to determine the best match for gaze direction determination.

17. The apparatus according to claim 16, further comprising a graphical processing unit (GPU), wherein the tasks of comparing an extracted head position, contour, “

T”

-symbol or eye position with said archived head position, contour, “

T”

-symbol or eye position and determining the best match for gaze direction determination are performed solely in said graphical processing unit (GPU) or in the central processing unit (CPU) and the graphical processing unit (GPU) in a cooperative manner.

19. A computer program comprising computer-executable instructions when run in a computer system adapted to cause the computer system to perform the method according to claim 1.

18. A computer readable storage medium having stored therein instructions enabling the performance of the steps of displaying text on the display screen, determining the view direction of the user while reading text on the display screen, highlighting text in a segment of the display screen which corresponds to the determined view direction, and wherein in the following a tracking of the user'"'"'s view direction is performed by successively determining the view direction of the user while reading text on the display screen and successively highlighting text in the corresponding segments of the display screen.