Using Eye Tracking to Display Content According to Subject's Interest in an Interactive Display System

• US 20180024633A1
• Filed: 07/21/2017
• Published: 01/25/2018
• Est. Priority Date: 07/21/2016
• Status: Abandoned Application

• Current Assignee: AIVIA, Inc.
• Original Assignee: AIVIA, Inc.
• Inventors: Lo, Chungwen Dennis
• Application Number:
  US15/657,067
• Publication Number:
  US 20180024633A1
• Time in Patent Office: Days
• CPC Class Codes: A47F 13/00 Shop or like accessories devices for drawing-off and cutting paper from a roll B65H; movably-mounted ladders E06C9/06
• CPC Class Codes: A47F 2010/025 using stock management systems
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06F 2203/04803 Split screen, i.e. subdividing the display area or the window area into separate subareas
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/1423 controlling a plurality of local displays, e.g. CRT and flat panel display
• CPC Class Codes: G06Q 30/0241 Advertisements
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/166 using acquisition arrangements
• CPC Class Codes: G06V 40/19 : Sensors therefor
• CPC Class Codes: G06V 40/23 : Recognition of whole body movements, e.g. for sport training
• CPC Class Codes: G09G 2320/0693 : Calibration of display systems
• CPC Class Codes: G09G 2354/00 : Aspects of interface with display user
• CPC Class Codes: G09G 2370/022 : Centralised management of display operation, e.g. in a server instead of locally
• CPC Class Codes: G09G 2380/06 : Remotely controlled electronic signs other than labels
• CPC Class Codes: G09G 5/32 : with means for controlling the display position see provisionally G09G5/42
• CPC Class Codes: H04N 23/69 : Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
• CPC Class Codes: H04N 23/90 : Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 7/183 : for receiving images from a single remote source

Abstract

A system interactively displays content according to subject'"'"'s interest. An interactive display system includes a display and an imaging unit or camera. The interactive display system tracks a subject'"'"'s eyes or head movement to determine a subject'"'"'s interest. Then, the system will analyze the subject'"'"'s behavior and make decisions on what content to display on a screen based on the subject'"'"'s interest.

Claims

1. A method comprising:

receiving first, second, and third content for display on a first display;

storing the first, second, and third content in a memory;

displaying the first content on the first display;

receiving a stream of images from a first imaging device;

analyzing the stream of images from the imaging device to obtain a first analysis; and

based on the first analysis of the images, altering the content shown on the first display to show either the second content or the third content, wherein the content shown on the first display does not comprise images received using the first imaging device.

2. The method of claim 1 wherein the first, second, and third content are received over a network connection.

4. The method of claim 1 wherein the analyzing the stream of images from the imaging device to obtain a first analysis comprises:

in the stream of images, detecting a gaze event of a person, wherein the gaze event indicates a selection by the person'"'"'s eye gaze of either at least a first portion of the first content or a second portion of the first content shown on the display;

upon determining the gaze event is for the first portion, display the second content associated with the first portion on the first display; and

upon determining the gaze event is for the second portion, display a third content associated with the second portion on the first display.

5. The method of claim 1 comprising:

calibrating the first imaging device and the first display by using a point of interest on the first display at about a frame center, between a frame left edge and a frame right edge of the first display and between a frame top edge and a frame bottom edge of the first display.

6. The method of claim 1 comprising:

calibrating the first imaging device and the first display by using a point of interest on the first display moving from a frame left edge to a frame right edge of the first display or the point of interest moving from the frame right edge to the frame left edge of the first display.

7. The method of claim 1 comprising:

calibrating the first imaging device and the first display by using a point of interest at one side of a frame of the first display and then at an opposite side of the frame of the first display.

8. The method of claim 1 comprising:

using a real-time processor, performing an image analysis of gaze click detection, group classification, movement detection, and location estimation.

9. The method of claim 1 wherein the memory comprises embedded storage or external storage, and the storage is used to store content images received from a server for a presenter and a reporter, and method comprises based on image analysis data, determining associated display content.

10. The method of claim 1 wherein the image analysis comprises determining a gaze duration, estimate face location, and generate a gaze_click flag when a duration is greater than a predetermined threshold time value.

11. The method of claim 1 wherein the image analysis comprises detecting a movement of a person'"'"'s eyes, a movement of a person'"'"'s head, and movement of a person'"'"'s body, a person'"'"'s gender, a person'"'"'s age, a person'"'"'s movement behavior or patterns, a person'"'"'s distance from first display, a person'"'"'s hair color, a person'"'"'s clothing color, a person'"'"'s clothing style such as pants, skirt, or other, appearance, posture, face recognition or face tracking, or any combination of these.

12. The method of claim 1 wherein the altering the content shown on the display is enabled by generating of a gaze_click_through flag which comprises a gaze_click and weighting factors comprising at least one of a specific gender, age group, specific area, distance, preferred viewers, or other factors, or any combination of these.

13. The method of claim 1 comprising migrating altered content from a primary content group to secondary content group to match to a classified viewer'"'"'s group.

14. The method of claim 1 comprising updating a content size, including text font size, of the altered content according to a viewer'"'"'s distance, content color to match viewer'"'"'s dress color, content waving to get attention, content still from waving, or different content, or any combination of these.

15. The method of claim 1 wherein the imaging device positioned in a separate location than the first display.

16. The method of claim 1 wherein the imaging device is housed in at least one of a mannequin, a merchandise, a holder, or a stand, separate from the display.

17. The method of claim 1 wherein the imaging device incorporates a motor to rotate the imaging device itself or to rotate a front mirror to increase its field of view.

18. The method of claim 1 wherein multiple display units and imaging units are linked together, such that when a subject moves from a display unit A'"'"'s coverage to a display unit B'"'"'s coverage, a subject'"'"'s eye is tracked such that display unit B will display content that related to what was displayed in unit A when the subject'"'"'s gaze was detected in unit A.

19. The method of claim 1 wherein for each instance content is displayed on the first display, captured images associated with the content on the first display are analyzed to determine an interest level, where lower interest content will be replaced with content similar to high interest content either in a single display or using multiple display units.

20. The method of claim 1 wherein the image analysis comprises detecting a gaze blinking which is used to determine if a real human.

3. A method comprising:

receiving first, second, third, fourth, fifth, and sixth content for display on a first display;

storing the first, second, third, fourth, fifth, and sixth content in a memory;

displaying the first content on the first display;

displaying the second content on the second display;

receiving a stream of images from a first imaging device;

analyzing the stream of images from the imaging device to obtain a first analysis;

based on the first analysis of the images, altering the content shown on the first display to show either the third content or the fourth content, wherein the content shown on the first and second displays does not comprise images acquired received using the first imaging device; and

based on the first analysis of the images, altering the content shown on the second display to show either the fifth content or the sixth content, wherein the second display is separate from the first display.