Multiple viewpoint image capture of a display user

• US 10,108,258 B2
• Filed: 09/06/2013
• Issued: 10/23/2018
• Est. Priority Date: 09/06/2013
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Sundaram, Arvind, Jothiswaran, Gangatharan
• Primary Examiner(s): Kelley, Christopher S
• Assistant Examiner(s): Walsh, Kathleen M
• Application Number:
  US14/019,983
• Publication Number:
  US 20150070481A1
• Time in Patent Office: 1,873 Days
• Field of Search: 348 78, 34833301
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

Embodiments of the invention describe apparatuses, systems and methods for capturing, selecting and analyzing image data of a user of a computing device display from multiple viewpoints. Embodiments of the invention capture or receive data comprising a plurality of images from an image capture system, wherein the plurality of images comprises images of a user of a computing system from multiple viewpoints. A gaze direction of the user with respect to the display is determined based, at least in part, on the plurality of images. Embodiments further determine a target area on the computing device display based, at least in part, on the determined gaze direction of the user, and execute an application operation in response to determining the target area of the user'"'"'s gaze.

Claims

1. An apparatus comprising:

a plurality of separate image sensors to capture a plurality of images of a user of a display of a computing device from an array of viewpoints included on the display of the computing device, each of the plurality of separate image sensors to capture images of the user from a different viewpoint of the array of viewpoints on the display;

a deflector comprising;

a first side to deflect light away from each of the array of viewpoints included on the display towards a respective image sensor of the plurality of image sensors, and

a second transmissive side, opposite the first side, to allow light from the display to pass through to the user; and

logic to;

identify an eye of the user in each of the plurality of images captured from the viewpoint associated with each of the plurality of image sensors,

determine a gaze direction of the user with respect to the display based, at least in part, on comparison of the identified eye of the user captured in each of the plurality of images, wherein determining the gaze direction of the user includes comparing iris eccentricity of the identified eye of the user captured in each of the plurality of the images of the user to analyze roundness of the iris in each of the plurality of images of the user to determine an image in the plurality of images with less distortion than any other images in the plurality of images as being a viewpoint closest to the gaze direction of the user;

determine a target area of the user'"'"'s gaze on the display based, at least in part, on the determined gaze direction of the user, and

execute an application operation in response to determining the target area of the user'"'"'s gaze.

2. The apparatus of claim 1, wherein the first side of the deflector comprises a Fresnel prism.

3. The apparatus of claim 2, further comprising:

a Fresnel lens disposed over the plurality of image sensors.

19. The apparatus of claim 1, wherein each image sensor of the plurality of image sensors includes a separate complementary metal-oxide semiconductor (CMOS) image sensor.

4. A system comprising:

a display of a computing device;

a plurality of separate image sensors disposed next to the display of the computing device to capture a plurality of images of a user of the display, each of the plurality of image sensors to capture images of the user from a different viewpoint of an array of viewpoints on the display;

a deflector comprising;

a first side to deflect light away from each of the array of viewpoints included on the display towards a respective image sensor of the plurality of image sensors, and

a second transmissive side, opposite the first side, to allow light from the display to pass through to the user; and

logic to;

identify an eye of the user in each of the plurality of images captured from the viewpoint associated with each of the plurality of image sensors;

determine a gaze direction of the user with respect to the display based, at least in part, on comparison of the identified eye of the user captured in each of the plurality of images, wherein determining the gaze direction of the user includes comparing iris eccentricity of the identified eye of the user captured in each of the plurality of the images of the user to analyze roundness of the iris in each of the plurality of images of the user to determine an image in the plurality of images with less distortion than any other images in the plurality of images as being a viewpoint closest to the gaze direction of the user,

determine a target area of the user'"'"'s gaze on the display based, at least in part, on the determined gaze direction of the user, and

execute an application operation in response to determining the target area of the user'"'"'s gaze.

5. The system of claim 4, wherein the plurality of image sensors are disposed below the display of the computing device.

6. The system of claim 4, wherein the plurality of image sensors and the deflector are included in a bezel surrounding the display of the computing device.

7. The system of claim 4, wherein the display of the computing device comprises a tabletop display surface.

8. The system of claim 4, wherein the first side of the deflector comprises a Fresnel prism.

9. The system of claim 8, further comprising:

a Fresnel lens disposed over the plurality of image sensors.

10. The system of claim 4, wherein the plurality of images captured by the plurality of imaging sensors comprises real-time video data, the application comprises a streaming video application, and the application operation comprises selecting the real-time video data corresponding to the target area of the user'"'"'s gaze as an output of the streaming video application.

11. The system of claim 4, wherein the application operation executed in response to determining the target area of the user'"'"'s gaze comprises a user input/output (I/O) operation.

12. The system of claim 11, wherein the application is to display a plurality of click-thru icons, and the user I/O operation comprises selecting one of the click-thru icons.

13. The system of claim 4, wherein the deflector comprises a plurality of regions, the user of the computing device comprises a plurality of users, and the logic to further:

assign one or more of the regions of the deflector to each of the plurality of users.

14. The system of claim 4, further comprising:

an antenna; and

radio frequency circuitry coupled to the antenna to receive signal data to be processed by the system.

20. The system of claim 4, wherein each image sensor of the plurality of image sensors includes a separate complementary metal-oxide semiconductor (CMOS) image sensor.

15. An article of manufacture comprising a computer-readable non-transitory storage medium having instructions stored thereon to cause a processor to perform operations including:

receiving data comprising a plurality of images from an image capture system, wherein the plurality of images comprises images of a user of a computing system captured by a plurality of image sensors from an array of viewpoints on the display, each of the plurality of image sensors to capture images of the user from a different viewpoint of the array of viewpoints on the display, light being deflected away from each of the array of viewpoints towards a respective image sensor of the plurality of image sensors;

identifying an eye of the user in each of the plurality of images captured from the viewpoint associated with each of the plurality of image sensors;

determining a gaze direction of the user with respect to the display based, at least in part, on comparison of the identified eye of the user captured in each of the plurality of images captured, wherein determining the gaze direction of the user includes comparing iris eccentricity of the identified eye of the user captured in each of the plurality of the images of the user to analyze roundness of the iris in each of the plurality of images of the user to determine an image in the plurality of images with less distortion than any other images in the plurality of images as being a viewpoint closest to the gaze direction of the user;

determining a target area on the display based, at least in part, on the determined gaze direction of the user; and

executing an application operation in response to determining the target area of the user'"'"'s gaze.

16. The article of manufacture of claim 15, wherein the plurality of images captured by the plurality of imaging sensors comprises real-time video data, the application comprises a streaming video application, and the application operation comprises selecting the real-time video data corresponding to the target area of the user'"'"'s gaze as an output of the streaming video application.

17. The article of manufacture of claim 15, wherein the application operation executed in response to determining the target area of the user'"'"'s gaze comprises a user input/output (I/O) operation.

18. The article of manufacture of claim 15, wherein the deflector comprises a plurality of regions, the user of the computing device comprises a plurality of users, and the operations further include:

assigning one or more of the regions of the deflector to each of the plurality of users.