Display system, method, and computer program product for capturing images using multiple integrated image sensors

• US 8,587,682 B2
• Filed: 02/12/2011
• Issued: 11/19/2013
• Est. Priority Date: 02/12/2010
• Status: Active Grant

• Current Assignee: NVIDIA Corporation
• Original Assignee: NVIDIA Corporation
• Inventors: Motta, Ricardo J.
• Primary Examiner(s): Ye, Lin
• Assistant Examiner(s): DAGNEW, MEKONNEN D
• Application Number:
  US13/026,238
• Publication Number:
  US 20110199497A1
• Time in Patent Office: 1,011 Days
• Field of Search: 348/218.1, 348 36- 39, 348/333.11, 348/333.12
• US Class Current: 348/218.1
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/366 using viewer tracking
• CPC Class Codes: H04N 7/144 camera and display on the same optical axis, e.g. optically multiplexing the camera and display for eye to eye contact
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

A display system, method, and computer program product are provided for capturing images using multiple integrated image sensors. The display system includes a front panel for displaying an image. The display system further includes a matrix of image sensors situated behind the front panel.

Claims

1. A display apparatus, comprising:

a front panel for displaying an image to a first user situated in front of the front panel; and

a pipeline including;

a matrix of image sensors situated behind the front panel for capturing a plurality of images of the first user situated in front of the front panel, and

a correlation and tracking module which is adapted to;

identify a point of view of a second user to which a single final image of the first user to be generated based on the plurality of captured images is to be displayed, wherein the point of view of the second user is identified from a remote system capturing an image of the second user,

trace rays from the point of view of the second user through each of the image sensors to identify pixels of the images captured by the image sensors that coincide with the point of view of the second user, and

generate the single final image of the first user using the identified pixels for display to the second user, such that the single final image of the first user is specific to the point of view of the second user.

2. The display apparatus of claim 1, wherein the display apparatus includes a liquid crystal display (LCD) system.

3. The display apparatus of claim 1, wherein the display apparatus includes a fluorescent glass display system.

4. The display apparatus of claim 1, wherein the display apparatus includes an active matrix organic light emitting diode (AMOLED) display system.

5. The display apparatus of claim 1, wherein the front panel includes a transparent color liquid crystal display (LCD) panel.

6. The display apparatus of claim 1, wherein the front panel includes a thin film transistor (TFT) LCD panel.

7. The display apparatus of claim 1, wherein the image sensors each include a camera module.

8. The display apparatus of claim 1, wherein each of the image sensors is operable to capture a respective one of the plurality of images through the front panel.

9. The display apparatus of claim 8, further comprising a controller to synchronize the image sensors to capture the plurality of images through the front panel.

10. The display apparatus of claim 1, wherein the image sensors are located on a back panel.

11. The display apparatus of claim 10, wherein the back panel further includes a plurality of light emitting diodes (LEDs) for displaying the image to the first user.

12. The display apparatus of claim 11, wherein the image sensors are interspersed among the LEDs.

13. The display apparatus of claim 12, wherein the LEDs are configured in a matrix on the back panel, such that lattices exist between the LEDs.

14. The display apparatus of claim 13, wherein the image sensors occupy positions in at least a portion of the lattices existing between the LEDs.

15. The display apparatus of claim 11, further comprising a shutter situated between the front panel and the image sensors.

16. The display apparatus of claim 15, wherein the shutter is operable in a diffuse state to diffuse an illumination provided by the LEDs.

17. The display apparatus of claim 15, wherein the shutter is operable in a transparent state to allow the image sensors to capture the plurality of images through the front panel.

18. The display apparatus of claim 17, wherein the image sensors are operable to capture the plurality of images when the shutter is in the transparent state.

19. The display apparatus of claim 1, further including a controller operable to switch the display apparatus between a display state in which the front panel displays the image to the first user, and a camera state in which the image sensors situated behind the front panel capture the plurality of images of the first user through the front panel.

20. The display apparatus of claim 19, wherein the plurality of images are captured through the front panel at regular intervals between the display of image frames.

21. A method, comprising:

identifying a transparent state of a front panel utilized for displaying an image to a first user situated in front of the front panel;

capturing a plurality of images of the first user situated in front of the front panel utilizing a matrix of image sensors situated behind the front panel, in response to the identification of the transparent state;

identifying a point of view of a second user to which a single final image of the first user to be generated based on the plurality of captured images is to be displayed, wherein the point of view of the second user is identified from a remote system capturing an image of the second user;

tracing rays from the point of view of the second user through each of the image sensors to identify pixels of the images captured by the image sensors that coincide with the point of view of the second user; and

generating the single final image of the first user using the identified pixels for display to the second user, such that the single final image of the first user is specific to the point of view of the second user.

22. A computer program product embodied on a non-transitory computer readable medium, comprising:

computer code for identifying a transparent state of a front panel utilized for displaying an image to a first user situated in front of the front panel;

computer code for capturing a plurality of images of the first user situated in front of the front panel utilizing a matrix of image sensors situated behind the front panel, in response to the identification of the transparent state;

identifying a point of view of a second user to which a single final image of the first user to be generated based on the plurality of captured images is to be displayed, wherein the point of view of the second user is identified from a remote system capturing an image of the second user;

tracing rays from the point of view of the second user through each of the image sensors to identify pixels of the images captured by the image sensors that coincide with the point of view of the second user; and

computer code for the single final image of the first user using the identified pixels for display to the second user, such that the single final image of the first user is specific to the point of view of the second user.