Image capturing and display apparatus and method

• US 20110304532A1
• Filed: 05/25/2011
• Published: 12/15/2011
• Est. Priority Date: 05/27/2010
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Suh, Sung Joo, Choi, Chang Kyu, Yi, Kwon Ju, Park, Du Sik
• Granted Patent: US 9,549,134 B2
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: G09G 3/3406 Control of illumination source illumination devices structurally associated with liquid crystal cells G02F1/1336
• CPC Class Codes: H04N 23/12 with one sensor only
• CPC Class Codes: H04N 25/00 Circuitry of solid-state image sensors [SSIS]; Control thereof
• CPC Class Codes: H04N 9/12 Picture reproducers H04N9/11 takes precedence

Abstract

Provided is a display apparatus and method. The display apparatus may sense light reflected from an object and passed through a display panel, and may control a power of a backlight unit depending on whether the light has passed through the display panel.

Claims

1. A display apparatus, comprising:

a sensor panel configured to sense a first incident light, incident to the sensor panel subsequent to passing through a display, and to at least partially block transmission of light through the sensor panel; and

a controller to selectively control the display to be configured to pass the first incident light toward the sensor panel and to control the display to be configured to pass light out of the display to display an image,

wherein the sensor panel comprises plural distinct portions configured to each forward light from the sensor panel towards the display.

2. The display apparatus of claim 1, wherein the controller selectively controls the display to only pass the first incident light toward the sensor panel in a first cycle, of plural fixed time length cycles, and controls the display to only pass the light out of the display to display the image in a second cycle, of the plural fixed time length cycles.

3. The display apparatus of claim 2, wherein the fixed time is a time it takes for the display to perform one refresh.

4. The display apparatus of claim 1, wherein, when the controller selectively controls plural images to be captured from plural sensings of respective first incident lights, representing a capture rate, from a pattern of the display controlled by the controller to not change upon the controller controlling the display to display the image, the controller is configured to control the capture rate plus a display rate, for displaying plural images, to be greater than a maximum refresh rate of the display.

5. The display apparatus of claim 1, wherein the controller selectively controls opacities of pixels and/or sub-pixels of the display to produce patterns on the display for displaying the image on the display, in a second mode, and to form at least one pattern on the display controlled to be transparent and an area of the display surrounding the at least one pattern controlled to be opaque, to capture an image by at least one sensor unit of the sensor, in a first mode, wherein the display is controlled by the controller differently in the first mode and the second mode.

6. The display apparatus of claim 5, wherein the controller determines a depth of an object represented in the first incident light based upon at least one detected size of a blur of the at least one pattern detected by the sensor panel.

7. The display apparatus of claim 6, wherein the controller determines plural depth measurements of the object represented in plural first incident lights each sensed by the sensor panel based respectively upon at least one detected size of a blur pattern for each sensed incident light, and generates 3D information of the object.

8. The display apparatus of claim 7, wherein the controller controls a monitoring of movement by the object over plural captured images and respectively generated plural 3D information and determines whether the monitored movement and generated plural 3D information indicates that a gesture, of plural defined gestures, is being made by the movement of the object.

9. The display apparatus of claim 6, wherein the controller restores the captured image by calculating a distance-based impulse response based on the determined depth and by performing devolution of the captured image using the distance-based impulse response to compensate for refraction and/or geometric effects caused by the at least one pattern of the display.

10. The display apparatus of claim 1, further comprising:

a display panel, as the display; and

a backlight unit to emit a second light towards the sensor panel,

wherein the controller selectively controls the display panel for passing the first incident light toward the sensor panel, in a first mode, and controls the display panel for passing the second light through the display panel to display the image, in a second mode.

11. The display apparatus of claim 10, wherein the plural distinct portions each are configured to always forward the second light from the backlight unit towards the display unit.

12. The display apparatus of claim 10, wherein the controller controls the backlight unit to emit light in the second mode and to not emit light during the first mode.

13. The display apparatus of claim 10, wherein the display device is one of a television, a personal computer system, a tablet computer device, a mobile phone, a PDA, teleconferencing devices or system, or set-top box, with each including at least one processor and memory, an encoder to encode an image captured by the sensing by the sensor panel in the first mode, a decoder to at least decode image data received from an external source, and with the controller forwarding decoded image data from the decoder to the display panel and controlling the display panel to display the decoded image data in the second mode.

14. The display apparatus of claim 13, wherein the controller selectively controls opacities of pixels and/or sub-pixels of the display panel to produce patterns on the display panel for displaying the image on the display panel, in a second mode, and to form at least one pattern on the display panel controlled to include at least one transparent pixel or sub-pixel and an area of the display panel surrounding the at least one pattern controlled to be opaque, to capture an image by at least one sensor unit of the sensor panel, in a first mode, wherein the display panel is controlled by the controller differently in the first mode and the second mode.

15. The display apparatus of claim 13, wherein the controller controls a monitoring of movement of an object, represented in the first incident light, over plural captured images and respectively generated plural 3D information for the object and determines whether the monitored movement and generated plural 3D information indicates that a gesture, of plural defined gestures, is being made by the movement of the object.

16. The display apparatus of claim 13 wherein the controller restores the captured image by calculating a distance-based impulse response based on a determined depth of an object, represented in the first incident light, and by performing devolution of the captured image using the distance-based impulse response to compensate for refraction and/or geometric effects caused by a pattern of the display panel from where the first incident light passed through the display panel,

wherein the determined depth of the object is calculated by the controller based upon at least one detected size of a blur of the first incident light, as incident on the sensor panel.

17. The display apparatus of claim 1, wherein, when the controller selectively controls plural images to be captured from plural sensings of respective first incident lights, representing a capture rate, from a pattern of the display controlled by the controller to not change upon the controller controlling the display to display the image, the controller is configured to control the capture rate plus a display rate, for displaying plural images, to be greater than a maximum refresh rate of the display.

18. The display apparatus of claim 1, further comprising:

a display panel, as the display; and

plural backlight units respectively being the plural distinct portions of the sensor panel, each to respectively emit a second light towards the display panel,

wherein the controller selectively controls the display panel to pass the first incident light toward the sensor panel in a first mode and controls the display panel to pass each of the second lights through the display panel in a second mode.

19. The display apparatus of claim 18, wherein the controller controls each of the plural backlight units to emit light in the second mode and to not emit light during the first mode.

20. The display apparatus of claim 18, wherein the display device is one of a television, a personal computer system, a tablet computer device, a mobile phone, a PDA, teleconferencing devices or system, or set-top box, with each including at least one processor and memory, an encoder to encode an image captured by the sensing by the sensor panel in the first mode, a decoder to at least decode image data received from an external source, and with the controller forwarding decoded image data from the decoder to the display panel and controlling the display panel to display the decoded image data in the second mode.

21. The display apparatus of claim 20, wherein the controller selectively controls opacities of pixels and/or sub-pixels of the display panel to produce patterns on the display panel for displaying the image on the display panel, in a second mode, and to form at least one pattern on the display panel controlled to include at least one transparent pixel or sub-pixel and an area of the display panel surrounding the at least one pattern controlled to be opaque, to capture an image by at least one sensor unit of the sensor panel, in a first mode, wherein the display panel is controlled by the controller differently in the first mode and the second mode.

22. The display apparatus of claim 20, wherein the controller controls a monitoring of movement of an object, represented in the first incident light, over plural captured images and respectively generated plural 3D information for the object and determines whether the monitored movement and generated plural 3D information indicates that a gesture, of plural defined gestures, is being made by the movement of the object.

23. The display apparatus of claim 20, wherein the controller restores the captured image by calculating a distance-based impulse response based on a determined depth of an object, represented in the first incident light, and by performing devolution of the captured image using the distance-based impulse response to compensate for refraction and/or geometric effects caused by a pattern of the display panel from where the first incident light passed through the display panel,

wherein the determined depth of the object is calculated by the controller based upon at least one detected size of a blur of the first incident light, as incident on the sensor panel.

24. The display apparatus of claim 1, further comprising a backlight unit to generate the light to be forwarded by the plural distinct portions of the sensor panel, wherein the sensor panel comprises:

plural sensor units to sense the first incident light, in a first mode; and

plural apertures, respectively as the plural distinct portions, to pass the second light emitted from the backlight unit towards the display in first and second modes, with the controller controlling the display to display an image in the second mode,

wherein the plural sensor units and plural apertures are arranged in a grid pattern across the sensor panel.

25. The display apparatus of claim 24, wherein—

the apertures respectively include a diffusers to diffuse the second light.

26. The display apparatus of claim 1, wherein:

the sensor panel comprises a first sensor unit and a second sensor unit to sense the incident first light, and

each of the first sensor unit and the second sensor unit comprise a unit color filter having a different color filter.

27. The display apparatus of claim 26, wherein plural sensor units of the sensor panel are collectively organized in different Bayer patterns.

28. The display apparatus of claim 1, wherein the sensor panel comprises a first sensor unit and a second sensor unit to sense the incident first light, and each of the first sensor unit and the second sensor unit comprise a plurality of color filters respectively filtering different colors.

29. The display apparatus of claim 1, wherein the controller further comprises a sensor image restoring unit to interpolate image data acquired by the sensing of the first incident light and to restore image data that is not captured by any sensing of the sensor panel at the plural distinct portions of the sensor panel.

30. The display apparatus of claim 1, further comprising a display panel, as the display, wherein, in a first mode, the controller controls the display panel to selectively form at least one pattern to pass the first incident light through the display panel to the sensor panel by selectively controlling respective opacities of pixels or sub-pixels of the display panel to be different, controlled differently than when the controller controls the display panel to display the image, in a second mode.

31. The display apparatus of claim 30, wherein the controller controls, at a current time in the first mode, the display panel to have at least one pattern different than a pattern formed at a previous time when the display apparatus was in the first mode, by changing at least one of a number of first pixels of the display panel that are selectively controlled to be opaque during the first mode, a number of second pixels of the display panel that are selectively controlled to be transparent during the first mode, positions of one or more pixels of the display panel that are controlled to be opaque during the first mode, and positions of one or more pixels of the display panel that are controlled to be transparent pixels during the first mode, than in the previous time.

32. The display apparatus of claim 30, wherein the controller selectively controls the display panel to form, as the at least one pattern, at least one of a circular pattern, a polygonal pattern, and a modified uniformly redundant array (MURA) by controlling plural first pixels or plural first sub-pixels to be opaque and plural second pixels or plural second sub-pixels to be transparent within a predetermined area of the display panel.

33. The display apparatus of claim 32, wherein the predetermined area is one of 11×

11 pixels, 11×

11 sub-pixels, 17×

17 pixels, 17×

17 sub-pixels, 23×

23 pixels, and 23×

23 sub-pixels.

34. The display apparatus of claim 1, wherein the sensor panel is constructed of a substrate material that does not transmit light.

35. A display apparatus, comprising:

a display panel;

a sensor panel including plural sensor units configured to sense incident light from the display panel and plural distinct portions configured to each forward light from the sensor panel towards the display panel; and

a controller to selectively control opacities of pixels and/or sub-pixels of the display panel to produce patterns on the display panel for displaying an image on the display panel, in a second mode, and to form at least one pattern on the display panel controlled to be transparent and an area of the display panel surrounding the at least one pattern controlled to be opaque, to capture an image by the at least one sensor unit, in a first mode,

wherein the plural distinct portions are respectively plural apertures or plural backlight units, with one or more of the plural apertures including a diffuser element, and with each of the plural backlight units respectively generating and directing light toward the display panel.

37. The method of claim 35, further comprising controlling, at a current time in the first mode, the display to have at least one pattern different than a pattern formed at a previous time when the current mode was the first mode, by changing at least one of a number of first pixels of the display that are selectively controlled to be opaque during the first mode, a number of second pixels of the display that are selectively controlled to be transparent during the first mode, positions of one or more pixels of the display that are controlled to be opaque during the first mode, and positions of one or more pixels of the display that are controlled to be transparent pixels during the first mode, than in the previous time.

38. The method of claim 35, further comprising controlling, at a current time in the first mode, the display to have at least one pattern different than a pattern formed at a previous time when the current mode was the first mode, by changing at least one of a number of first sub-pixels of the display that are selectively controlled to be opaque during the first mode, a number of second sub-pixels of the display that are selectively controlled to be transparent during the first mode, positions of one or more sub-pixels of the display that are controlled to be opaque during the first mode, and positions of one or more sub-pixels of the display that are controlled to be transparent pixels during the first mode, than in the previous time.

39. The method of claim 35, further comprising selectively controlling a combination of a capture rate, for capturing images from the sensor panel, and a display rate, for displaying plural images, to be greater than a maximum refresh rate of the display.

40. The method of claim 35, further comprising selectively controlling the display to only pass the first incident light toward the sensor panel in a first cycle, of plural fixed time length cycles, and controlling the display to only pass the light out of the display to display the image in a second cycle, of the plural fixed time length cycles,

wherein the fixed time is a time it takes for the display to perform one refresh.

41. The method of claim 35, further comprising selectively controlling opacities of pixels and/or sub-pixels of the display to produce patterns on the display for displaying the image on the display, in a second mode, and to form at least one pattern on the display controlled to be transparent and an area of the display surrounding the at least one pattern controlled to be opaque, to capture an image by at least one sensor unit of the sensor, in a first mode, wherein the display is controlled differently in the first mode and the second mode.

42. The method of claim 40, further comprising determining a depth of an object represented in the first incident light based upon at least one detected size of a blur of the at least one pattern detected by the sensor panel.

43. The method of claim 41, wherein the determining further comprises determining plural depth measurements of the object represented in plural first incident lights each sensed by the sensor panel based respectively upon at least one detected size of a blur pattern for each sensed incident light, and generating 3D information of the object.

44. The method of claim 42, further comprising monitoring a movement by the object over plural captured images and respectively generated plural 3D information and determining whether the monitored movement and generated plural 3D information indicates that a gesture, of plural defined gestures, is being made by the movement of the object, and controlling an operation of a device based on the determined gesture.

45. The method of claim 41, further comprising restoring the captured image by calculating a distance-based impulse response based on the determined depth, performing devolution of the captured image using the distance-based impulse response to compensate for refraction and/or geometric effects caused by the at least one pattern of the display, and encoding the restored captured image.

36. A display method, comprising:

selectively configuring a display to form a pattern with at least one transparent pixel or sub-pixel to pass a first incident light through the display and toward a sensor panel, behind the display, and to form an opaque area that at least surrounds the pattern, when a current mode is a first mode, and configuring the display to pass light from a backlight, behind the display, through the display, when the current mode is a second mode;

sensing the first incident light, having passed through the display, upon incidence to the sensor panel that at least partially blocks transmission of light, in a first mode;

displaying an image on the display by passing light in a direction from the sensor layer through the display; and

repeatedly controlling the current mode to only one of the first mode and the second mode, including at least a changing of the current mode plural times.

46. A display method, comprising:

selectively configuring a display to form a pattern with at least one transparent pixel or sub-pixel to pass a first incident light through the display and toward a sensor panel, behind the display, and to form an opaque area that at least surrounds the pattern, when a current mode is a first mode, and configuring the display to pass light from plural backlights of the sensor panel through the display, when the current mode is a second mode;

sensing the first incident light, having passed through the display, upon incidence to the sensor panel that blocks transmission of light, in a first mode;

displaying an image on the display by passing respectively generated light from each backlight from the sensor layer through the display; and

repeatedly controlling the current mode to be one of the first mode and the second mode, including at least a changing of the current mode plural times.