Image display device, image display method, and program

• US 9,307,220 B2
• Filed: 01/26/2012
• Issued: 04/05/2016
• Est. Priority Date: 01/26/2011
• Status: Active Grant

• Current Assignee: NLT Technologies, Ltd.
• Original Assignee: NLT Technologies, Ltd.
• Inventors: Wu, Di, Sakamoto, Michiaki
• Primary Examiner(s): Tryder, Gregory J
• Assistant Examiner(s): LEE, KWANG B
• Application Number:
  US13/358,997
• Publication Number:
  US 20120188235A1
• Time in Patent Office: 1,531 Days
• Field of Search: 345/422
• US Class Current: 1/1
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/349 Multi-view displays for displaying three or more geometrical viewpoints without viewer tracking for viewing without the aid of special glasses using fly-eye lenses H04N13/307
• CPC Class Codes: H04N 13/361 Reproducing mixed stereoscopic images; Reproducing mixed monoscopic and stereoscopic images, e.g. a stereoscopic image overlay window on a monoscopic image background

Abstract

An image display device includes region of interest extraction unit, parallax image generation unit and 3D image display unit. Region of interest extraction unit generates depth image signal by depth image conversion employing depth threshold, depth image signal including information on distance in three-dimensional space between viewpoint and each pixel of two-dimensional image including region of interest desired to be noted by observer, depth image conversion being such that depth value indicating distance between viewpoint and each pixel of two-dimensional image is converted to depth value for 2D display when depth value is equal to or larger than depth threshold. Parallax image generation unit generates both-eye parallax image having parallax in region of interest alone, from two-dimensional image and image obtained by conversion of region of interest image representing region of interest at each of both-eye viewpoints, based on two-dimensional image and depth image signal. 3D image display unit displays both eye parallax image.

Claims

1. An image display device, comprising:

a region of interest extraction unit that i) generates a depth image signal by depth image conversion employing a depth threshold generated by an amount of movement of images in a 2D/3D mixed display, the depth image signal including information on a distance in a three-dimensional space between a viewpoint and each pixel of a two-dimensional image including a region of interest desired to be noted by an observer, the depth image conversion being such that a depth value for each pixel, the depth value indicating the distance between the viewpoint and each pixel of the two-dimensional image is converted to a new depth value for 2D display when the depth value is equal to or larger than the depth threshold, and ii) extracts the region of interest based on the generated depth image signal;

a parallax image generation unit that generates a left- and right-eye parallax images each having a parallax in a new region of interest resulting from the new depth value and having no parallax in a region other than the new region of interest, from said two-dimensional image and an image obtained by conversion of a region of interest image to be displayed in the region of interest at each of right and left viewpoints, based on said two-dimensional image and said depth image signal; and

a 3D image display unit that displays the left-eye parallax image and the right-eye parallax image,

wherein the region of interest extraction unit comprises

an image segmentation unit that receives a first two-dimensional image of a most recent frame and a second two-dimensional image of a frame immediately preceding the most recent frame of the first two-dimensional image, divides each of the first and second two-dimensional images into divided regions each having common characteristic(s) to the first and second two-dimensional images, and generates a divided first two-dimensional image and a divided second two-dimensional image,

an optical flow computing unit that computes a difference value between center positions of gravity of a divided region of the divided first two-dimensional image and a divided region of the divided second two-dimensional image, as an optical flow of the divided region of the divided first two-dimensional image,

a depth image computing unit that generates a depth estimation value for each pixel of the divided region of the divided first two-dimensional image based on luminance information on the divided region of the divided first two-dimensional image, and

a depth image conversion unit that computes the depth threshold from the optical flow, thereby performing the depth image conversion on the depth estimation value,

and

wherein the depth threshold is computed by multiplying the optical flow by a proportionality constant and then adding to the multiplied optical flow a predetermined constant indicating a depth value when the optical flow is zero.

3. The image display device according to claim 1, wherein when the optical flow exceeds a predetermined value, the proportionality constant is reduced.

4. The image display device according to claim 1, wherein the depth image conversion unit does not perform the depth image conversion when the depth value is less than the depth threshold.

5. The image display device according to claim 1, wherein the 3D image display unit comprises:

a LED backlight that adjusts luminance of a display screen; and

a backlight control unit that can partially change luminance of the LED backlight and reduce the LED luminance for a region other than the region of interest; and

the parallax image generation unit increases grayscale of the region other than the region of interest when said LED luminance is reduced.

6. The image display device according to claim 1, wherein the region of interest extraction unit comprises:

an image segmentation unit that receives a first two-dimensional image and a look-up table defining a relationship among shape information, area information and a depth value of each region of the image, divides the first two-dimensional image into regions of the first two-dimensional image each having common characteristic(s), and generates a divided first two-dimensional image; and

a depth image generation unit that generates a depth estimation value based on the look-up table defining the relationship among the shape information, the area information, and the depth value of each of the divided regions; and

a depth image conversion unit performs the depth image conversion using the depth estimation value and the depth threshold set in advance.

2. An image display device, comprising:

a region of interest extraction unit that i) generates a depth image signal by depth image conversion employing a depth threshold generated by an amount of movement of images in a 2D/3D mixed display, the depth image signal including information on a distance in a three-dimensional space between a viewpoint and each pixel of a two-dimensional image including a region of interest desired to be noted by an observer, the depth image conversion being such that a depth value for each pixel, the depth value indicating the distance between the viewpoint and each pixel of the two-dimensional image is converted to a new depth value for 2D display when the depth value is equal to or larger than the depth threshold, and ii) extracts the region of interest based on the generated depth image signal;

a parallax image generation unit that generates a left- and right-eye parallax images each having a parallax in a new region of interest resulting from the new depth value and having no parallax in a region other than the new region of interest, from said two-dimensional image and an image obtained by conversion of a region of interest image to be displayed in the region of interest at each of right and left viewpoints, based on said two-dimensional image and said depth image signal; and

a 3D image display unit that displays the left-eye parallax image and the right-eye parallax image,

wherein the region of interest extraction unit comprises

an image segmentation unit that receives first left- and right-eye two-dimensional images of a most recent frame and second left- and right-eye two-dimensional images of a frame immediately preceding the most recent frame of the first left- and right-eye two-dimensional images, divides each of the first and second left-eye two-dimensional images, and, the first and second right-eye two-dimensional images into divided regions each having common characteristic(s), thereby generating divided first left- and right-eye two-dimensional images, and divided second left- and right-eye two-dimensional images, respectively,

an optical flow computing unit that computes a difference value between center positions of gravity of the divided regions of the divided first and second left-eye two-dimensional images or a difference value between center positions of gravity of the divided regions of the divided first and second right-eye two-dimensional images, as an optical flow for each of the divided regions of the divided first left- and right-eye two-dimensional images, respectively,

a depth image computing unit that generates a depth estimation value for each pixel of the divided region of the divided first left- or right-eye two-dimensional image, based on a parallax amount between the divided regions of the divided first left- and right-eye two-dimensional images, and

a depth image conversion unit that computes the depth threshold from the optical flow, thereby performing the depth image conversion on the depth estimation value,

and

wherein the depth threshold is computed by multiplying the optical flow by a proportionality constant and then adding to the multiplied optical flow a predetermined constant indicating a depth value when the optical flow is zero.

7. An image display method, comprising:

extracting a region of interest from a two-dimensional image a region of interest desired to be noted by an observer, by i) generating a depth image signal through depth image conversion employing a depth threshold generated by an amount of movement of images in a 2D/3D mixed display, the depth image signal including information on a distance in a three-dimensional space between a viewpoint and each pixel of the two-dimensional image including the region of interest, the depth image conversion being such that a depth value for each pixel, the depth value indicating the distance between the viewpoint and each pixel of the two-dimensional image is converted to a new depth value for 2D display when the depth value is equal to or larger than the depth threshold, and ii) extracting the region of interest based on the generated depth image signal;

generating a parallax image by generating left- and right-eye parallax images each having a parallax in a new region of interest resulting from the new depth value and having no parallax in a region other than the new region of interest, from said two-dimensional image and an image obtained by conversion of a region of interest image to be displayed in the region of interest at each of right and left viewpoints, based on said two-dimensional image and said depth image signal; and

displaying a 3D image by displaying the left-eye parallax image and the right-eye parallax image,

wherein said extracting the region of interest comprises;

receiving a first two-dimensional image of a most recent frame and a second two-dimensional image of a frame immediately preceding the most recent frame of the first two-dimensional image, dividing each of the first and second two-dimensional images into divided regions each having common characteristic(s) to the first and second two-dimensional images, and generating divided first and second two-dimensional images;

computing a difference value between center positions of gravity of a divided region of the divided first two-dimensional image and a divided region of the divided second two-dimensional image, as an optical flow of the divided region of the divided first two-dimensional image;

generating a depth estimation value for each pixel of the divided region of the divided first two-dimensional image based on luminance information on the divided region of the divided first two-dimensional image; and

computing the depth threshold from the optical flow, thereby performing the depth image conversion on the depth estimation value,

and

wherein when performing the depth image conversion, the depth threshold is computed by multiplying the optical flow by a proportionality constant and then adding to the multiplied optical flow a predetermined constant indicating a depth value when the optical flow is zero.

9. The image display method according to claim 7, wherein said performing the depth image conversion comprises:

reducing the proportionality constant when the optical flow exceeds a predetermined value.

10. The image display method according to claim 7, wherein the depth image conversion is not performed when the depth value is less than the depth threshold.

11. The image display method according to claim 7, wherein said displaying the 3D image comprises:

reducing LED luminance of a LED backlight for a region other than the region of interest, the LED backlight adjusting luminance of a display screen; and

said generating the parallax images comprises;

increasing grayscale of the region other than the region of interest when the LED luminance is reduced.

8. An image display method, comprising:

extracting a region of interest from a two-dimensional image a region of interest desired to be noted by an observer, by i) generating a depth image signal through depth image conversion employing a depth threshold generated by an amount of movement of images in a 2D/3D mixed display, the depth image signal including information on a distance in a three-dimensional space between a viewpoint and each pixel of the two-dimensional image including the region of interest, the depth image conversion being such that a depth value for each pixel, the depth value indicating the distance between the viewpoint and each pixel of the two-dimensional image is converted to a new depth value for 2D display when the depth value is equal to or larger than the depth threshold, and ii) extracting the region of interest based on the generated depth image signal;

generating a parallax image by generating left- and right-eye parallax images each having a parallax in a new region of interest resulting from the new depth value and having no parallax in a region other than the new region of interest, from said two-dimensional image and an image obtained by conversion of a region of interest image to be displayed in the region of interest at each of right and left viewpoints, based on said two-dimensional image and said depth image signal; and

displaying a 3D image by displaying the left-eye parallax image and the right-eye parallax image,

wherein said extracting the region of interest comprises;

receiving first left- and right-eye two-dimensional images of a most recent frame and second left- and right-eye two-dimensional images of a frame immediately preceding the most recent frame of the first left- and right-eye two-dimensional images, dividing each of the first and second left-eye two-dimensional images, and, the first and second right-eye two-dimensional images into divided regions each having common characteristic(s), thereby generating divided first left- and right-eye two-dimensional images, and, divided second left- and right-eye two-dimensional images, respectively;

computing a difference value between center positions of gravity of the divided regions of the divided first and second left-eye two-dimensional images, or a difference value between center positions of gravity of the divided regions of the divided first and second right-eye two-dimensional images, as an optical flow for each of the divided regions of the divided first left- and right-eye two-dimensional images, respectively;

generating a depth estimation value for each pixel of the divided region of the divided first left- or right-eye two-dimensional image, based on a parallax amount between the divided regions of the divided first left- and right-eye two-dimensional images; and

computing the depth threshold from the optical flow, thereby performing the depth image conversion on the depth estimation value,

and

wherein, when performing the depth image conversion, the depth threshold is computed by multiplying the optical flow by a proportionality constant and then adding to the multiplied optical flow a predetermined constant indicating a depth value when the optical flow is zero.