3D imaging device and 3D imaging method

• US 9,188,849 B2
• Filed: 03/03/2011
• Issued: 11/17/2015
• Est. Priority Date: 03/05/2010
• Status: Active Grant

• Current Assignee: Panasonic Intellectual Property Management Co., Ltd. (Panasonic Holdings Corporation)
• Original Assignee: Panasonic Intellectual Property Management Co., Ltd. (Panasonic Holdings Corporation)
• Inventors: Yamashita, Haruo, Ito, Takeshi, Ono, Hiromichi
• Primary Examiner(s): Repko, Jason M
• Assistant Examiner(s): Yang, Yi
• Application Number:
  US13/582,051
• Publication Number:
  US 20120327197A1
• Time in Patent Office: 1,720 Days
• Field of Search: 345/660, 345/653, 345/664, 345/679, 348/42, 382/106
• US Class Current: 1/1
• CPC Class Codes: G03B 35/08 by simultaneous recording
• CPC Class Codes: G03B 35/18 by simultaneous viewing
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/144 for flicker reduction
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof

Abstract

A 3D imaging device determines, during imaging, whether the captured images will be perceived three-dimensionally without causing fatigue while simulating actual human perception. In a 3D imaging device, a display information setting unit obtains a display parameter associated with an environment in which a 3D video is viewed, and a control unit determines during 3D imaging whether a scene to be imaged three-dimensionally will be perceived three-dimensionally based on the viewing environment.

Claims

1. A 3D imaging device for three-dimensionally imaging a subject, and capturing a 3D image formed by a first viewing point image and a second viewing point image, the 3D imaging device comprising:

an optical system configured to form an optical image of the subject;

an imaging unit configured to generate the 3D image based on the formed optical image;

an imaging parameter obtaining unit configured to obtain an imaging parameter associated with the optical system used when the optical image has been formed;

a display parameter obtaining unit configured to obtain a display parameter associated with an environment in which the 3D image is viewed;

a disparity detection unit configured to detect a disparity for the 3D image;

a recording image generation unit configured to generate a 3D image for recording based on the generated 3D image,

a 3D perception determination unit configured to determine, before the 3D image for recording is generated, viewability of three-dimensional viewing of the generated 3D image based on the imaging parameter, the display parameter, and the detected disparity using a predetermined determination condition,

the 3D perception determination unit determines whether a scene to be imaged three-dimensionally will be perceived three-dimensionally by determining whether a state of a predetermined subject included in the scene to be imaged satisfies the formula;

|z*(W2/W1)*(L1/L2)*(α

1−

β

1)−

Δ

α

|<

δ

,

where L1, W1, α

1, β

1, and z are values during 3D imaging, and L1 is a distance from a line segment SB1 connecting the first viewing point and the second viewing point to a virtual screen, W1 is a width of the virtual screen, α

1 is a disparity angle formed when a point of convergence is formed on a point of intersection between a normal to the virtual screen extending through a midpoint of the line segment SB1 and the virtual screen, β

1 is a disparity angle formed when the point of convergence is on the predetermined subject included in the scene to be imaged, and z is a zoom ratio,

where L2, W2, and Δ

α

are values during 3D displaying, and L2 is a distance from a line segment SB2 connecting the first viewing point and the second viewing point to a display screen, W2 is a width of the display screen, and Δ

α

is a disparity adjustment angle corresponding to an amount of disparity adjustment performed during 3D displaying when the disparity adjustment is enabled by image shifting of the first viewing point image and/or the second viewing point image, and

δ

is a relative disparity angle defining a 3D viewing enabling range.

2. The 3D imaging device according to claim 1, wherein

the display parameter obtaining unit obtains the display parameter including information indicating at least a viewing distance from a viewer to a screen on which the 3D image is displayed, and

the 3D perception determination unit changes the predetermined determination condition to set a determination condition with which the viewability of the three-dimensional viewing of the generated 3D image will be determined to be lower as the viewing distance is shorter, and determine the viewability of the three-dimensional viewing based on the set determination condition.

3. The 3D imaging device according to claim 1, wherein

the imaging parameter obtaining unit obtains then imaging parameter including information indicating at least a zoom ratio used in the optical system when the optical image has been formed, and

the 3D perception determination unit changes the predetermined determination condition to set a determination condition with which the viewability of the three-dimensional viewing of the generated 3D image will be determined to be lower as the zoom ratio is larger, and determines the viewability of the three-dimensional viewing based on the set determination condition.

4. The 3D imaging device according to claim 1, wherein

the 3D perception determination unit sets a condition for determining the 3D viewing enabling range in a manner that the condition is less severe as a subject at the farthest point and/or a subject at the nearest point included in the scene to be imaged three-dimensionally occupies a smaller area on the virtual screen.

5. The 3D imaging device according to claim 1, wherein

the 3D perception determination unit sets a condition for determining the 3D viewing enabling range in a manner that the condition is less severe as a subject at the farthest point and/or a subject at the nearest point included in the scene to be imaged three-dimensionally is at a position nearer an edge of the virtual screen.

6. The 3D imaging device according to claim 2, wherein

the 3D perception determination unit sets a condition for determining the 3D viewing enabling range in a manner that the condition is less severe as a subject at the farthest point and/or a subject at the nearest point is more out of focus.

7. The 3D imaging device according to claim 1, further comprising:

a maximum disparity detection unit configured to detect a maximum disparity max based on the disparity detected by the disparity detection unit;

a minimum disparity detection unit configured to detect a minimum disparity min based on the disparity detected by the disparity detection unit; and

an imaging parameter adjustment unit configured to adjust the imaging parameter used for 3D imaging,

wherein the 3D perception determination unit determines whether the farthest point included in a scene being imaged and the nearest point included in the scene being imaged are within a 3D viewing enabling range, the farthest point corresponding to the maximum disparity max, the nearest point corresponding to the minimum disparity min, and

(1) when the 3D perception determination unit determines that both the farthest point and the nearest point are out of a 3D viewing enabling range,

the imaging parameter adjustment unit adjusts the imaging parameter, and

after the imaging parameter is adjusted, the 3D perception determination unit determines whether a 3D image obtained when the scene is imaged three-dimensionally will be perceived three-dimensionally,

(2) when the 3D perception determination unit determines that the farthest point is out of the 3D viewing enabling range and the nearest point is within the 3D viewing enabling range,

the disparity adjustment unit performs disparity adjustment that causes the farthest point to fall within the 3D viewing enabling range, and

after the disparity adjustment is performed, the 3D perception determination unit determines whether a 3D image obtained when the scene is imaged three-dimensionally will be perceived three-dimensionally,

(3) when the 3D perception determination unit determines that the farthest point is within the 3D viewing enabling range and the nearest point is out of the 3D viewing enabling range,

the disparity adjustment unit performs disparity adjustment that causes the farthest point to be a boundary point of the 3D viewing enabling range, and

after the disparity adjustment is performed, the 3D perception determination unit determines whether a 3D image obtained when the scene is imaged three-dimensionally will be perceived three-dimensionally, and

(4) when the 3D perception determination unit determines that both the farthest point and the nearest point are within the 3D viewing enabling range,

the 3D perception determination unit determines that a 3D image obtained when the scene is imaged three-dimensionally will be perceived three-dimensionally.

8. A 3D imaging device for three-dimensionally imaging a subject and capturing a 3D image formed by a first viewing point image and a second viewing point image, the 3D imaging device comprising:

an imaging unit configured to image the subject viewed from a first viewing point as the first viewing point image and generate a first image signal forming the first viewing point image, and image the subject viewed from a second viewing point as the second viewing point image and generate a second image signal forming the second viewing point image;

a disparity detection unit configured to detect a disparity from the first viewing point image and the second viewing point image for each pixel block consisting of one or more pixels;

a display parameter obtaining unit configured to obtain a display parameter associated with an environment in which the 3D image is viewed;

a disparity map image generation unit configured to generate a two-dimensional disparity map image by mapping a disparity of the each pixel block detected by the disparity detection unit;

a disparity angle obtaining unit configured to use disparities for two blocks on the two-dimensional disparity map image to obtain a disparity angle α

1 and a disparity angle α

2 corresponding to the disparities for the two blocks and obtain a distance h between the two blocks on the two-dimensional disparity map image;

a correction disparity angle calculation unit configured to calculate a correction disparity angle f(α

1, α

2, h) based on the disparity angles α

1 and α

2 of the two blocks and the distance h between the two blocks on the two-dimensional disparity map image;

a correction disparity angle maximum value obtaining unit configured to obtain a maximum value fmax of the correction disparity angle f(α

1, α

2, h) calculated for the two blocks included in the two-dimensional disparity map image; and

a 3D perception determination unit configured to compare the maximum value fmax with a disparity angle δ

indicating a 3D viewing enabling range, and, during imaging, determine that the scene to be imaged will be perceived three-dimensionally when determining that the maximum value fmax is less than the disparity angle δ

indicating the 3D viewing enabling range.

9. The 3D imaging device according to claim 8, wherein

the disparity angle δ

indicating the 3D viewing enabling range is a predetermined angle equal to or less than 2 degrees.

10. The 3D imaging device according to claim 8, wherein

the correction disparity angle f(α

1, α

2, h) is written as

f(α

1, α

2, h) =g(h)*|α

1−

α

2|, where g(h) is a monotonously decreasing function that yields a larger value as an absolute value of h is smaller.

11. A 3D imaging device for three-dimensionally imaging a subject and capturing a 3D image formed by a first viewing point image and a second viewing point image, the 3D imaging device comprising:

an imaging unit configured to image the subject viewed from a first viewing point as the first viewing point image and generate a first image signal forming the first viewing point image, and image the subject viewed from a second viewing point different from the first viewing point as the second viewing point image and generate a second image signal forming the second viewing point image;

a disparity detection unit configured to detect a disparity from the first viewing point image and the second viewing point image for each pixel block consisting of one or more pixels;

a display parameter obtaining unit configured to obtain a display parameter associated with an environment in which a 3D image is viewed;

a disparity map image generation unit configured to generate a two-dimensional disparity map image by mapping a disparity of the each pixel block detected by the disparity detection unit;

a disparity histogram generation unit configured to generate a disparity histogram that shows a frequency distribution for the disparity of the each pixel block based on the two-dimensional disparity map image; and

a 3D perception determination unit configured to determine, during imaging, whether a scene to be imaged will be perceived three-dimensionally based on the disparity histogram,

the disparity histogram generation unit performs clustering of the two-dimensional disparity map image, and generates a weighted disparity histogram in which each cluster obtained through the clustering is weighted using the function Weight;

Weight(x, y, z)=Cent(x)*Size(y)*Blur(z),

where Cent(x) is a function that yields a larger value as a position of a cluster is nearer a central position of the two-dimensional disparity map image, and x is a two-dimensional vector indicating a position of the cluster on the two-dimensional disparity map image,

Size(y) is a function that yields a larger value as an area occupied by the cluster formed by blocks of the two-dimensional disparity map image is greater, and y indicates an area occupied by the cluster formed by the blocks of the two-dimensional disparity map image,

Blur(z) is a function that yields a smaller value as a degree of blurring of the cluster is greater, and z indicates the degree of burring of the cluster, and

the 3D perception determination unit determines, during imaging, whether the scene to be imaged will be perceived three-dimensionally by comparing a target area AR1 that is an area having a disparity range of B4 to C4 to be subjected to the 3D perception determination process with a 3D viewing enabling area AR0, where C4 is a disparity that initially exceeds a long-range view threshold Th3 when the weighted disparity histogram is traced from a long-range view end toward a short-range view end, and B4 is a disparity that initially exceeds a short-range view threshold Th4 when the weighted disparity histogram is traced from a short-range view end toward a long-range view end.

12. The 3D imaging device according to claim 11, wherein

the 3D perception determination unit determines, during imaging, whether the scene to be imaged will be perceived three-dimensionally by comparing a target area AR1 that is an area having a disparity range of B2 to C2 to be subjected to a 3D perception determination process with a 3D viewing enabling area AR0, where B2 is a disparity that initially exceeds a short-range view threshold Th2 when the disparity histogram is traced from a short-range view end toward a long-range view end, and C2 is a long-range view limited point of the target area AR1.

13. The 3D imaging device according to claim 12, wherein

the long-range view threshold Th1 is greater than the short-range view threshold Th2.

14. The 3D imaging device according to claim 12, wherein

the 3D perception determination unit sets, using the disparity histogram, the 3D viewing enabling range AR0 based on a disparity C1 that initially exceeds a long-range view threshold Th1 when the disparity histogram is traced from a long-range view end toward a short-range view end.

15. The 3D imaging device according to claim 11, wherein

the long-range view threshold Th3 is greater than the short-range view threshold Th4.

16. The 3D imaging device according to claim 11, wherein

the 3D perception determination unit sets the 3D viewing enabling range AR0 based on the disparity C4 that initially exceeds the long-range view threshold Th3 when the weighted disparity histogram is traced from the long-range view end toward the short-range view end.

17. A 3D imaging method used in a 3D imaging device for three-dimensionally imaging a subject and capturing a 3D image formed by a first viewing point image and a second viewing point image, the 3D imaging device including an optical system configured to form an optical image of the subject, the method comprising:

generating the 3D image based on the formed optical image;

obtaining an imaging parameter associated with the optical system used when the optical image has been formed;

obtaining a display parameter associated with an environment in which the 3D image is viewed;

detecting a disparity for the 3D image;

generating a 3D image for recording based on the generated 3D image;

determining, before the 3D image for recording is generated, viewability of three-dimensional viewing of the generated 3D image based on the imaging parameter, the display parameter, and the detected disparity using a predetermined determination condition;

determining whether a scene to be imaged three-dimensionally will be perceived three-dimensionally by determining whether a state of a predetermined subject included in the scene to be imaged satisfies the formula;

|z*(W2/W1)*(L1/L2)*(α

1−

β

1)−

Δ

α

|<

δ

,

where L1, W1, α

1, β

1, and z are values during 3D imaging, and L1 is a distance from a line segment SB1 connecting the first viewing point and the second viewing point to a virtual screen, W1 is a width of the virtual screen, α

1 is a disparity angle formed when a point of convergence is formed on a point of intersection between a normal to the virtual screen extending through a midpoint of the line segment SB1 and the virtual screen, β

1 is a disparity angle formed when the point of convergence is on the predetermined subject included in the scene to be imaged, and z is a zoom ratio,

where L2, W2, and Δ

α

are values during 3D displaying, and L2 is a distance from a line segment SB2 connecting the first viewing point and the second viewing point to a display screen, W2 is a width of the display screen, and Δ

α

is a disparity adjustment angle corresponding to an amount of disparity adjustment performed during 3D displaying when the disparity adjustment is enabled by image shifting of the first viewing point image and/or the second viewing point image, and

δ

is a relative disparity angle defining a 3D viewing enabling range.

18. A 3D imaging method for three-dimensionally imaging a subject and capturing a 3D image formed by a first viewing point image and a second viewing point image, the method comprising:

imaging the subject viewed from a first viewing point as the first viewing point image and generating a first image signal forming the first viewing point image, and imaging the subject viewed from a second viewing point as the second viewing point image and generating a second image signal forming the second viewing point image;

detecting a disparity from the first viewing point image and the second viewing point image for each pixel block consisting of one or more pixels;

obtaining a display parameter associated with an environment in which the 3D image is viewed;

generating a two-dimensional disparity map image by mapping a disparity of the each pixel block detected in the detecting of the disparity;

using disparities for two blocks on the two-dimensional disparity map image to obtain a disparity angle α

1 and a disparity angle α

2 corresponding to the disparities for the two blocks and obtaining a distance h between the two blocks on the two-dimensional disparity map image;

calculating a correction disparity angle f(α

1, α

2, h) based on the disparity angles α

1 and α

2 of the two blocks and the distance h between the two blocks on the two-dimensional disparity map image;

obtaining a maximum value fmax of the correction disparity angle f(α

1, α

2, h) calculated for the two blocks included in the two-dimensional disparity map image; and

comparing the maximum value fmax with a disparity angle δ

indicating a 3D viewing enabling range, and, during imaging, determining that the scene to be imaged will be perceived three-dimensionally when determining that the maximum value fmax is less than the disparity angle δ

indicating the 3D viewing enabling range.

19. A 3D imaging method for three-dimensionally imaging a subject and capturing a 3D image formed by a first viewing point image and a second viewing point image, the method comprising:

imaging a subject viewed from a first viewing point as the first viewing point image and generating a first image signal forming the first viewing point image, and imaging a subject viewed from a second viewing point different from the first viewing point as the second viewing point image and generating a second image signal forming the second viewing point image;

detecting a disparity from the first viewing point image and the second viewing point image for each pixel block consisting of one or more pixels;

obtaining a display parameter associated with an environment in which a 3D image is viewed;

generating a two-dimensional disparity map image by mapping a disparity of the each pixel block detected in the detecting of the disparity;

generating a disparity histogram that shows a frequency distribution for the disparity of the each pixel block based on the two-dimensional disparity map image; and

determining, during imaging, whether a scene to be imaged will be perceived three-dimensionally based on the disparity histogram,

clustering of the two-dimensional disparity map image and generating a weighted disparity histogram in which each cluster obtained through the clustering is weighted using the function Weight;

Weight(x, y, z)=Cent(x)*Size(y)*Blur(z),

where Cent(x) is a function that yields a larger value as a position of a cluster is nearer a central position of the two-dimensional disparity map image, and x is a two-dimensional vector indicating a position of the cluster on the two-dimensional disparity map image,

Size(y) is a function that yields a larger value as an area occupied by the cluster formed by blocks of the two-dimensional disparity map image is greater, and y indicates an area occupied by the cluster formed by the blocks of the two-dimensional disparity map image,

Blur(z) is a function that yields a smaller value as a degree of blurring of the cluster is greater, and z indicates the degree of burring of the cluster, and

determining, during imaging, whether the scene to be imaged will be perceived three-dimensionally by comparing a target area AR1 that is an area having a disparity range of B4 to C4 to be subjected to the 3D perception determination process with a 3D viewing enabling area AR0, where C4 is a disparity that initially exceeds a long-range view threshold Th3 when the weighted disparity histogram is traced from a long-range view end toward a short-range view end, and B4 is a disparity that initially exceeds a short-range view threshold Th4 when the weighted disparity histogram is traced from a short-range view end toward a long-range view end.