Compound-eye imaging device, and parallax adjusting method and program thereof

• US 9,071,759 B2
• Filed: 09/13/2010
• Issued: 06/30/2015
• Est. Priority Date: 03/30/2010
• Status: Active Grant

• Current Assignee: Fujifilm Corporation (Fujifilm Holdings Corporation)
• Original Assignee: Fujifilm Corporation (Fujifilm Holdings Corporation)
• Inventors: Mori, Koji
• Primary Examiner(s): Kelley, Christopher S
• Assistant Examiner(s): KWAN, MATTHEW K
• Application Number:
  US13/260,143
• Publication Number:
  US 20120140043A1
• Time in Patent Office: 1,751 Days
• Field of Search: 348/47
• US Class Current: 1/1
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/305 using lenticular lenses, e.g. arrangements of cylindrical lenses
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 23/61 based on recognised objects
• CPC Class Codes: H04N 23/62 Control of parameters via user interfaces
• CPC Class Codes: H04N 23/635 Region indicators; Field of view indicators

Abstract

A composite eye camera has: imaging sections (21A, 21B) that generate plural viewpoint images per frame by imaging a same subject from plural viewpoints; a three-dimensional processing section (30) that acquires a parallax amount on the basis of the generated plural viewpoint images; and a CPU (35) that judges that there is an abnormality in the parallax amount in at least one case among a case in which fluctuation of a fixed interval in the acquired parallax amount is greater than a predetermined value, and a case in which the parallax amount reaches a predetermined allowed limit value, and a case in which an object of acquisition of the parallax amount can no longer be detected, and judges that there is no abnormality in the parallax amount in a case that is none of these, and, when it is judged that there is no abnormality in the parallax amount, carries out first parallax adjustment, and, when it is judged that there is an abnormality in the parallax amount, switches to control of a second parallax adjustment, that is different than control of the first parallax adjustment, and carries out parallax adjustment. Accordingly, parallax adjustment is carried out appropriately even when there are problems in the parallax amount itself that is determined from the plural viewpoint images, or when an object of parallax adjustment can no longer be detected.

Claims

1. A composite eye imaging device comprising:

imaging sections that generate a plurality of viewpoint images per frame by imaging a same subject from a plurality of viewpoints, wherein the imaging sections generate a right eye image and a left eye image for each frame of consecutive frames;

a parallax amount acquiring section that acquires a parallax amount on the basis of the plurality of viewpoint images generated by the imaging sections, wherein the parallax amount acquiring section detects a face from each of the right eye image and the left eye image and computes a parallax amount using a characteristic point of the detected face from the right eye image and a characteristic point of the detected face from the left eye image, and wherein the parallax amount acquiring section computes said parallax amount for each of the consecutive frames;

an abnormality judging section that judges that there is an abnormality in the parallax amount in at least one case of a case in which fluctuation of a fixed interval in the parallax amount acquired by the parallax amount acquiring section is greater than a predetermined value, a case in which the parallax amount reaches an allowed limit value that is determined in advance, or a case in which an object of acquisition of the parallax amount can no longer be detected, and judges that there is no abnormality in the parallax amount in a case that is none of a case in which fluctuation of a fixed interval in the parallax amount is greater than a predetermined value, a case in which the parallax amount reaches an allowed limit value that is determined in advance, or a case in which an object of acquisition of the parallax amount can no longer be detected, wherein the abnormality judging section judges whether there is an abnormality in the parallax amount for each of the consecutive frames; and

a parallax adjusting section that judges whether there is an abnormality in the parallax amount, such that;

when it is judged by the abnormality judging section that there is no abnormality in the parallax amount in one or more consecutive frames, the parallax adjusting section carries out a parallax adjustment for each of the consecutive frames for which there is no abnormality in the parallax amount, wherein said parallax adjustment is a first parallax adjustment, and the parallax adjusting section also increases an index number for each of the consecutive frames for which there is no abnormality in the parallax amount, and

when it is judged by the abnormality judging section that there is an abnormality in the parallax amount, the parallax adjusting section switches to control of a second parallax adjustment that is different from the control of the first parallax adjustment, whereby in the control of the second parallax adjustment the parallax adjusting section lowers a parallax adjustment frequency such that the parallax adjusting section carries out said parallax adjustment for each N frame of the consecutive frames, wherein N is a predetermined adjustment parameter higher than 1, such that;

when it is judged by the abnormality judging section that there is an abnormality in the parallax amount in one consecutive frame, the parallax adjusting section increases said index number for said one consecutive frame and checks if said index number corresponding to said one consecutive frame is less than N, and

if said index number is less than N, then the parallax adjusting section does not carry out said parallax adjustment for said one consecutive frame,

but if said index number is at least N, the parallax adjusting section carries out said parallax adjustment for said one consecutive frame.

2. The composite eye imaging device of claim 1, wherein, when it is judged by the abnormality judging section that there is an abnormality in the parallax amount, the parallax adjusting section carries out parallax adjustment within a range of a parallax amount maximum change amount that is determined in advance.

3. The composite eye imaging device of claim 1 or claim 2, wherein, when it is judged by the abnormality judging section that there is an abnormality in the parallax amount, the parallax adjusting section carries out parallax adjustment by using a parallax amount of a previous frame.

4. The composite imaging device of claim 2, further comprising an associating section that associates at least one information of information that is obtained from the plurality of viewpoint images generated by the imaging sections and that is used for acquiring the parallax amount at the parallax amount acquiring section, information that expresses the parallax amount acquired by the parallax amount acquiring section by using the information obtained from the plurality of viewpoint images, or information expressing whether or not a predetermined processing has been carried out by using the parallax amount, with the plurality of viewpoint images that correspond to the at least one information and on which parallax adjustment has been carried out.

5. The composite imaging device of claim 1, further comprising an associating section that associates at least one information of information that is obtained from the plurality of viewpoint images generated by the imaging sections and that is used for acquiring the parallax amount at the parallax amount acquiring section, information that expresses the parallax amount acquired by the parallax amount acquiring section by using the information obtained from the plurality of viewpoint images, or information expressing whether or not a predetermined processing has been carried out by using the parallax amount, with the plurality of viewpoint images that correspond to the at least one information and on which parallax adjustment has been carried out.

6. The composite eye imaging device of claim 5, wherein the associating section associates the at least one information with the plurality of viewpoint images that correspond to the at least one information and on which parallax adjustment has been carried out, and collects the at least one information in one electronic file.

7. The composite eye imaging device of claim 1, wherein the parallax amount acquiring section computes the parallax amount using a plurality of characteristic points of the detected face from the right eye image, and a plurality of characteristic points of the detected face from the left eye image.

8. The composite eye imaging device of claim 1, wherein, after said index number becomes at least N and the parallax adjusting section carries out said parallax adjustment for said one consecutive frame, said index number is reset.

9. A composite eye imaging device comprising:

imaging sections that generate a plurality of viewpoint images per frame by imaging a same subject from a plurality of viewpoints, wherein the imaging sections generate a right eye image and a left eye image for each frame of consecutive frames;

a parallax amount acquiring section that acquires a parallax amount of a predetermined object on the basis of the plurality of viewpoint images generated by the imaging sections, wherein the parallax amount acquiring section detects a face from each of the right eye image and the left eye image and computes a parallax amount using a characteristic point of the detected face from the right eye image and a characteristic point of the detected face from the left eye image, and wherein the parallax amount acquiring section computes said parallax amount for each of the consecutive frames;

an abnormality judging section that judges that there is an abnormality in the parallax amount in at least one case of a case in which fluctuation of a fixed interval in the parallax amount acquired by the parallax amount acquiring section is greater than a predetermined value, a case in which the parallax amount reaches an allowed limit value that is determined in advance, or a case in which an object of acquisition of the parallax amount can no longer be detected, and judges that there is no abnormality in the parallax amount in a case that is none of a case in which fluctuation of a fixed interval in the parallax amount is greater than a predetermined value, a case in which the parallax amount reaches an allowed limit value that is determined in advance, or a case in which an object of acquisition of the parallax amount can no longer be detected, wherein the abnormality judging section judges whether there is an abnormality in the parallax amount for each of the consecutive frames; and

a parallax adjusting section that judges whether there is an abnormality in the parallax amount, such that;

when it is judged by the abnormality judging section that there is no abnormality in the parallax amount in one or more consecutive frames, the parallax adjusting section carries out a parallax adjustment of the predetermined object for each of the consecutive frames for which there is no abnormality in the parallax amount, wherein said parallax adjustment is a first parallax adjustment, and the parallax adjusting section also increases an index number for each of the consecutive frames for which there is no abnormality in the parallax amount, and

when it is judged by the abnormality judging section that there is an abnormality in the parallax amount, the parallax adjusting section switches to control of a second parallax adjustment that is different from the control of the first parallax adjustment, whereby in the control of the second parallax adjustment the parallax adjusting section lowers a parallax adjustment frequency and carries out parallax adjustment on another object that is different than the predetermined object, such that the parallax adjusting section carries out said parallax adjustment for each N frame of the consecutive frames, wherein N is a predetermined adjustment parameter higher than 1, such that;

when it is judged by the abnormality judging section that there is an abnormality in the parallax amount in one consecutive frame, the parallax adjusting section increases said index number for said one consecutive frame and checks if said index number corresponding to said one consecutive frame is less than N, and

if said index number is less than N, then the parallax adjusting section does not carry out said parallax adjustment for said one consecutive frame,

but if said index number is at least N, the parallax adjusting section carries out said parallax adjustment for said one consecutive frame.

10. The composite eye imaging device of claim 9, wherein the parallax adjusting section makes an object, which has a closest distance to the predetermined object in a direction orthogonal to an image plane of the viewpoint images, be the other object.

11. The composite eye imaging device of claim 9, wherein the parallax adjusting section makes an object, which has a closest distance to the predetermined object in an image plane of the viewpoint images, be the other object.

12. The composite imaging device of claim 9, further comprising an associating section that associates at least one information of information that is obtained from the plurality of viewpoint images generated by the imaging sections and that is used for acquiring the parallax amount at the parallax amount acquiring section, information that expresses the parallax amount acquired by the parallax amount acquiring section by using the information obtained from the plurality of viewpoint images, or information expressing whether or not a predetermined processing has been carried out by using the parallax amount, with the predetermined object or the other object that corresponds to the at least one information and on which parallax adjustment has been carried out.

13. The composite eye imaging device of claim 12, wherein the associating section associates the at least one information with the predetermined object or the other object that corresponds to the at least one information and on which parallax adjustment has been carried out, and collects the at least one information in one electronic file.

14. The composite eye imaging device of claim 9, wherein the parallax amount acquiring section computes the parallax amount using a plurality of characteristic points of the detected face from the right eye image, and a plurality of characteristic points of the detected face from the left eye image.

15. The composite eye imaging device of claim 9, wherein, after said index number becomes at least N and the parallax adjusting section carries out said parallax adjustment for said one consecutive frame, said index number is reset.

16. A parallax adjusting method of a composite eye imaging device, the method comprising:

acquiring a parallax amount on the basis of a plurality of viewpoint images generated by imaging sections that generate the plurality of viewpoint images per frame by imaging a same subject from a plurality of viewpoints, wherein the plurality of viewpoint images are a right eye image and a left eye image for each frame of consecutive frames,

wherein said acquiring step detects a face from each of the right eye image and the left eye image and computes a parallax amount using a characteristic point of the detected face from the right eye image and a characteristic point of the detected face from the left eye image, and wherein the acquiring step computes said parallax amount for each of the consecutive frames;

judging that there is an abnormality in the parallax amount in at least one case of a case in which fluctuation of a fixed interval in the acquired parallax amount is greater than a predetermined value, a case in which the parallax amount reaches an allowed limit value that is determined in advance, or a case in which an object of acquisition of the parallax amount can no longer be detected, and judging that there is no abnormality in the parallax amount in a case that is none of a case in which fluctuation of a fixed interval in the parallax amount is greater than a predetermined value, a case in which the parallax amount reaches an allowed limit value that is determined in advance, or a case in which an object of acquisition of the parallax amount can no longer be detected, wherein the judging step judges whether there is an abnormality in the parallax amount for each of the consecutive frames; and

when it is judged that there is no abnormality in the parallax amount in one or more consecutive frames, carrying out a parallax adjustment for each of the consecutive frames for which there is no abnormality in the parallax amount, wherein said parallax adjustment is a first parallax adjustment, and also increasing an index number for each of the consecutive frames for which there is no abnormality in the parallax amount, and,

when it is judged that there is an abnormality in the parallax amount, switching to control of a second parallax adjustment that is different from the control of the first parallax adjustment, whereby the control of the second parallax adjustment lowers a parallax adjustment frequency such that said parallax adjustment for each N frame of the consecutive frames is carried out, wherein N is a predetermined adjustment parameter higher than 1, such that;

when it is judged that there is an abnormality in the parallax amount in one consecutive frame, said index number for said one consecutive frame is increased and a check is performed to determine if said index number corresponding to said one consecutive frame is less than N, and

if said index number is less than N, then said parallax adjustment for said one consecutive frame is not carried out,

but if said index number is at least N, said parallax adjustment for said one consecutive frame is carried out.

17. The parallax adjusting method of claim 16, wherein the acquiring step computes the parallax amount using a plurality of characteristic points of the detected face from the right eye image, and a plurality of characteristic points of the detected face from the left eye image.

18. The parallax adjusting method of claim 16, wherein, after said index number becomes at least N and said parallax adjustment for said one consecutive frame is carried out, said index number is reset.

19. A parallax adjusting method of a composite eye imaging device, the method comprising:

acquiring a parallax amount of a predetermined object on the basis of a plurality of viewpoint images generated by imaging sections that generate the plurality of viewpoint images per frame by imaging a same subject from a plurality of viewpoints, wherein the plurality of viewpoint images are a right eye image and a left eye image for each frame of consecutive frames,

wherein said acquiring step detects a face from each of the right eye image and the left eye image and computes a parallax amount using a characteristic point of the detected face from the right eye image and a characteristic point of the detected face from the left eye image, and wherein the acquiring step computes said parallax amount for each of the consecutive frames;

judging that there is an abnormality in the parallax amount in at least one case of a case in which fluctuation of a fixed interval in the acquired parallax amount is greater than a predetermined value, a case in which the parallax amount reaches an allowed limit value that is determined in advance, of a case in which an object of acquisition of the parallax amount can no longer be detected, and judging that there is no abnormality in the parallax amount in a case that is none of a case in which fluctuation of a fixed interval in the parallax amount is greater than a predetermined value, a case in which the parallax amount reaches an allowed limit value that is determined in advance, or a case in which an object of acquisition of the parallax amount can no longer be detected, wherein the judging step judges whether there is an abnormality in the parallax amount for each of the consecutive frames; and

when it is judged that there is no abnormality in the parallax amount in one or more consecutive frames, carrying out a parallax adjustment of the predetermined object for each of the consecutive frames for which there is no abnormality in the parallax amount, wherein said parallax adjustment is a first parallax adjustment, and also increasing an index number for each of the consecutive frames for which there is no abnormality in the parallax amount, and,

when it is judged that there is an abnormality in the parallax amount, switching to control of a second parallax adjustment that is different from the control of the first parallax adjustment, whereby the control of the second parallax adjustment lowers a parallax adjustment frequency and carries out parallax adjustment on another object that is different than the predetermined object, said parallax adjustment for each N frame of the consecutive frames being carried out, wherein N is a predetermined adjustment parameter higher than 1, such that;

when it is judged that there is an abnormality in the parallax amount in one consecutive frame, said index number for said one consecutive frame is increased and a check is performed to determine if said index number corresponding to said one consecutive frame is less than N, and

if said index number is less than N, then said parallax adjustment for said one consecutive frame is not carried out,

but if said index number is at least N, said parallax adjustment for said one consecutive frame is carried out.

20. The parallax adjusting method of claim 19, wherein the acquiring step computes the parallax amount using a plurality of characteristic points of the detected face from the right eye image, and a plurality of characteristic points of the detected face from the left eye image.

21. The parallax adjusting method of claim 19, wherein, after said index number becomes at least N and said parallax adjustment for said one consecutive frame is carried out, said index number is reset.

22. A non-transitory computer-readable medium storing a parallax adjusting program of a composite eye imaging device including computer-readable instructions, the parallax adjusting program when executed causing a computer to function as:

a parallax amount acquiring section that acquires a parallax amount on the basis of a plurality of viewpoint images generated by imaging sections that generate the plurality of viewpoint images per frame by imaging a same subject from a plurality of viewpoints, wherein the plurality of viewpoint images are a right eye image and a left eye image for each frame of consecutive frames, and

wherein the parallax amount acquiring section detects a face from each of the right eye image and the left eye image and computes a parallax amount using a characteristic point of the detected face from the right eye image and a characteristic point of the detected face from the left eye image, and wherein the parallax amount acquiring section computes said parallax amount for each of the consecutive frames;

an abnormality judging section that judges that there is an abnormality in the parallax amount in at least one case of a case in which fluctuation of a fixed interval in the parallax amount acquired by the parallax amount acquiring section is greater than a predetermined value, a case in which the parallax amount reaches an allowed limit value that is determined in advance, or a case in which an object of acquisition of the parallax amount can no longer be detected, and judges that there is no abnormality in the parallax amount in a case that is none of a case in which fluctuation of a fixed interval in the parallax amount is greater than a predetermined value, a case in which the parallax amount reaches an allowed limit value that is determined in advance, or a case in which an object of acquisition of the parallax amount can no longer be detected, wherein the abnormality judging section judges whether there is an abnormality in the parallax amount for each of the consecutive frames; and

a parallax adjusting section that judges whether there is an abnormality in the parallax amount, such that;

when it is judged by the abnormality judging section that there is no abnormality in the parallax amount in one or more consecutive frames, the parallax adjusting section carries out a parallax adjustment for each of the consecutive frames for which there is no abnormality in the parallax amount, wherein said parallax adjustment is a first parallax adjustment, and the parallax adjusting section also increases an index number for each of the consecutive frames for which there is no abnormality in the parallax amount, and

when it is judged by the abnormality judging section that there is an abnormality in the parallax amount, the parallax adjusting section switches to control of a second parallax adjustment that is different from the control of the first parallax adjustment, whereby in the control of the second parallax adjustment the parallax adjusting section lowers a parallax adjustment frequency such that the parallax adjusting section carries out said parallax adjustment for each N frame of the consecutive frames, wherein N is a predetermined adjustment parameter higher than 1, such that;

when it is judged by the abnormality judging section that there is an abnormality in the parallax amount in one consecutive frame, the parallax adjusting section increases said index number for said one consecutive frame and checks if said index number corresponding to said one consecutive frame is less than N, and

if said index number is less than N, then the parallax adjusting section does not carry out said parallax adjustment for said one consecutive frame,

but if said index number is at least N, the parallax adjusting section carries out said parallax adjustment for said one consecutive frame.

23. The non-transitory computer-readable medium storing the parallax adjusting program of claim 22, wherein the parallax amount acquiring section computes the parallax amount using a plurality of characteristic points of the detected face from the right eye image, and a plurality of characteristic points of the detected face from the left eye image.

24. The non-transitory computer-readable medium storing the parallax adjusting program of claim 22, wherein, after said index number becomes at least N and the parallax adjusting section carries out said parallax adjustment for said one consecutive frame, said index number is reset.

25. A non-transitory computer-readable medium storing a parallax adjusting program of a composite eye imaging device including computer-readable instructions, the parallax adjusting program when executed causing a computer to function as:

a parallax amount acquiring section that acquires a parallax amount of a predetermined object on the basis of a plurality of viewpoint images generated by imaging sections that generate the plurality of viewpoint images per frame by imaging a same subject from a plurality of viewpoints, wherein the plurality of viewpoint images are a right eye image and a left eye image for each frame of consecutive frames, and

wherein the parallax amount acquiring section detects a face from each of the right eye image and the left eye image and computes a parallax amount using a characteristic point of the detected face from the right eye image and a characteristic point of the detected face from the left eye image, and wherein the parallax amount acquiring section computes said parallax amount for each of the consecutive frames;

an abnormality judging section that judges that there is an abnormality in the parallax amount in at least one case of a case in which fluctuation of a fixed interval in the parallax amount acquired by the parallax amount acquiring section is greater than a predetermined value, a case in which the parallax amount reaches an allowed limit value that is determined in advance, or a case in which an object of acquisition of the parallax amount can no longer be detected, and judges that there is no abnormality in the parallax amount in a case that is none of a case in which fluctuation of a fixed interval in the parallax amount is greater than a predetetinined value, a case in which the parallax amount reaches an allowed limit value that is determined in advance, or a case in which an object of acquisition of the parallax amount can no longer be detected, wherein the abnormality judging section judges whether there is an abnormality in the parallax amount for each of the consecutive frames; and

a parallax adjusting section that judges whether there is an abnormality in the parallax amount, such that;

when it is judged by the abnormality judging section that there is no abnormality in the parallax amount in one or more consecutive frames, the parallax adjusting section carries out a parallax adjustment of the predetermined object for each of the consecutive frames for which there is no abnormality in the parallax amount, wherein said parallax adjustment is a first parallax adjustment, and the parallax adjusting section also increases an index number for each of the consecutive frames for which there is no abnormality in the parallax amount, and

when it is judged by the abnormality judging section that there is an abnormality in the parallax amount, the parallax adjusting section switches to control of a second parallax adjustment that is different from the control of the first parallax adjustment, whereby in the control of the second parallax adjustment the parallax adjusting section lowers a parallax adjustment frequency and carries out parallax adjustment on another object that is different than the predete miined object, such that the parallax adjusting section carries out said parallax adjustment for each N frame of the consecutive frames, wherein N is a predetermined adjustment parameter higher than 1, such that;

when it is judged by the abnormality judging section that there is an abnormality in the parallax amount in one consecutive frame, the parallax adjusting section increases said index number for said one consecutive frame and checks if said index number corresponding to said one consecutive frame is less than N, and

if said index number is less than N, then the parallax adjusting section does not carry out said parallax adjustment for said one consecutive frame,

but if said index number is at least N, the parallax adjusting section carries out said parallax adjustment for said one consecutive frame.

26. The non-transitory computer-readable medium storing the parallax adjusting program of claim 25, wherein the parallax amount acquiring section computes the parallax amount using a plurality of characteristic points of the detected face from the right eye image, and a plurality of characteristic points of the detected face from the left eye image.

27. The non-transitory computer-readable medium storing the parallax adjusting program of claim 25, wherein, after said index number becomes at least N and the parallax adjusting section carries out said parallax adjustment for said one consecutive frame, said index number is reset.