Image recognition and focus adjustment using weighted image difference information

• US 8,325,977 B2
• Filed: 06/18/2008
• Issued: 12/04/2012
• Est. Priority Date: 07/03/2007
• Status: Active Grant

• Current Assignee: Nikon Corporation
• Original Assignee: Nikon Corporation
• Inventors: Muramatsu, Keiko
• Primary Examiner(s): DRENNAN, BARRY T
• Application Number:
  US12/141,480
• Publication Number:
  US 20090010492A1
• Time in Patent Office: 1,630 Days
• Field of Search: None
• US Class Current: 382/103
• CPC Class Codes: G02B 7/34 using different areas in a pupil plane
• CPC Class Codes: G06T 7/248 involving reference images or patches

Abstract

An image recognition device includes a detection unit which is configured to detect a first difference between partial information of at least a part of the first image information and the reference information and to detect a second difference between partial information of at least a part of the second image information and the reference information. A recognition unit is configured to recognize a first area corresponding to the reference image in the first image information. A calculation unit is configured to calculate a determination value based on a reference area in the second image information corresponding to the first area by weighting the second difference. The recognition unit is configured to recognize a second area corresponding to the reference image in the second image information based on at least one of the second difference and the determination value.

Claims

1. An image recognition device comprising:

an image information acquiring unit configured to acquire first image information and second image information as a first image and a second image, respectively, the first image including a reference image to be recognized on the second image;

a holding unit configured to hold reference information of the reference image on the first image, the reference image information including a first reference position on the first image;

a detection unit configured to detect a difference between the reference image and a plurality of image areas on the second image respectively using the reference image information and the second image information;

a calculation unit configured to calculate a determination value at the plurality of image areas on the second image by weighting the difference detected by the detection unit,

wherein the weighting of the difference includes multiplying a weight by a weighting coefficient,

the weight being assigned to each of the plurality of image areas on the second image and determined based on a relationship between a position of an image area based on which the difference is detected and a second reference position on the second image that corresponds to the first reference position on the first image, and

the weight coefficient being adjusted based on the difference detected by the detection unit; and

a recognition unit configured to recognize an image area from among the plurality of image areas on the second image based on the determination value as an image area on the second image that corresponds to the reference image on the first image.

2. The image recognition device according to claim 1,

wherein the calculation unit is configured to assign the weight to each of the plurality of image areas on the second image so that the determination value becomes relatively smaller as an image area based on which the difference is detected is closer to the second reference position on the second image, and

wherein the recognition unit is configured to recognize an image area having a smallest determination value as an image area on the second image that corresponds to the reference image on the first image.

3. The image recognition device according to claim 1,

wherein the calculation unit is configured to assign the weight to each of the plurality of image areas on the second image depending on a distance between the second reference position on the second image and a position of an image area based on which the difference is detected.

4. The image recognition device according to claim 3,

wherein the calculation unit is configured to assign the weight to each of the plurality of image areas on the second image so that the determination value increases as the distance increases, and

wherein the recognition unit is configured to recognize an image area having a smallest determination value as an image area on the second image that corresponds to the reference image on the first image.

5. The image recognition device according to claim 1,

wherein the first reference position on the first image is a center of the first image, and

wherein the calculation unit is configured to assign the weight to each of the plurality of image areas on the second image depending on a distance between a center of the second image and a position of an image area based on which the difference is detected.

6. The image recognition device according to claim 5,

wherein the calculation unit is configured to assign the weight to each of the plurality of image areas on the second image so that the determination value decreases as an image area based on which the difference is detected approaches the center of the second image, and

wherein the recognition unit is configured to recognize an image area having a smallest determination value as an image area on the second image that corresponds to the reference image on the first image.

7. The image recognition device according to claim 1,

wherein the plurality of image areas on the second image are arranged within a predetermined range from the second reference position on the second image.

8. The image recognition device according to claim 1,

wherein the detection unit is configured to detect the difference using at least one of color information and luminance information of the reference image on the first image and the plurality of image areas on the second image.

9. The image recognition device according to claim 1, further comprising:

an updating unit configured to combine the reference image information with image information of the image area recognized on the second image, and to update the reference image information of the reference image held on the holding unit with the combined reference image information.

10. A focus adjustment device comprising:

an image information acquiring unit configured to acquire first image information and second image information as a first image and a second image, respectively, through an optical system, the first image including a reference image to be recognized on the second image;

a holding unit configured to hold reference information of the reference image on the first image, the reference information including a first reference position on the first image;

a detection unit configured to detect a difference between the reference image and a plurality of image areas on the second image respectively using the reference image information and the second image information;

a calculation unit configured to calculate a determination value at the plurality of image areas on the second image by weighting the difference detected by the detection unit,

wherein the weighting of the difference includes multiplying a weight by a weighting coefficient,

the weight being assigned to each of the plurality of image areas on the second image and determined based on a relationship between a position of an image area based on which the difference is detected and a second reference position on the second image that corresponds to the first reference position on the first image, and

the weight coefficient being adjusted based on the difference detected by the detection unit;

a recognition unit configured to recognize an image area from among the plurality of image areas on the second image based on the determination value as an image area on the second image that corresponds to the reference image on the first image;

a focus detection unit configured to detect a focus state of the optical system at a plurality of detection areas on the second image by the optical system;

a focus area selection unit configured to select at least one focus detection area from among the plurality of detection areas on the second image based on a position of the image area on the second image recognized by the recognition unit and the focus state detected by the focus detection unit; and

a focus adjustment unit configured to perform focus adjustment of the optical system at the focus detection area selected by the selection unit.

11. The focus adjustment device according to claim 10, further comprising:

a specifying unit configured to specify at least one focus detection area from among the plurality of detection areas on the first image when the first image is captured,

wherein the first reference position on the first image is a position of the at least one focus detection area on the first image, and

wherein the calculation unit is configured to assign the weight to each of the plurality of image areas on the second image depending on a distance between the second reference position on the second image and a position of an image area based on which the difference is detected.

12. The focus adjustment device according to claim 11,

wherein the calculation unit is configured to assign the weight to each of the plurality of image areas on the second image so that the determination value decreases as the distance decreases, and

wherein the recognition unit is configured to recognize an image area having a smallest determination value as an image area on the second image that corresponds to the reference image on the first image.

13. The focus adjustment device according to claim 10, further comprising:

an updating unit configured to combine the reference image information with image information of the image area recognized on the second image, and to update the reference image information of the reference image held on the holding unit with the combined reference image information.

14. The focus adjustment device according to claim 10,

wherein the detection unit is configured to detect the difference using at least one of color information and luminance information of the reference image on the first image and the plurality of image areas on the second image.

15. An imaging apparatus comprising:

an image information acquiring unit configured to acquire first image information and second image information as a first image and a second image, respectively, through an optical system, the first image including a reference image to be recognized on the second image;

a holding unit configured to hold reference information of the reference image on the first image, the reference information including a first reference position of the reference image on the first image;

a detection unit configured to detect a difference between the reference image and a plurality of image areas on the second image respectively using the reference image information and the second image information;

a calculation unit configured to calculate a determination value at the plurality of image areas on the second image by weighting the difference detected by the detection unit,

wherein the weighting of the difference includes multiplying a weight by a weighting coefficient,

the weight being assigned to each of the plurality of image areas on the second image and determined based on a relationship between a position of an image area based on which the difference is detected and a second reference position on the second image that corresponds to the first reference position on the first image, and

the weight coefficient being adjusted based on the difference detected by the detection unit;

a recognition unit configured to recognize an image area from among the plurality of image areas on the second image based on the determination value as an image area on the second image that corresponds to the reference image on the first image;

a focus detection unit configured to detect a focus state of the optical system at a plurality of detection areas provided on the second image by the optical system;

a focus area selection unit configured to select at least one focus detection area from among the plurality of detection areas on the second image based on a position of the image area on the second image recognized by the recognition unit and the focus state detected by the focus detection unit; and

a focus adjustment unit configured to perform focus adjustment of the optical system at the focus detection area selected by the selection unit.

16. The imaging apparatus according to claim 15, further comprising:

an updating unit configured to combine the reference image information with image information of the image area recognized on the second image, and to update the reference image information of the reference image held on the holding unit with the combined reference image information.

17. The imaging apparatus according to claim 15,

wherein the detection unit is configured to detect the difference using at least one of color information and luminance information of the reference image on the first image and the plurality of image areas on the second image.

18. An image recognition method comprising:

acquiring first image information and second image information as a first image and a second image, respectively, the first image including a reference image to be recognized on the second image;

holding reference information of the reference image on the first image, the reference information including a first reference position on the first image;

detecting a difference between the reference image and a plurality of image areas on the second image respectively using reference information and the second image information;

calculating a determination value at the plurality of image areas on the second image by weighting the difference,

wherein the weighting of the difference includes multiplying a weight by a weighting coefficient,

the weight being assigned to each of the plurality of image areas on the second image and determined based on a relationship between a position of an image area based on which the second difference is detected and a second reference position on the second image that corresponds to the first reference position on the first image, and

the weight coefficient being adjusted based on the difference detected; and

recognizing an image area from among the plurality of image areas on the second image based on the determination value as an image area on the second image that corresponds to the reference image on the first image.

19. The image recognition method according to claim 18,

wherein, in the step of calculating the determination value, the weight is assigned to each of the plurality of image areas on the second image so that the determination value becomes relatively smaller as an image area based on which the difference is detected is closer to the second reference position on the second image, and

wherein, in the step of recognizing the image area from among the plurality of image areas on the second image, an image area having a smallest determination value is recognized an image area on the second image that corresponds to the reference image on the first image.

20. The image recognition method according to claim 18,

wherein, in the step of calculating the determination value, the weight is assigned to each of the plurality of image areas on the second image depending on a distance between the second reference position on the second image and a position of an image area based on which the difference is detected.

21. The image recognition method according to claim 20,

wherein, in the step of calculating the determination value, the weight is assigned to each of the plurality of image areas on the second image so that the determination value is increases as the distance increases, and

wherein, in the step of recognizing the image area from among the plurality of image areas on the second image, an image area having a smallest determination value is recognized as an image area on the second image that corresponds to the reference image on the first image.

22. The image recognition method according to claim 18,

wherein the first reference position on the first image is a center of the first image, and

wherein, in the step of calculating the determination value, the weight is assigned to each of the plurality of image areas on the second image depending on a distance between a center of the second image and a position of an image area based on which the difference is detected.

23. The image recognition method according to claim 22,

wherein, in the step of calculating the determination value, the weight is assigned to each of the plurality of image areas on the second image so that the determination value decreases as an image area based on which the difference is detected approaches the center of the second image, and

wherein, in the step of recognizing the image area from among the plurality of image areas on the second image, an image area having a smallest determination value is recognized as an image area on the second image that corresponds to the reference image on the first image.

24. The image recognition method according to claim 18,

wherein the plurality of image areas on the second image are arranged within a predetermined range from the second reference position on the second image.

25. The image recognition method according to claim 18, further comprising:

combining the reference image information with image information of the image area recognized on the second image; and

updating the reference image information of the reference image with the combined reference image information.

26. The image recognition method according to claim 18,

wherein, in the step of detecting the difference, the difference are detected using at least one of color information and luminance information of the reference image on the first image and the plurality of image areas on the second image.

27. A focus adjustment method comprising:

acquiring first image information and second image information as a first image and a second image, respectively, through an optical system, the first image including a reference image to be recognized on the second image;

holding reference information of the reference image on the first image, the reference information including a first reference position on the first image;

detecting a difference between the reference image and a plurality of image areas on the second image respectively using the reference image information and the second image information;

calculating a determination value at the plurality of image areas on the second image by weighting the difference,

wherein the weighting of the difference includes multiplying a weight by a weighting coefficient,

the weight being assigned to each of the plurality of image areas on the second image and determined based on a relationship between a position of an image area based on which the difference is detected and a second reference position on the second image that corresponds to the first reference position on the first image, and

the weight coefficient being adjusted based on the difference detected by the detection unit;

recognizing an image area from among the plurality of image areas on the second image based on the determination value as an image area on the second image that corresponds to the reference image on the first image;

detecting a focus state of the optical system at a plurality of detection areas on the second image by the optical system;

selecting at least one focus detection area from among the plurality of detection areas on the second image based on the image area recognized on the second image and the focus state; and

performing focus adjustment of the optical system at the focus detection area selected by the selection unit.

28. The focus adjustment method according to claim 27, further comprising:

specifying at least one focus detection area from among the plurality of detection areas on the first image when the first image is captured,

wherein the first reference position on the first image is a position of the at least one focus detection area specified on the first image; and

wherein, in the step of calculating the determination value, the weight is assigned to each of the plurality of image areas on the second image depending on a distance between the second reference position on the second image and a position of an image area based on which the difference is detected.

29. The focus adjustment method according to claim 28,

wherein, in the step of calculating the determination value, the weight is assigned to each of the plurality of image areas on the second image so that the determination value decreases as the distance decreases.

30. The focus adjustment method according to claim 27, further comprising:

combining the reference image information with image information of the image area recognized on the second image; and

updating the reference image information of the reference image with the combined reference image information.

31. The focus adjustment method according to claim 27,

wherein, in the step of detecting the difference, the difference are detected using at least one of color information and luminance information of the reference image on the first image and the plurality of image areas on the second image.

32. An image recognition device comprising:

imaging means for acquiring first image information and second image information as a first image and a second image, respectively, the first image including a reference image to be recognized on the second image;

holding means for holding reference information of the reference image on the first image, the reference image information including a first reference position on the first image;

detection means for detecting a difference between the reference image and a plurality of image areas on the second image respectively using the reference image information and the second image information;

calculation means for calculating a determination value at the plurality of image areas on the second image by weighting the difference detected by the detection unit,

wherein the weighting of the difference includes multiplying a weight by a weighting coefficient,

the weight being assigned to each of the plurality of image areas on the second image and determined based on a relationship between a position of an image area based on which the difference is detected and a second reference position on the second image that corresponds to the first reference position on the first image, and

the weight coefficient being adjusted based on the difference detected by the detection unit; and

a recognition means for recognizing an image area from among the plurality of image areas on the second image based on the determination value as an image area on the second image that corresponds to the reference image on the first image.

33. The image recognition device according to claim 32, further comprising:

an updating means for combining the reference image information with image information of the image area recognized on the second image, and updating the reference image information of the reference image held on the holding means with the combined reference image information.

34. The image recognition device according to claim 32,

wherein the detection means detects the difference using at least one of color information and luminance information of the reference image on the first image and the plurality of image areas on the second image.