Image processing apparatus that determines processing target area of an image based on degree of saliency, image processing method, and storage medium

• US 10,121,067 B2
• Filed: 09/10/2015
• Issued: 11/06/2018
• Est. Priority Date: 09/30/2014
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Kobayashi, Satoru
• Primary Examiner(s): Bella, Matthew
• Assistant Examiner(s): Shin, Brian
• Application Number:
  US14/850,599
• Publication Number:
  US 20160093064A1
• Time in Patent Office: 1,153 Days
• Field of Search: None
• CPC Class Codes: G02B 7/36 using image sharpness techniques , e.g. image processing techniques for generating autofocus signals image data processing per se G06T; in cameras having a solid state image sensor H04N23/67
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 10/464 using a plurality of salient features, e.g. bag-of-words [BoW] representations
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: H04N 23/61 based on recognised objects
• CPC Class Codes: H04N 23/672 based on the phase difference signals
• CPC Class Codes: H04N 23/71 Circuitry for evaluating the brightness variation

Abstract

An image processing apparatus is provided with a spatial information calculation unit for calculating spatial information of a subject, which is the information of an area in which the subject in an image is predicted to be present, a first area setting unit for setting a first area in the image based on the spatial information, a second area setting unit for setting a second area outside the first area, a first feature amount calculation unit for calculating a first feature amount of the first area, a second feature amount calculation unit for calculating a second feature amount of the second area, the second feature amount being a feature amount of the same type as the first feature amount, and an saliency calculation unit for calculating a degree of visual saliency of the subject.

Claims

1. An image processing apparatus comprising:

at least one processor;

a memory that includes instructions for the at least one processor to, by being executed by the at least one processor;

set a search area based on spatial information of a subject, the spatial information being information of an area in which the subject in an image is predicted to be present;

set a first area inside the search area but not outside the search area;

set a second area outside the first area;

calculate a first feature amount of the first area;

calculate a second feature amount of the second area, the second feature amount being a feature amount of the same type as the first feature amount; and

calculate a degree of visual saliency of the subject in the first area based on a difference between the first feature amount and the second feature amount,

wherein in a case that a plurality of search areas are set, the first area and the second area are set and the degree of visual saliency is calculated in each of the search areas, and a processing target area is determined from the first areas of the plurality of search areas based on the calculated degree of visual saliency.

2. The image processing apparatus according to claim 1,

wherein the spatial information includes at least one of a position and a size of the subject in the image.

3. The image processing apparatus according to claim 2,

wherein a size of the first area is set based on the size of the subject contained in the spatial information.

4. The image processing apparatus according to claim 2,

wherein a position of the first area is set based on the position of the subject contained in the spatial information.

5. The image processing apparatus according to claim 1,

wherein a plurality of first areas are set in the search area.

6. The image processing apparatus according to claim 1, wherein the memory includes an instruction for the at least one processor to, by being executed by the at least one processor:

calculate information of a focused area,

wherein the spatial information of the subject is calculated based on the information of the focused area.

7. The image processing apparatus according to claim 6,

wherein the information of the focused area is calculated based on information indicating whether or not the area is in focus during a focusing operation.

8. The image processing apparatus according to claim 1, wherein the memory includes an instruction for the at least one processor to, by being executed by the at least one processor:

calculate information of an area in which a moving body is present,

wherein the spatial information of the subject is calculated based on the information of the area in which the moving body is present.

9. The image processing apparatus according to claim 8,

wherein the information of the area in which the moving body is present is calculated based on an absolute difference between a plurality of images.

10. The image processing apparatus according to claim 1, wherein the memory includes an instruction for the at least one processor to, by being executed by the at least one processor:

calculate information of an area in which a subject is present,

wherein spatial information of the subject is calculated based on the information of the area in which the subject is present.

11. The image processing apparatus according to claim 10,

wherein the information of the area in which the subject is present is calculated in accordance with at least one of face detection information and human body detection information.

12. The image processing apparatus according to claim 1,

wherein degree of visual saliency of only an area whose area measurement is larger than a predetermined value is calculated.

13. The image processing apparatus according to claim 1,

wherein the first and second feature amounts include at least one of brightness, color, edge intensity, a brightness histogram, a color histogram, and an edge intensity histogram.

14. The image processing apparatus according to claim 1, further comprising a lens group including a focus lens.

15. The image processing apparatus according to claim 14,

wherein the memory includes an instruction for the at least one processor to, by being executed by the at least one processor:

perform an autofocus control using the focus lens so that the processing target area become in-focus state.

16. The image processing apparatus according to claim 1,

wherein the second area is an area whose center is the same as that of the first area and is a remaining area of an area that is larger than the first area excluding the first area.

17. An image processing method comprising:

setting a search area based on spatial information of a subject, the spatial information being information of an area in which a subject in an image is predicted to be present;

setting a first area inside the search area but not outside the search area;

setting a second area outside the first area;

calculating a first feature amount of the first area;

calculating a second feature amount of the second area, the second feature amount being a feature amount of the same type as the first feature amount; and

calculating a degree of visual saliency of the subject in the first area based on a difference between the first feature amount and the second feature amount,

wherein in a case that a plurality of search areas are set, the first area and the second area are set and the degree of visual saliency is calculated in each of the search areas, and a processing target area is determined from the first areas of the plurality of search areas based on the calculated degree of visual saliency.

18. A non-transitory computer readable storage medium storing a computer-executable program for executing a method for controlling an image processing apparatus using the computer,

the method comprising:

setting a search area based on spatial information of a subject, the spatial information being information of an area in which a subject in an image is predicted to be present;

setting a first area inside the search area but not outside the search area;

setting a second area outside the first area;

calculating a first feature amount of the first area;

calculating a second feature amount of the second area, the second feature amount being a feature amount of the same type as the first feature amount; and

calculating a degree of visual saliency of the subject in the first area based on a difference between the first feature amount and the second feature amount,

wherein in a case that a plurality of search areas are set, the first area and the second area are set and the degree of visual saliency is calculated in each of the search areas, and a processing target area is determined from the first areas of the plurality of search areas based on the calculated degree of visual saliency.