Method, apparatus, and program for detecting object

• US 8,577,151 B2
• Filed: 04/01/2009
• Issued: 11/05/2013
• Est. Priority Date: 04/04/2008
• Status: Active Grant

• Current Assignee: Fujifilm Corporation (Fujifilm Holdings Corporation)
• Original Assignee: Fujifilm Corporation (Fujifilm Holdings Corporation)
• Inventors: Hu, Yi
• Primary Examiner(s): Zarka, David
• Assistant Examiner(s): LE, HA N
• Application Number:
  US12/921,502
• Publication Number:
  US 20110019920A1
• Time in Patent Office: 1,679 Days
• Field of Search: None
• US Class Current: 382/195
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/167 using comparisons between temporally consecutive images

Abstract

A region where a detecting target object exists is extracted by a comparison between an evaluated value indicating a probability that the detecting target object exists and a threshold, through the process of producing a differential image between different frames in plural frames constituting a continuous image, of setting an average value in an averaging region extended around each pixel of the differential image as a new value of each pixel, of obtaining the evaluated value by applying a filter that acts on a search region on an image to a search region extended around a search pixel that is extracted by comparing the new value and the threshold, on the differential image.

Claims

1. An object detecting method that detects a specific kind of object from a continuous image including a plurality of frames each of which is expressed by two-dimensionally arrayed pixels, the object detecting method comprising:

a differential image producing step of producing a differential image between a first frame and a second frame in the plurality of frames constituting the continuous image, the first frame and the second frame being different from each other;

a search pixel extracting step of computing a differential value between new values of pixels corresponding to the first frame and the second frame, of comparing the differential value and a threshold, and of extracting a search pixel on the differential image, when an average value of values of a plurality of pixels in an averaging region having a two-dimensional predetermined size around each pixel for each of the first frame and the second frame is set at a new value of each pixel, the search pixel searching existence of the specific kind of object; and

an object detecting step of applying a filter to a search region including the search pixel extracted in the search pixel extracting step on the differential image produced in the differential image producing step, of obtaining an evaluated value with the filter, and of comparing the evaluated value and the threshold to extract a region where the specific kind of object exists, the filter acting on a search region having an area and a shape that are identical to those of the averaging region on the image, the filter obtaining the evaluated value indicating a probability that the specific kind of object exists in the search region,

wherein the differential image producing step is a step of producing a differential image between the first frame and the second frame and producing a differential image group from a plurality of differential images by producing at least one differential image between a first thin-out frame and a second thin-out frame, the plurality of differential images having an identical scene and different resolutions, the first thin-out frame and the second thin-out frame having an identical size, the first thin-out frame and the second thin-out frame being formed by thinning out the pixels constituting each frame at a predetermined ratio for each of the first frame and the second frame or by gradually thinning out the pixels at the predetermined ratio,

the object detecting step is a step of detecting a specific kind of object from the differential images by sequentially repeating a plurality of extraction processes from an extraction process of applying a filter acting on a relatively narrow search region to a relatively small differential image toward an extraction process of applying a filter acting on a relatively wide search region to a relatively differential large image,

the plurality of extraction processes including a first extraction process and a second extraction process,

in the first extraction process, a primary candidate region being extracted by applying a first filter to a first differential image that is relatively small in the differential image group produced in the differential image producing step, an evaluated value exceeding a predetermined first threshold being obtained in the primary candidate region, the first filter acting on a relatively narrow search region in a filter group including a plurality of filters acting on a plurality of search regions having a plurality of sizes, the number of pixels corresponding to the size of the region on the image being changed at the predetermined ratio or gradually changed at the predetermined ratio in the plurality of search regions having the plurality of sizes, the plurality of filters acting on the search region that is two-dimensionally extended on the image, the plurality of filters producing an evaluated value indicating a probability that the specific kind of object exists in the search region,

in the second extraction process, a secondary candidate region being extracted by applying a second filter to a region corresponding to the primary candidate region in a second differential image in which the number of pixels is larger by one stage than that of the first differential image in the differential image group produced in the differential image producing step, the evaluated value exceeding a predetermined second threshold being obtained in the secondary candidate region, the second filter acting on the search region that is wider by one stage than that of the first filter in the filter group,

the search pixel extracting step is a step of employing an averaging region having an area and a shape that are identical to those of a maximum search region to which a maximum region filter acting on the maximum search region in the filter group and of extracting the search pixel from the differential image to which the maximum region filter is applied in the object detecting step in the plurality of differential images constituting the differential image group produced in the differential image producing step, and

the object detecting step is a step of extracting the primary candidate region from the search region by applying the first filter to the search region on the first differential image in the first extraction process, the search region including the pixel corresponding to the search pixel extracted in the search pixel extracting step.

2. The object detecting method according to claim 1, wherein the differential image producing step is a step of producing the differential image group and of producing newly at least one differential image group, the at least one differential image group including one interpolated differential image or a plurality of interpolated differential images and at least one thinned-out differential image, the one interpolated differential image being obtained by an interpolation operation, the one interpolated differential image having a range of the number of pixels that is smaller than the number of pixels of the maximum-area differential image constituting the differential image group and larger than the number of pixels of the thinned-out differential image obtained by thinning out the maximum-area differential image at the predetermined ratio, the number of pixels being changed within the range in the plurality of interpolated differential images, the at least one thinned-out differential image being formed by thinning out the pixels constituting interpolated differential image at the predetermined ratio for each of the interpolated differential images or formed by gradually thinning out the pixels at the predetermined ratio, and

the object detecting step is a step of detecting the specific kind of object by sequentially repeating the extraction process from the plurality of extraction processes of applying the filter acting on the relatively narrow search region to the relatively small image toward the extraction process of applying the filter acting on the relatively wide search region to the relatively large image to each of the plurality of differential image groups produced in the differential image producing step.

3. The object detecting method according to claim 1, wherein a plurality of kinds of filters is prepared for each search region having one size, each of the plurality of kinds of filters computing an outline of the specific kind of object and one of feature quantities in the specific kind of object, and a correspondence relationship between the feature quantity computed by each filter and the primary evaluated value indicating the probability of the specific kind of object is prepared, and

the object detecting step is a step of computing the plurality of feature quantities by applying the plurality of kinds of filters to one search region according to the size of the search region, of obtaining each primary evaluated value corresponding to each feature quantity, of comparing a secondary evaluated value in which the plurality of primary evaluated values is integrated and a threshold, and of determining whether or not the search region is a candidate region where the specific kind of object exists.

4. The object detecting method according to claim 1, further comprising a region integrating step of integrating a plurality of regions into one region according to a degree of overlapping of the plurality of regions when the plurality of regions is detected in the object detecting step.

5. The object detecting method according to claim 1, wherein the filter group includes a plurality of filters that produce an evaluated value indicating a probability of existence of a human head, and

the human head emerging in an image is set at a detecting target in the object detecting method.

6. An object detecting apparatus that detects a specific kind of object from a continuous image including a plurality of frames each of which is expressed by two-dimensionally arrayed pixels, the object detecting apparatus comprising:

a differential image producing section that produces a differential image between a first frame and a second frame in the plurality of frames constituting the continuous image, the first frame and the second frame being different from each other;

a search pixel extracting section that computes a differential value between new values of pixels corresponding to the first frame and the second frame, compares the differential value and a threshold, and extracts a search pixel on the differential image, when an average value of values of a plurality of pixels in an averaging region having a two-dimensional predetermined size around each pixel for each of the first frame and the second frame is set at a new value of each pixel, the search pixel searching existence of the specific kind of object;

a filter storage section in which a filter is stored, the filter acting on a search region having an area and a shape that are identical to those of the averaging region on the image, the filter obtaining an evaluated value indicating a probability that the specific kind of object exists in the search region; and

an object detecting section that applies a filter stored in the filter storage section to the search region including the search pixel extracted by the search pixel extracting section on the differential image produced by the differential image producing section, obtains the evaluated value with the filter, and compares the evaluated value and the threshold to extract a region where the specific kind of object exists,

wherein the differential image producing section produces a differential image between the first frame and the second frame and produces a differential image group from a plurality of differential images by producing at least one differential image between a first thin-out frame and a second thin-out frame, the plurality of differential images having an identical scene and different resolutions, the first thin-out frame and the second thin-out frame having an identical size, the first thin-out frame and the second thin-out frame being formed by thinning out the pixels constituting each frame at a predetermined ratio for each of the first frame and the second frame or by gradually thinning out the pixels at the predetermined ratio,

a filter group including a plurality of filters is stored in the filter storage section, the plurality of filters acting on a plurality of search regions having a plurality of sizes, the number of pixels corresponding to the size of the region on the image being changed at the predetermined ratio or gradually changed at the predetermined ratio in the plurality of search regions having the plurality of sizes, the plurality of filters acting on the search region that is two-dimensionally extended on the image, the plurality of filters producing an evaluated value indicating a probability that the specific kind of object exists in the search region,

the object detecting section detects the specific kind of object from the differential images by sequentially repeating a plurality of extraction processes from an extraction process of applying a filter acting on a relatively narrow search region to a relatively small differential image toward an extraction process of applying a filter acting on a relatively wide differential region to a relatively large search image,

the plurality of extraction processes including a first extraction process and a second extraction process,

in the first extraction process, a primary candidate region being extracted by applying a first filter to a first differential image that is relatively small in the differential image group produced in the differential image producing step, an evaluated value exceeding a predetermined first threshold being obtained in the primary candidate region, the first filter acting on a relatively narrow search region in a filter group stored in the filter storage section,

in the second extraction process, a secondary candidate region being extracted by applying a second filter to a region corresponding to the primary candidate region in a second differential image in which the number of pixels is larger by one stage than that of the first differential image in the differential image group produced by the differential image producing section, the evaluated value exceeding a predetermined second threshold being obtained in the secondary candidate region, the second filter acting on the search region that is wider by one stage than that of the first filter in the filter group,

the search pixel extracting section employs an averaging region having an area and a shape that are identical to those of a maximum search region to which a maximum region filter acting on the maximum search region in the filter group and extracts the search pixel from the differential image to which the maximum region filter is applied by the object detecting section in the plurality of differential images constituting the differential image group produced by the differential image producing section, and

the object detecting section extracts the primary candidate region from the search region by applying the first filter to the search region on the first differential image in the first extraction process, the search region including the pixel corresponding to the search pixel extracted by the search pixel extracting section.

7. The object detecting apparatus according to claim 6, wherein the differential image producing section produces the differential image group and newly at least one differential image group, the at least one differential image group including one interpolated differential image or a plurality of interpolated differential images and at least one thinned-out differential image, the one interpolated differential image being obtained by an interpolation operation, the one interpolated differential image having a range of the number of pixels that is smaller than the number of pixels of the maximum-area differential image constituting the differential image group and larger than the number of pixels of the thinned-out differential image obtained by thinning out the maximum-area differential image at the predetermined ratio, the number of pixels being changed within the range in the plurality of interpolated differential images, the at least one thinned-out differential image being formed by thinning out the pixels constituting interpolated differential image at the predetermined ratio for each of the interpolated differential images or formed by gradually thinning out the pixels at the predetermined ratio, and

the object detecting section detects the specific kind of object by sequentially repeating the plurality of extraction processes from the extraction process of applying the filter acting on the relatively search narrow region to the relatively small image toward the extraction process of applying the filter acting on the relatively wide search region to the relatively large image to each of the plurality of differential image groups produced by the differential image producing section.

8. The object detecting apparatus according to claim 6, wherein a plurality of kinds of filters for each search region having one size is stored in the filter storage section, each of the plurality of kinds of filters detecting an outline of the specific kind of object and one of feature quantities in the specific kind of object, and a correspondence relationship between the feature quantity computed by each filter and the primary evaluated value indicating a probability of the specific kind of object is stored in the filter storage section, and

the object detecting section computes the plurality of feature quantities by applying the plurality of kinds of filters to one search region according to the size of the search region, obtains each primary evaluated value corresponding to each feature quantity, compares a secondary evaluated value in which the plurality of primary evaluated values is integrated and a threshold, and determines whether or not the search region is a candidate region where the specific kind of object exists.

9. The object detecting apparatus according to claim 6, further comprising a region integrating section that integrates a plurality of regions into one region according to a degree of overlapping of the plurality of regions when the plurality of regions is detected by the object detecting section.

10. The object detecting apparatus according to claim 6, wherein the filter group including a plurality of filters that produces an evaluated value indicating a probability of existence of a human head is stored in the filter storage section, and

the object detecting apparatus sets the human head emerging in an image at a detecting target.

11. A non-transitory computer-readable storage medium storing an object detecting program that is executed in an operation device, the object detecting program causing the operation device to be operated as a object detecting apparatus that detects a specific kind of object from a continuous image including a plurality of frames each of which is expressed by two-dimensionally arrayed pixels, the object detecting apparatus comprising:

a differential image producing section that produces a differential image between a first frame and a second frame in the plurality of frames constituting the continuous image, the first frame and the second frame being different from each other;

a search pixel extracting section that computes a differential value between new values of pixels corresponding to the first frame and the second frame, compares the differential value and a threshold, and extracts a search pixel on the differential image, when an average value of values of a plurality of pixels in an averaging region having a two-dimensional predetermined size around each pixel for each of the first frame and the second frame is set at a new value of each pixel, the search pixel searching existence of the specific kind of object;

a filter storage section in which a filter is stored, the filter acting on a search region having an area and a shape that are identical to those of the averaging region on the image, the filter obtaining an evaluated value indicating a probability that the specific kind of object exists in the search region; and

an object detecting section that applies a filter stored in the filter storage section to the search region including the search pixel extracted by the search pixel extracting section on the differential image produced by the differential image producing section, obtains the evaluated value with the filter, and compares the evaluated value and the threshold to extract a region where the specific kind of object exists,

wherein the differential image producing section produces a differential image between the first frame and the second frame and produces a differential image group from a plurality of differential images by producing at least one differential image between a first thin-out frame and a second thin-out frame, the plurality of differential images having an identical scene and different resolutions, the first thin-out frame and the second thin-out frame having an identical size, the first thin-out frame and the second thin-out frame being formed by thinning out the pixels constituting each frame at a predetermined ratio for each of the first frame and the second frame or by gradually thinning out the pixels at the predetermined ratio,

a filter group including a plurality of filters is stored in the filter storage section, the plurality of filters acting on a plurality of search regions having a plurality of sizes, the number of pixels corresponding to the size of the region on the image being changed at the predetermined ratio or gradually changed at the predetermined ratio in the plurality of search regions having the plurality of sizes, the plurality of filters acting on the search region that is two-dimensionally extended on the image, the plurality of filters producing an evaluated value indicating a probability that the specific kind of object exists in the search region,

the object detecting section detects the specific kind of object from the differential images by sequentially repeating a plurality of extraction processes from an extraction process of applying a filter acting on a relatively narrow search region to a relatively small differential image toward an extraction process of applying a filter acting on a relatively wide search region to a relatively large differential image,

the plurality of extraction processes including a first extraction process and a second extraction process,

in the first extraction process, a primary candidate region being extracted by applying a first filter to a first differential image that is relatively small in the differential image group produced in the differential image producing step, an evaluated value exceeding a predetermined first threshold being obtained in the primary candidate region, the first filter acting on a relatively narrow search region in a filter group stored in the filter storage section,

in the second extraction process, a secondary candidate region being extracted by applying a second filter to a region corresponding to the primary candidate region in a second differential image in which the number of pixels is larger by one stage than that of the first differential image in the differential image group produced by the differential image producing section, the evaluated value exceeding a predetermined second threshold being obtained in the secondary candidate region, the second filter acting on the search region that is wider by one stage than that of the first filter in the filter group,

the search pixel extracting section employs an averaging region having an area and a shape that are identical to those of a maximum search region to which a maximum region filter acting on the maximum search region in the filter group and extracts the search pixel from the differential image to which the maximum region filter is applied by the object detecting section in the plurality of differential images constituting the differential image group produced by the differential image producing section, and

the object detecting section extracts the primary candidate region from the search region by applying the first filter to the search region on the first differential image in the first extraction process, the search region including the pixel corresponding to the search pixel extracted by the search pixel extracting section.

12. The non-transitory computer-readable storage medium according to claim 11, wherein the differential image producing section produces the differential image group and newly at least one differential image group, the at least one differential image group including one interpolated differential image or a plurality of interpolated differential images and at least one thinned-out differential image, the one interpolated differential image being obtained by an interpolation operation, the one interpolated differential image having a range of the number of pixels that is smaller than the number of pixels of the maximum-area differential image constituting the differential image group and larger than the number of pixels of the thinned-out differential image obtained by thinning out the maximum-area differential image at the predetermined ratio, the number of pixels being changed within the range in the plurality of interpolated differential images, the at least one thinned-out differential image being formed by thinning out the pixels constituting interpolated differential image at the predetermined ratio for each of the interpolated differential images or formed by gradually thinning out the pixels at the predetermined ratio, and

the object detecting section detects the specific kind of object by sequentially repeating the extraction process from the plurality of extraction processes of applying the filter acting on the relatively narrow search region to the relatively small image toward the extraction process of applying the filter acting on the relatively wide search region to the relatively large image to each of the plurality of differential image groups produced by the differential image producing section.

13. The non-transitory computer-readable storage medium according to claim 11, wherein a plurality of kinds of filters for each search region having one size is stored in the filter storage section, each of the plurality of kinds of filters detecting an outline of feature quantities of and one of feature quantities in the specific kind of object, and a correspondence relationship between the feature quantity computed by each filter and the primary evaluated value indicating a probability of the specific kind of object is stored in the filter storage section, and

the object detecting section computes the plurality of feature quantities by applying the plurality of kinds of filters to one search region according to the size of the search region, obtains each primary evaluated value corresponding to each feature quantity, compares a secondary evaluated value in which the plurality of primary evaluated values is integrated and a threshold, and determines whether or not the search region is a candidate region where the specific kind of object exists.

14. The non-transitory computer-readable storage medium according to claim 11, wherein the operation device is operated as the object detecting apparatus further including a region integrating section that integrates a plurality of regions into one region according to a degree of overlapping of the plurality of regions when the plurality of regions is detected by the object detecting section.

15. The non-transitory computer-readable storage medium according to claim 11, wherein the filter group including a plurality of filters that produce an evaluated value indicating a probability of existence of a human head is stored in the filter storage section, and

the object detecting program causes the operation device to be operated as the object detecting apparatus that sets the human head emerging in an image at a detecting target.