IMAGE PROCESSING METHOD, IMAGE PROCESSING APPARATUS, AND RECORDING MEDIUM

• US 20170262727A1
• Filed: 03/06/2017
• Published: 09/14/2017
• Est. Priority Date: 03/11/2016
• Status: Active Grant

• Current Assignee: Panasonic Intellectual Property Corporation of America (Panasonic Holdings Corporation)
• Original Assignee: Panasonic Intellectual Property Corporation of America (Panasonic Holdings Corporation)
• Inventors: KOZUKA, KAZUKI, TANIGAWA, TORU, SAITO, MASAHIKO
• Granted Patent: US 10,210,420 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 18/2155 characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06V 10/7753 Incorporation of unlabelled data, e.g. multiple instance learning [MIL]
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G08G 1/04 using optical or ultrasonic detectors
• CPC Class Codes: G08G 1/166 for active traffic, e.g. moving vehicles, pedestrians, bikes
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

An image processing method includes acquiring consecutive time-series images captured by an onboard camera and including at least one image having a first annotation indicating a first region; determining, for each of the images, in reverse chronological order from an image of the last time point, whether the first region exists in the image based on whether the first annotation is attached; identifying the first image of a first time point for which the first region is determined not to exist, and setting a second region including a partial region of an object in the identified first image, indicating the moving object that is obstructed by the object before appearing on the path, and having dimensions based on dimensions of the first region in an image of a second time point immediately after the first time point; and attaching a second annotation to the image corresponding to the second time point, the second annotation indicating the second region.

Claims

1. An image processing method comprising:

acquiring a plurality of consecutive time-series images that have been captured by an onboard camera mounted on a vehicle and that include at least one image to which a first annotation is attached, the first annotation indicating a first region representing a moving object that is present in the vicinity of an object and on a path of the vehicle in the at least one image;

determining, for each of the plurality of consecutive time-series images acquired in the acquiring, in reverse chronological order from an image corresponding to the last time point in the time series, whether the first region exists in the image on the basis of whether the first annotation is attached to the image;

identifying a first image corresponding to a first time point for which it is determined for the first time in the determining that the first region does not exist from among the plurality of consecutive time-series images, and setting a second region including a part of a region of the object in the identified first image corresponding to the first time point and indicating a situation where the moving object is obstructed by the object before appearing on the path from behind the object, the second region having dimensions based on dimensions of the first region in an image corresponding to a second time point that is the next time point after the first time point in the time series; and

attaching a second annotation to the image corresponding to the second time point, the second annotation indicating the second region set in the identifying and setting.

2. The image processing method according to claim 1, wherein in the identifying and setting, the second region including a part of the region of the object in the identified first image corresponding to the first time point is set by shifting the first region in the image corresponding to the second time point in a direction from the first region toward the object by a predetermined distance.

3. The image processing method according to claim 1, wherein in the identifying and setting, one or more images are identified in a range from the first time point of the first image to a time point that is a predetermined period before the first time point in the time series, and the second region including a part of the region of the object is set in the one or more identified images.

4. The image processing method according to claim 1, further comprising:

performing a first extracting process of selecting, from among all of consecutive time-series images that have been captured by the onboard camera mounted on the vehicle and that are associated with information representing braking force or acceleration of the vehicle, first extracted images that are a plurality of images up to a time point that precedes, by a predetermined period, a time point at which the braking force or acceleration of the vehicle is larger than a threshold; and

performing a second extracting process of extracting the plurality of consecutive time-series images including at least one image to which the first annotation is attached from among all of the consecutive time-series images by selecting, from among the first extracted images selected through the first extracting process, a plurality of consecutive time-series images including one or more images to which an annotation indicating a region representing a moving object that is present on the path of the vehicle is attached,

wherein in the acquiring, the plurality of consecutive time-series images extracted through the second extracting process are acquired.

5. The image processing method according to claim 4, further comprising:

causing crowdsourcing workers to attach, to each of the all of the consecutive time-series images, an annotation indicating a region representing a moving object that is present in the image prior to the first extracting process.

6. The image processing method according to claim 5, further comprising:

causing crowdsourcing workers to attach, to each of the first extracted images selected through the first extracting process, an annotation indicating a region representing a moving object that is present in the first extracted image prior to the second extracting process.

7. The image processing method according to claim 4, wherein the second region is a hazard region involving a risk of collision with the moving object when the vehicle is in motion, and

wherein the image processing method further comprises:

adding, to the second annotation attached in the attaching, a hazard level based on the braking force or acceleration at a time point at which the braking force or acceleration of the vehicle is larger than the threshold.

8. The image processing method according to claim 1,

wherein the moving object is a person, and

wherein the second region has dimensions equal to dimensions of the first region.

9. The image processing method according to claim 1,

wherein the object is a vehicle that is stationary,

wherein the moving object is a door of the vehicle, and

wherein the second region has dimensions equal to dimensions of the first region.

10. The image processing method according to claim 1,

wherein the moving object is an object for a child to play with, and

wherein the second region has dimensions equal to dimensions of a region obtained by enlarging the first region in a height direction of the first image corresponding to the first time point.

11. The image processing method according to claim 8,

wherein the second region is a hazard region involving a risk of collision with the moving object when the vehicle is in motion, and

wherein the image processing method further comprises:

adding, to the second annotation attached in the attaching, a hazard level based on an attribute of the moving object.

12. The image processing method according to claim 1, wherein the second region is a hazard region involving a risk of collision with the moving object when the vehicle is in motion, and

wherein the image processing method further comprises:

adding, to the second annotation attached in the attaching, a hazard level that increases as the dimensions of the second region increase.

13. The image processing method according to claim 1, wherein the determining includes

performing a first determining process of determining, in reverse chronological order from the image corresponding to the last time point in the time series, the first image corresponding to a third time point to which the first annotation is not attached from among the plurality of consecutive time-series images acquired in the acquiring, and

performing a second determining process of determining, through image processing, whether the first region exists at a position in each of the images that is obtained by shifting the first region in an image corresponding to the next time point after the third time point of the first image that has been determined in the first determining process in a direction from the first region toward the object in the image in reverse chronological order from the image corresponding to the third time point.

14. An image processing apparatus comprising:

an acquirer that acquires a plurality of consecutive time-series images that have been captured by an onboard camera mounted on a vehicle and that include at least one image to which a first annotation is attached, the first annotation indicating a first region representing a moving object that is present in the vicinity of an object and on a path of the vehicle in the at least one image;

a determiner that determines, for each of the plurality of consecutive time-series images acquired by the acquirer, in reverse chronological order from an image corresponding to the last time point in the time series, whether the first region exists in the image on the basis of whether the first annotation is attached to the image;

a setter that identifies a first image corresponding to a first time point for which it is determined for the first time by the determiner that the first region does not exist from among the plurality of consecutive time-series images, and sets a second region including a part of a region of the object in the identified first image corresponding to the first time point and indicating a situation where the moving object is obstructed by the object before appearing on the path from behind the object, the second region having dimensions based on dimensions of the first region in an image corresponding to the next time point after the first time point in the time series; and

an attacher that attaches a second annotation to the image corresponding to the second time point, the second annotation indicating the second region set by the setter.

15. A non-transitory computer-readable recording medium storing a program thereon, the program causing a computer to execute:

acquiring a plurality of consecutive time-series images that have been captured by an onboard camera mounted on a vehicle and that include at least one image to which a first annotation is attached, the first annotation indicating a first region representing a moving object that is present in the vicinity of an object and on a path of the vehicle in the at least one image;

determining, for each of the plurality of consecutive time-series images acquired in the acquiring, in reverse chronological order from an image corresponding to the last time point in the time series, whether the first region exists in the image on the basis of whether the first annotation is attached to the image;

identifying a first image corresponding to a first time point for which it is determined for the first time in the determining that the first region does not exist from among the plurality of consecutive time-series images, and setting a second region including a part of a region of the object in the identified first image corresponding to the first time point and indicating a situation where the moving object is obstructed by the object before appearing on the path from behind the object, the second region having dimensions based on dimensions of the first region in an image corresponding to the next time point after the first time point in the time series; and

attaching a second annotation to the image corresponding to the second time point, the second annotation indicating the second region set in the identifying and setting.