IMAGE PROCESSING DEVICE, IMAGE PROCESSING SYSTEM, AND IMAGE PROCESSING METHOD

• US 20170069090A1
• Filed: 08/30/2016
• Published: 03/09/2017
• Est. Priority Date: 09/07/2015
• Status: Active Grant

• Current Assignee: Kabushiki Kaisha Toshiba (Toshiba Corporation)
• Original Assignee: Kabushiki Kaisha Toshiba (Toshiba Corporation)
• Inventors: OKUMURA, Mayu, TAKATANI, Tsuyoshi, NISHIYAMA, Manabu, TASAKI, Tsuyoshi
• Granted Patent: US 10,318,782 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06V 20/54 of traffic, e.g. cars on the road, trains or boats
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads

Abstract

An image processing device obtains first images capturing a street, on which a target vehicle is running, from a plurality of directions, and position information indicating positions at which the first images are taken, and associates a background object in a second image with a background object in a third image. The second image is an image which the target object is detected from and the third image is an image which is taken by an imaging device having a common imaging range with the second image among imaging devices mounted on the target vehicle. An output image is generated to include the position information indicating a position at which the second image is taken, the image of the detection rectangle which is clipped from the second image and includes the target object and the background object, and the first image of surroundings of the target vehicle.

Claims

1. An image processing device comprising:

hardware circuitry configured to;

obtain first images capturing a street, on which a target vehicle is running, from a plurality of directions, and position information indicating imaging positions at which the first images are taken;

detect an target object from the first images;

associate a background object included in the second image with a background object included in the third image, the first images including the second image and the third image, the second image being an image which the target object is detected from, the third image being an image which is taken by an imaging device having a common imaging range with the second image among imaging devices mounted on the target vehicle;

clip, from the second image, an image of a detection rectangle which includes the target object and the background object; and

generate an output image including the position information indicating a position at which the second image is taken, the image of the detection rectangle, and the first image of surroundings of the target vehicle.

2. The device according to claim 1, wherein the detection rectangle is certain percentage of a smallest rectangle including the target object in entirely and the associated background object.

3. The image processing device according to claim 1, wherein the hardware circuitry divides a certain area around the target object in the second image as well as divide the third image using at least either color information or edge information, wherein the hardware circuitry associate, using at least either color information, or texture information, or area size, areas formed by division of the second image with areas formed by division of the third image, and set such an area among mutually-associated areas which has highest degree of similarity as an background object area.

4. The image processing device according to claim 1, wherein the hardware circuitry detects an image of the background object from a certain area around the detection for target in the second image, wherein

the hardware circuitry searches the third image for the image of the background object as detected by the object detecting unit, and set such an area in the third image which has highest degree of similarity with the image of the background object as an background object area.

5. The image processing device according to claim 1, wherein the hardware circuitry detects an image of the background object and attribute information of the background object from a certain area around the target object in the second image as well as from the third image, wherein the hardware circuitry associates the images of the background object, which are detected from the second image and the third image, among the second image and the third image using at least one of color information, texture information, and the attribute information, and set a detection rectangle of the associated image of the background object as a background object area.

6. The image processing device according to claim 1, wherein the hardware circuitry obtains the first images obtained by an imaging device mounted on vicinity of the street on which the target vehicle is running and obtained by an imaging device mounted on mobile objects including the running target vehicle, and obtain the position information indicating imaging positions of the first images.

7. The image processing device according to claim 1, wherein, when the hardware circuitry associates a single of the background object captured within a certain area around the target object and when height and width of smallest rectangle including an image of the background object associated is smaller than a certain percentage of height and width of the target object, the hardware circuitry clips a rectangle including the smallest rectangle in entirety and the target object in entirety as the detection rectangle from the second image.

8. The image processing device according to claim 1, wherein, when the hardware circuitry associates a single of the background object captured within a certain area around the target object and when height and width of smallest rectangle including an image of the background object associated is greater than a certain percentage of height and width of the target object, the hardware circuitry determines whether or not the image of the background object associated represents an image of an object present on road surface, and

when the image of the background object associated is determined to represent an image of an object present on road surface, the hardware circuitry clips a rectangle including

an area formed by expansion in horizontal direction of smallest rectangle, which includes the image of the background object, in a certain percentage from one side of the smallest rectangle that is close in horizontal direction to in-image position of the target object toward direction from the smallest rectangle to the target object, and

the target object in entirety,

as the detection rectangle from the second image.

9. The image processing device according to claim 1, wherein, when the hardware circuitry associate a single of the background object captured within a certain area around the target object and when height and width of smallest rectangle including an image of the background object associated is greater than a certain percentage of height and width of the target object, the hardware circuitry determines whether or not the image of the background object associated represents an image of an object present on road surface, and

when the image of the background object associated is determined not to represent an image of an object present on road surface, the hardware circuitry clips a rectangle including

an area formed by expansion in vertical direction of smallest rectangle, which includes the image of the background object, in a certain percentage from one side of the smallest rectangle that is close in vertical direction to in-image position of the target object toward direction from the smallest rectangle to the target object, and

the target object in entirety,

as the detection rectangle from the second image.

10. The image processing device according to claim 7, wherein, when the hardware circuitry associates a plurality of the background objects captured within a certain area around the target object, the hardware circuitry identifies, as the single of the background object, an image of background object closest to the target object from among the associated images of a plurality of the background objects.

11. The image processing device according to claim 8, wherein, when the hardware circuitry associates a plurality of the background objects captured within a certain area around the target object, the hardware circuitry identifies, as the single of the background object, an image of background object closest to the target object from among the associated images of a plurality of the background objects.

12. The image processing device according to claim 9, wherein, when the hardware circuitry associates a plurality of the background objects captured within a certain area around the target object, the hardware circuitry identifies, as the single of the background object, an image of background object closest to the target object from among the associated images of a plurality of the background objects.

13. The image processing device according to claim 1, wherein the hardware circuitry performs wireless communication, wherein

the hardware circuitry obtains the first images and the position information using the wireless communication performed.

14. The image processing device according to claim 13, wherein, the hardware circuitry,

obtains the first images formed by imaging performed by an imaging device which is mounted on the target vehicle and obtain the position information corresponding to the first images,

obtains the first images formed by imaging performed by an imaging device which is mounted on surrounding environment of the target vehicle and obtain the position information corresponding to the first images using the wireless communication of to-fixed-object communication type performed, and

obtains the first images formed by imaging by an imaging device which is mounted on other vehicle different from the target vehicle and obtain the position information corresponding to the first images using the wireless communication of inter-mobile-object communication type performed.

15. The image processing device according to claim 13, wherein the hardware circuitry,

performs the wireless communication with a server device,

obtains the first images formed by imaging performed by an imaging device which is mounted on the target vehicle and obtain the position information corresponding to the first images, and

obtains the first images and the position information corresponding to the first images from the server device using the wireless communication performed.

16. An image processing system comprising:

the image processing device according to claim 13;

a first communication device; and

a second communication device, wherein

the first communication device sends the first images taken by an imaging device installed around a street and sends the position information corresponding to the first images,

the second communication device sends the first images taken by an imaging device mounted on other vehicle different from the target vehicle and sends the position information corresponding to the first images, and

the hardware circuitry is configured to,

obtain the first images formed by imaging performed an imaging device mounted on the target vehicle and obtain the position information corresponding to the first images, and

obtain, via the wireless communication, the first images and the position information sent by at least one of the first communication device and the second communication device.

17. An image processing system comprising:

the image processing device according to claim 13;

a first communication device;

a second communication device; and

a server device, wherein

the first communication device sends the first images taken by an imaging device installed around a street and sends the position information corresponding to the first images,

the second communication device sends the first images taken by an imaging device mounted on other vehicle different from the target vehicle and sends the position information corresponding to the first images,

the server device obtains the first images and the position information sent by the first communication device as well as the second communication device, and

the hardware circuitry is configured to,

obtain the first images formed by imaging performed an imaging device mounted on the target vehicle and obtain the position information corresponding to the first images, and

obtain, from the server device via the wireless communication, the first images and the position information sent by at least one of the first communication device and the second communication device.

18. An image processing method comprising:

obtaining first images capturing a street, on which a target vehicle is running, from a plurality of directions, and position information indicating imaging positions at which the first images are taken;

detecting an target object from the first images;

associating an image of a background object included in the second image with an image of a background object included in the third image, the first images including the second image and the third image, the second image being an image which the target object is detected from, the third image being an image which is taken by an imaging device having a common imaging range with the second image among imaging devices installed on the target vehicle;

clipping, from the second image, a detection rectangle which includes the target object and the image of the background object; and

generating an output image including the position information indicating a position at which the second image is taken, the image of the detection rectangle, and the first image around the target vehicle.