Method and apparatus of detecting object of interest

• US 10,599,930 B2
• Filed: 05/30/2018
• Issued: 03/24/2020
• Est. Priority Date: 08/04/2017
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Lee, Wonju, Lee, Jaewoo, Ji, Dae Hyun
• Primary Examiner(s): Le, Michael
• Application Number:
  US15/992,234
• Publication Number:
  US 20190042860A1
• Time in Patent Office: 664 Days
• CPC Class Codes: G06F 18/213 Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/30256 Lane; Road marking
• CPC Class Codes: G06T 2210/12 Bounding box
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/584 of vehicle lights or traffic lights
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road

Abstract

Disclosed is a method and apparatus of detecting an object of interest, where the apparatus acquires an input image, sets a region of interest (ROI) in the input image, and detects the object of interest from a restoration image, having a resolution greater than a resolution of the input image, corresponding to the ROI.

Claims

1. A method of detecting an object of interest, the method comprising:

acquiring an input image, the input image including at least one external image of a front view or each side view of the vehicle while driving on a road;

setting a region of interest (ROI) in the input image;

generating a restoration image corresponding to the ROI, a resolution of the restoration image being greater than a resolution of the input image; and

detecting the object of interest from the restoration image, wherein the detecting comprises;

setting the object of interest in the input image corresponding to a current point in time t or a next point in time t+1 based on a detection result of the input image corresponding to a previous point in time t−

1, and

redetecting an object based on a time-varying method of only detecting a region of the object of interest by performing up-sampling.

2. The method of claim 1, wherein the setting of the ROI comprises:

detecting at least one object comprising a lane marking in the input image; and

setting the ROI based on the at least one object.

3. The method of claim 1, wherein the setting of the ROI comprises:

detecting a road vanishing point where a lane marking converges based on a segmentation of the lane marking in the input image; and

setting a region corresponding to the road vanishing point as the ROI.

4. The method of claim 3, wherein the region corresponding to the road vanishing point comprises a region within a distance from the road vanishing point, and the distance is an inverse function of a number of lanes on a road.

5. The method of claim 1, wherein the setting of the ROI comprises:

calculating a reliability score of at least one object in the input image; and

setting the ROI based on the reliability score.

6. The method of claim 5, wherein the calculating of the reliability score comprises calculating the reliability score of the at least one object in the input image by detecting a bounding box based on deep learning or image processing.

7. The method of claim 6, wherein the setting of the ROI comprises setting a region of the bounding box as the ROI based on a comparison of the reliability score to a threshold value.

8. The method of claim 7, wherein the threshold value varies depending on a type of the object.

9. The method of claim 1, wherein the generating of the restoration image comprises:

performing super resolution to increase pixel information by up-sampling on an image corresponding to the ROI; and

generating the restoration image based on the increased pixel information.

10. The method of claim 9, wherein the detecting of the object of interest comprises:

detecting the object of interest from the ROI having the increased pixel information.

11. The method of claim 1, wherein the generating of the restoration image comprises:

capturing images associated with the ROI; and

performing up-scaling on at least a portion of the images to generate a super resolution image associated with the ROI.

12. The method of claim 1, further comprising:

estimating a distance from the object based on a pixel difference of a restoration image corresponding to a stereo image,

wherein the input image is the stereo image, and the generating of the restoration image comprises generating the restoration image corresponding to the stereo image.

13. The method of claim 1, wherein the ROI comprises at least one of a portion of a region of the input image or at least one object included in the input image.

14. The method of claim 13, wherein the object of interest comprises at least one of a road vanishing point, a road marking, a pedestrian, a vehicle, a traffic light, a lane marking, a traffic sign, a human, an animal, a plant, or a building.

15. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 1.

16. An apparatus for detecting an object of interest, the apparatus comprising:

a sensor configured to acquire an input image, the input image including at least one external image of a front view or each side view of the vehicle while driving on a road; and

a processor configured to set a region of interest (ROI) in the input image, to generate a restoration image corresponding to the ROI, and to detect the object of interest from the restoration image by;

setting the object of interest in the input image corresponding to a current point in time t or a next point in time t+1 based on a detection result of the input image corresponding to a previous point in time t−

1, and

redetecting an object based on a time-varying method of only detecting a region of the object of interest by performing up-sampling,

wherein a resolution of the restoration image is greater than a resolution of the input image.

17. The apparatus of claim 16, wherein the processor is further configured to detect at least one object comprising a lane marking in the input image, and to set the ROI based on the at least one object.

18. The apparatus of claim 16, wherein the processor is further configured to detect a road vanishing point where a lane marking converges based on a segmentation of the lane marking in the input image, and to set a region corresponding to the road vanishing point as the ROI.

19. The apparatus of claim 16, wherein the processor is further configured to calculate a reliability score of at least one object in the input image, and to set the ROI based on the reliability score.

20. The apparatus of claim 19, wherein the processor is further configured to calculate the reliability score of the at least one object by detecting a bounding box based on deep learning or image processing, and to set a region of the bounding box as the ROI based on a comparison of the reliability score to a threshold value.

21. The apparatus of claim 16, wherein the processor is further configured to perform super resolution to increase pixel information by up-sampling an image corresponding to the ROI, and to generate the restoration image based on the increased pixel information.

22. An apparatus for detecting an object of interest comprising:

a sensor configured to acquire an input image, the input image including at least one external image of a front view or each side view of the vehicle while driving on a road;

a head-up display (HUD);

a memory configured to store map information and instructions; and

a processor configured to execute the instructions to;

identify an object comprising a lane marking in the input image,

set a region of interest (ROI) based on the object,

generate a restoration image corresponding to the ROI,

detect the object of interest from the restoration image having a higher resolution than a resolution of the input image, and

output the restoration image and the object of interest through the HUD, wherein the processor is configured to detect the object of interest by;

setting the object of interest in the input image corresponding to a current point in time t or a next point in time t+1 based on a detection result of the input image corresponding to a previous point in time t−

1, and

redetecting an object based on a time-varying method of only detecting a region of the object of interest by performing up-sampling.

23. The apparatus of claim 22, wherein the memory is further configured to store information on the object of interest detected from the restoration image, and to provide information from object of interest detected from previous restoration image.