Object detection using location data and scale space representations of image data

• US 10,133,947 B2
• Filed: 01/16/2015
• Issued: 11/20/2018
• Est. Priority Date: 01/16/2015
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Yang, Ruiduo, Bi, Ning, Yang, Sichao, Wu, Xinzhou, Guo, Feng, Ren, Jianfeng
• Primary Examiner(s): Bitar, Nancy
• Application Number:
  US14/598,892
• Publication Number:
  US 20160210525A1
• Time in Patent Office: 1,404 Days
• Field of Search: None
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 10/32 Normalisation of the pattern dimensions
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 30/2504 Coarse or fine approaches, e.g. resolution of ambiguities or multiscale approaches
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition

Abstract

An apparatus includes an object detector configured to receive image data of a scene viewed from the apparatus and including an object. The image data is associated with multiple scale space representations of the scene. The object detector is configured to detect the object responsive to location data and a first scale space representation of the multiple scale space representations.

Claims

1. An apparatus comprising:

a memory;

a processor configured to;

receive location data associated with an object;

receive image data of a scene from an image sensor, the scene including the object, the image data associated with multiple scale space representations of the scene, the multiple scale space representations including a first scale space representation and a second scale space representation, wherein the first and second scale space representations are associated with different distances in a same direction from an image sensor and different resolution images of the same field of view of the scene;

identify a search area of a sub-region of one or more of the multiple scale space representations of the scene based on an overlap of a region indicated by the location data and one or more areas associated with one or more of the multiple scale space representations of the scene; and

detect the object in the search area of the one or more of the multiple scale space representations.

2. The apparatus of claim 1, wherein the search area corresponds to the overlap of a first area associated with the first scale space representation and the region.

3. The apparatus of claim 1, wherein the processor is further configured to generate a mask of the search area, the mask comprising a plurality of regions, and wherein each of the plurality of regions corresponds to one or more pixels associated with the image data and to a value indicative of a probability of detecting the object in the corresponding region.

4. The apparatus of claim 1, wherein the processor is configured to determine whether a first area associated with the second scale space representation overlaps the region.

5. The apparatus of claim 4, wherein the processor is further configured to determine whether a second area associated with the first scale space representation overlaps the region prior to detection of the object.

6. The apparatus of claim 4, wherein the first scale space representation corresponds to a first resolution image of the scene, the second scale space representation corresponds to a second resolution image of the scene, and wherein the second resolution image is a lower resolution image than the first resolution image.

7. The apparatus of claim 1, further comprising a location sensor, wherein the processor is configured to receive the location data from the location sensor.

8. The apparatus of claim 7, wherein the location sensor comprises a radar device, a light detection and ranging (lidar) device, a global positioning system (GPS) device, an ultrasound device, a dedicated short-range communication device, or any combination thereof.

9. The apparatus of claim 1, wherein the processor is configured to receive the location data from a location sensor that is separate from the apparatus.

10. The apparatus of claim 1, wherein the location data corresponds to one or more areas of a three-dimensional (3D) map indicative of where the object may be present, and further comprising a display coupled to the processor and configured to present the 3D map including an indication associated with detection of the object.

11. The apparatus of claim 10, further comprising a processor that includes the processor, wherein the location data is also associated with a location of the object provided by a location sensor.

12. The apparatus of claim 1, further comprising the image sensor, wherein the image sensor comprises a camera.

13. A method of detecting an object, the method comprising:

receiving from an image sensor, at a processor, image data of a scene including an object, the image data associated with multiple scale space representations of the scene, the multiple scale space representations including a first scale space representation and a second scale space representation, wherein the first and second scale space representations are associated with different distances in a same direction from an image sensor and different resolution images of the same field of view of the scene;

receiving, at the processor, location data associated with the object;

identifying, at the processor, a search area of a sub-region of one or more multiple scale space representations of the scene based on an overlaps of a region indicated by the location data and one or more areas associated with the one or more of the multiple scale space representations of the scene; and

performing, at the processor, object detection in the search area of the one or more of the scale space representations.

14. The method of claim 13, wherein the search area corresponds to the overlap of a first area associated with the first scale space representation and the region.

15. The method of claim 13, further comprising determining whether each of a first area associated with the first scale space representation and a second area associated with the second scale space representation of the multiple scale space representations overlaps the region, wherein the first scale space representation is associated with a first distance from the image sensor in a direction, the second scale space representation is associated with a second distance from the image sensor in the direction, and wherein the first distance is different from the second distance.

16. The method of claim 15, further comprising avoiding performance of object detection on the second scale space representation responsive to determining the second area does not overlap the region.

17. The method of claim 15, further comprising performing object detection on the second scale space representation responsive to determining the second area overlaps the region, wherein the second scale space representation is a lower resolution representation than the first scale space representation.

18. The method of claim 13, wherein the image data includes the first scale space representation of the multiple scale space representations, and further comprising generating the second scale space representation of the multiple scale space representations based on the first scale space representation.

19. The method of claim 13, wherein the first scale space representation corresponds to a 1080p resolution, wherein the second scale space representation comprises a downsampled version of the first scale space representation, the second scale space representation generated based on a scaling factor, and wherein the scaling factor is five.

20. The method of claim 13, wherein the image data corresponds to the first scale space representation, and further comprising receiving at the processor, from a second image sensor, second image data of the scene, the second image data corresponding to the second scale space representation.

21. The method of claim 13, further comprising generating a mask of the search area, the mask comprising a plurality of regions, wherein each of the plurality of regions corresponds to a value indicative of a probability of detecting the object in the corresponding region, and wherein performing object detection in the search area is based at least in part on one or more values corresponding to the plurality of regions.

22. The method of claim 13, wherein the second scale space representation comprises a downsampled version of the first scale space representation.

23. The method of claim 22, wherein the downsampled version is generated based on a scaling factor, and wherein the scaling factor is two.

24. The method of claim 13, wherein the location data corresponds to one or more areas of a three-dimensional (3D) map, and wherein the location data is also associated with a location of the object provided by a location sensor.

25. An apparatus comprising:

means for receiving image data of a scene including an object, the image data associated with multiple scale space representations of the scene, the multiple scale space representations including a first scale space representation and a second scale space representation, wherein the first and second scale space representations are associated with different distances in a same direction from an image sensor and different resolution images of the same field of view of the scene; and

means for processing, wherein the means for processing is configured to;

identify a search area of a sub-region of one or more multiple scale space representations of the scene based on an overlap of a region indicated by location data associated with the object and one or more areas associated with the one or more of the multiple scale space representations of the scene; and

perform object detection in the search area of the one or more of the multiple scale space representations of the scene.

26. The apparatus of claim 25, further comprising:

means for generating the location data; and

means for generating the image data.

27. The apparatus of claim 25, wherein the means for receiving and the means for processing are incorporated into a vehicle.

28. A computer-readable storage device storing instructions to detect an object, the instructions executable by a processor to cause the processor to:

receive, from an image sensor, image data of a scene including an object, the image data associated with multiple scale space representations of the scene, the multiple scale space representations including a first scale space representation and a second scale space representation, wherein the first and second scale space representations are associated with different distances in a same direction from an image sensor and different resolution images of the same field of view of the scene;

receive location data associated with the object;

identify a search area of a sub-region of one or more multiple scale space representations of the scene based on an overlap of a region indicated by the location data and one or more areas associated with the one or more of the multiple scale space representations of the scene; and

perform object detection in the search area of the one or more of the multiple scale space representations.

29. The computer-readable storage device of claim 28, wherein the first scale space representation is selected based on a determination that a first area associated with the first scale space representation overlaps the region, and wherein the instructions are further executable by the processor to cause the processor to:

determine whether a second area associated with the second scale space representation of the multiple scale space representations overlaps the region;

receive second location data associated with a second object, the second object included in the scene;

determine whether a third area associated with a third scale space representation of the multiple scale space representations overlaps a second region associated with the second object and indicated by the second location data, the third scale space representation comprising a downsampled version of the first scale space representation; and

in response to a determination that the third area overlaps the second region;

identify a second search area of the third scale space representation based on the overlap of the third area and the second region; and

perform object detection in the second search area of the third scale space representation.