Method and system to determine distance to an object in an image

• US 10,217,240 B2
• Filed: 01/23/2017
• Issued: 02/26/2019
• Est. Priority Date: 01/23/2017
• Status: Active Grant

• Current Assignee: Autonomous Fusion, Inc.
• Original Assignee: Autonomous Fusion, Inc.
• Inventors: Taylor, Thomas Steven
• Primary Examiner(s): Aynalem, Nathnael
• Application Number:
  US15/412,967
• Publication Number:
  US 20180211410A1
• Time in Patent Office: 764 Days
• CPC Class Codes: B60R 11/04 Mounting of cameras operative during drive; Arrangement of controls thereof relative to the vehicle
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 10/17 using hand-held instruments
• CPC Class Codes: G06V 10/44 Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 10/752 Contour matching
• CPC Class Codes: G06V 20/582 of traffic signs
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: H04M 2250/52 including functional features of a camera constructional features concerning the integration of cameras in portable phones H04M1/0264
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

A controller/application analyzes image data from a camera to determine the distance to an object in an image based on the size of the object in the image and based on a known focal length of a camera that captured the image and based on a known dimension of the actual object. The known dimension of the object may be retrieved from a database that is indexed according to outline shape, color, markings, contour, or other similar features or characteristics. The distance determined from analysis of the image and objects therein may be used to calibrate, or to verify the calibration of, complex distance determining systems that may include LIDAR. Object distance determinations in different image frames, whether to the same or different object, taken a known time apart may be used to determine speed of the camera that took the images, or speed of a vehicle associated with camera.

Claims

1. A method, comprising:

receiving with a computer device first image data representing a first image of a scene captured with a camera;

processing, with a processor of the computer device, the first image data to determine one or more objects in the scene captured by the camera;

determining from the first image data one or more object features associated with a first object of the one or more objects;

retrieving object-defining information associated with the first object from a data store based on the one or more first object features associated with the first object;

calculating the distance from the camera that captured the scene to the first object in the scene based on the first object-defining information;

analyzing clue information and clue objects proximate the first object in the image to provide clues that a second object is proximate the first object; and

wherein the clue information includes current sun angle information and current orientation of the camera relative to the second object, and wherein an angle of a shadow captured in the image cast by the first object onto the second object is a clue object.

2. The method of claim 1 wherein the one or more first object features associated with the first object is an outline shape of the first object.

3. The method of claim 1 wherein the retrieved object-defining information associated with the first object is a dimension associated with the first object.

4. The method of claim 3 wherein the calculating distance from the camera that captured the scene to the first object in the scene based on the object-defining information includes:

determining a number of pixels in the first image that corresponds to the dimension associated with the first object, calculating, based on the number of pixels, a known focal length of the camera, and the retrieved dimension associated with the first object, the distance to the first object from the camera that captured the image.

5. The method of claim 4, further comprising:

capturing a second image of the scene, wherein in the camera is associated with a moving vehicle, and wherein the scene is a scene of a roadway upon which the vehicle is traveling, and wherein the second image is captured in a second frame at a time that is a predetermined time from a time of a first frame that corresponds to the capturing of the first image;

calculating a distance from the camera to first object in the second image; and

determining a speed of the vehicle based on a change of distance between the camera to the first object in the first image and distance between the camera to the first object in the second image and the time between the capturing of the first image and the capturing of the second image.

6. The method of claim 4 further comprising calibrating a complex vision system based on the calculated distance from the camera that captured the scene to the first object in the scene based on the object-defining information associated with the first object.

7. The method of claim 4 further comprising calculating a distance to the second object in the first image based on the calculated distance to the first object.

8. The method of claim 7 wherein the first object and the second object are not similar.

9. The method of claim 7 wherein the first object and the second object are road signs.

10. The method of claim 4 wherein objects proximate the first object in the first image are analyzed to provide clues that the second object is proximate the first object, wherein object-defining information is not associated with the second object in the data store.

11. A computer device comprising:

a processor to;

receive first image data representing a first image of a scene captured with a camera;

process the first image data to determine one or more objects in the scene captured by the camera;

determine from the first image data one or more object features associated with a first object of the one or more objects;

retrieve object-defining information associated with the first object, from a data store in communication with the processor, based on the one or more object features associated with the first object;

calculate the distance from the camera that captured the scene to the first object in the scene based on the object-defining information;

analyze clue information and clue objects proximate the first object in the image to provide clues that a second object is proximate the first object; and

wherein the clue information includes current sun angle information and current orientation of the camera relative to the second object, and wherein an angle of a shadow captured in the image cast by the first object onto the second object is a clue object.

12. The computer device of claim 11 wherein the processor calculates the distance from the camera that captured the scene to the first object in the scene based on the object-defining information by determining a number of pixels in the first image that corresponds to the dimension associated with the first object, and calculating, based on the number of pixels, a known focal length of the camera, and the retrieved object-defining information associated with the first object, the distance to the first object from the camera that captured the image.

13. The computer device of claim 12, wherein the processor is further to:

receive second image data representing a second image of the scene, wherein the camera is associated with a moving vehicle, and wherein the scene is a scene of a roadway upon which the vehicle is traveling, and wherein the second image is captured at a time that is a predetermined time from a time of the capturing of the first image;

calculate a distance from the camera to the first object in the second image; and

determine a speed of the vehicle based on a change of distance between the camera to the first object in the first image and distance between the camera to the first object in the second image and the time between the capturing of the first image and the capturing of the second image.

14. The computer device of claim 12 wherein the processor is further to:

calibrate a complex vision system that determines distance to objects, wherein the processor calibrates the complex vision system at least based on the calculated distance from the camera that captured the scene to the first object in the scene based on the object-defining information associated with the first object.

15. The computer device of claim 12 wherein the processor is further to:

calculate a distance to a third object in the first image based on the calculated distance to the first object, wherein both first and third objects have known dimensions.

16. The computer device of claim 11 wherein the computer device is a smart phone that comprises the camera and the data store.

17. The computer device of claim 11 wherein the computer device is a smart phone that receives image data from a camera, wherein the smart phone does not comprise the camera.

18. A method, comprising:

receiving with a computer device image data representing an image of a scene captured with a camera;

processing, with a processor of the computer device, the image data to determine one or more objects in the scene captured by the camera;

determining from the image data one or more object features associated with a first object of the one or more objects;

retrieving object-defining information associated with the first object from a data store based on the one or more object features associated with the first object;

calculating the distance from the camera that captured the scene to the first object in the scene based on the object-defining information;

analyzing clue information and clue objects proximate the first object in the image to provide clues that a second object is proximate the first object, wherein object-defining information is not associated with the second object in the data store; and

wherein the clue information includes current sun angle information and current orientation of the camera relative to the second object, and wherein an angle of a shadow captured in the image cast by the first object onto the second object is a clue object.

19. The method of claim 18 wherein the clues include the second object blotting an associated clue object in the image at a location that appears in the image of the scene to correspond to the location of the first object.

20. The computer device of claim 18 wherein the computer device is a smart phone that receives image data from a camera, wherein the smart phone does not comprise the camera.