Image processing for vehicle control

• US 8,630,806 B1
• Filed: 10/20/2011
• Issued: 01/14/2014
• Est. Priority Date: 10/20/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Zhu, Jiajun
• Primary Examiner(s): Cheung, Mary
• Assistant Examiner(s): BERNS, MICHAEL ANDREW
• Application Number:
  US13/277,580
• Time in Patent Office: 817 Days
• Field of Search: 701/1, 701/523, 701/408
• US Class Current: 701/523
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2520/16 Pitch
• CPC Class Codes: B60W 2552/15 Road slope, i.e. the inclination of a road segment in the longitudinal direction
• CPC Class Codes: B60W 2554/801 Lateral distance
• CPC Class Codes: B60W 30/16 Control of distance between vehicles, e.g. keeping a distance to preceding vehicle
• CPC Class Codes: G01C 21/26 specially adapted for navigation in a road network
• CPC Class Codes: G01S 11/12 using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 17/931 of land vehicles
• CPC Class Codes: G05D 1/0251 extracting 3D information from a plurality of images taken from different locations, e.g. stereo vision stereoscopic image analysis H04N13/00; depth recovery from images G06T7/593
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G08G 1/165 for passive traffic, e.g. including static obstacles, trees
• CPC Class Codes: G08G 1/166 for active traffic, e.g. moving vehicles, pedestrians, bikes
• CPC Class Codes: G08G 9/02 Anti-collision systems

Abstract

A method is provided for processing an image in which only parts of the image that appear above a point on a horizon line are analyzed to identify an object. In one embodiment, the distance between the object and a vehicle is determined, and at least one of the speed and direction of the vehicle is changed when it is determined that the distance is less than the range of a sensor. The method for processing images is not limited to vehicular applications only and it may be used in all applications where computer vision is used to identify objects in an image.

Claims

1. A vehicle comprising:

a sensor configured to detect objects in the vehicle'"'"'s surroundings, the sensor being associated with a range;

a camera configured to capture a first image of a scene of at least a part of the vehicle'"'"'s surroundings and a second image overlapping at least a part of the scene of the first image;

a memory; and

a processor coupled to the memory, the sensor, and the camera, the processor being configured to;

determine, based on the vehicle'"'"'s pitch, a point P on a visual horizon in the first image;

process the first image and the second image to determine distance between the vehicle and an object that is depicted in the first image above and beyond the point P; and

change at least one of a speed of the vehicle and a heading of the vehicle when it is determined that the distance between the vehicle and the object is less than the range of the sensor.

2. The vehicle of claim 1, wherein

the camera is a stereo camera.

3. The vehicle of claim 1, wherein the processor changes at least one of the speed and heading of the vehicle based on at least one of:

(i) a grade of a surface on which the vehicle is standing or driving exceeding a threshold, and

(ii) pitch of the vehicle exceeding a threshold.

4. The vehicle of claim 1, wherein the changing of the heading of the vehicle comprises making a turn onto another roadway.

5. The vehicle of claim 1, wherein the changing of the heading of the vehicle comprises changing lanes.

6. The vehicle of claim 1, wherein the changing of the heading of the vehicle comprises selecting a new travel route for the vehicle.

7. A system comprising:

a camera configured to capture a first image and a second image;

a sensor configured to sense the pitch of the camera;

a processor coupled to the camera and to the sensor, the processor being configured to;

select an object that is depicted in the first image based on the object being depicted above and beyond a point P on a visual horizon in the first image, the point P being determined by the processor based on signal from the sensor that senses the pitch of the camera; and

process the first image and the second image to determine a distance between the camera and the reference object;

receive input from a second sensor having a range;

determine a geographic location of the vehicle based on the received input;

change at least one of a speed of a vehicle and a heading of the vehicle when it is determined that the distance between a vehicle and the object is less than the range of the second sensor.

8. The apparatus of claim 7 wherein

the camera is a stereo camera.

9. The apparatus of claim 7 wherein the processor is further configured to select the object further based on the reference object being represented in a database for storing object representations.

10. The apparatus of claim 7 wherein processor is further configured to select the object further based on proximity to the point P.

11. The apparatus of claim 7 wherein the processor is further configured to:

determine a geographic location of the reference object by extracting a template of the object from the first image and comparing the template against a database of object representations; and

determine a geographic location of the apparatus based on the geographic location of the reference object.

12. A method comprising:

controlling operation of a vehicle with a first control strategy;

capturing, by a camera, a first image of a scene at least part of the vehicle'"'"'s surroundings and a second image overlapping at least a part of the scene of the first image; and

processing the first image and the second image to determine a distance between the vehicle and an object appearing above and beyond a point P on a visual horizon in the first image, wherein;

(i) the point P is determined based on the pitch of the vehicle, and

(ii) the object is selected based on the object being depicted, at least in part, above the point P in the image.

13. The method of claim 12 further comprising switching to controlling the vehicle with a second control strategy based on the distance between the vehicle and the object.

14. The method of claim 13, wherein controlling the operation of the vehicle with the first control strategy includes maneuvering the vehicle in a first direction, and controlling the operation of the vehicle with the second control strategy includes maneuvering the vehicle in a second direction different from the first direction.

15. The method of claim 13 wherein the switching is performed when the distance is less than a range associated with a sensor.

16. The method of claim 12 comprising

defining a ray R extending from the camera;

and

projecting the ray R against a plane of the first image;

wherein the point P is a point where the projected ray R intercepts the plane of the first image.

17. The method of claim 12, wherein the camera is a stereo camera.