Construction zone sign detection

• US 9,195,914 B2
• Filed: 09/05/2012
• Issued: 11/24/2015
• Est. Priority Date: 09/05/2012
• Status: Expired due to Fees

• Current Assignee: Waymo LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Fairfield, Nathaniel, Ferguson, David Ian, Ogale, Abhijit, Wang, Matthew, Yee, Yangli Hector
• Primary Examiner(s): Beaulieu, Yonel
• Assistant Examiner(s): KLEINMAN, LAIL A
• Application Number:
  US13/603,615
• Publication Number:
  US 20140063232A1
• Time in Patent Office: 1,175 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: B60T 1/10 by utilising wheel movement for accumulating energy, e.g. driving air compressors
• CPC Class Codes: B60T 2201/08 Lane monitoring; Lane Keeping Systems
• CPC Class Codes: B60T 7/12 for automatic initiation; for initiation not subject to will of driver or passenger limiting speed of vehicles other than rail vehicles B60K31/00
• CPC Class Codes: B60T 7/18 operated by wayside apparatus
• CPC Class Codes: B60T 7/22 initiated by contact of vehicle, e.g. bumper, with an external object, e.g. another vehicle , or by means of contactless obstacle detectors mounted on the vehicle
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2552/00 Input parameters relating to infrastructure
• CPC Class Codes: B60W 2552/10 Number of lanes
• CPC Class Codes: B60W 2552/50 Barriers
• CPC Class Codes: B60W 2552/53 Road markings, e.g. lane marker or crosswalk
• CPC Class Codes: B60W 2554/408 Traffic behavior, e.g. swarm
• CPC Class Codes: B60W 2555/60 Traffic rules, e.g. speed limits or right of way
• CPC Class Codes: B60W 30/10 Path keeping cruise control for automatically following a preceding vehicle B60W30/165
• CPC Class Codes: B60W 30/143 Speed control B60W30/16 takes precedence
• CPC Class Codes: B60W 40/06 Road conditions
• CPC Class Codes: B60W 60/00184 related to infrastructure
• CPC Class Codes: B62D 15/025 Active steering aids, e.g. helping the driver by actively influencing the steering system after environment evaluation B62D1/28 takes precedence
• CPC Class Codes: B62D 15/0255 Automatic changing of lane, e.g. for passing another vehicle
• CPC Class Codes: B62D 15/0265 Automatic obstacle avoidance by steering
• CPC Class Codes: B62D 6/00 Arrangements for automatically controlling steering depending on driving conditions sensed and responded to, e.g. control circuits means for initiating a change in direction B62D1/00; steering valves B62D5/06; combined with means for inclining the vehicle body or wheels on bends B62D9/00
• CPC Class Codes: G01C 22/00 : Measuring distance traversed on the ground by vehicles, persons, animals or other moving solid bodies, e.g. using odometers, using pedometers
• CPC Class Codes: G01S 13/865 : Combination of radar systems with lidar systems
• CPC Class Codes: G01S 13/867 : Combination of radar systems with cameras
• CPC Class Codes: G01S 13/931 : of land vehicles
• CPC Class Codes: G01S 17/89 : for mapping or imaging
• CPC Class Codes: G01S 17/931 : of land vehicles
• CPC Class Codes: G01S 2013/9316 : combined with communication equipment with other vehicles or with base stations
• CPC Class Codes: G01S 2013/932 : using own vehicle data, e.g. ground speed, steering wheel direction
• CPC Class Codes: G01S 7/4802 : using analysis of echo signal for target characterisation; Target signature; Target cross-section
• CPC Class Codes: G01S 7/4808 : Evaluating distance, position or velocity data
• CPC Class Codes: G05D 1/024 : in combination with a laser lasers per se H01S
• CPC Class Codes: G05D 1/0246 : using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/0257 : using a radar radar systems designed for anti-collision purposes between land vehicles or between land vehicle and fixed obstacles G01S13/931
• CPC Class Codes: G05D 1/027 : comprising intertial navigation means, e.g. azimuth detector inertial navigation G01C21/16; inertial navigation combined with non-inertial navigation instruments G01C21/165
• CPC Class Codes: G05D 1/0278 : using satellite positioning signals, e.g. GPS
• CPC Class Codes: G06V 20/58 : Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/582 : of traffic signs
• CPC Class Codes: G06V 20/588 : Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
• CPC Class Codes: H04N 7/18 : Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

Methods and systems for detection of a construction zone sign are described. A computing device, configured to control the vehicle, may be configured to receive, from an image-capture device coupled to the computing device, images of a vicinity of the road on which the vehicle is travelling. Also, the computing device may be configured to determine image portions in the images that may depict sides of the road at a predetermined height range. Further, the computing device may be configured to detect a construction zone sign in the image portions, and determine a type of the construction zone sign. Accordingly, the computing device may be configured to modify a control strategy associated with a driving behavior of the vehicle; and control the vehicle based on the modified control strategy.

Claims

1. A method, comprising:

receiving, at a computing device configured to control a vehicle, from an image-capture device coupled to the vehicle, an image of a vicinity of a road on which the vehicle is travelling;

selecting, from the image, using the computing device, an image portion corresponding to a side of the road at a predetermined height;

detecting, using the computing device, a construction zone sign in the image portion;

determining, using the computing device, a type of the construction zone sign in the image portion;

detecting one or more construction zone cones or barrels on the road;

determining a number and locations of the one or more construction zone cones or barrels;

determining, based on the number and locations of the one or more construction zone cones or barrels, a severity of road changes;

modifying, using the computing device, a control strategy associated with a driving behavior of the vehicle based on (i) the type of the construction zone sign, and (ii) the severity of road changes; and

controlling, using the computing device, the vehicle based on the modified control strategy.

2. The method of claim 1, wherein the vehicle is in an autonomous operation mode.

3. The method of claim 1, wherein detecting the construction zone sign comprises:

detecting a candidate construction zone sign in the image portion; and

determining that the candidate construction zone sign relates to a construction zone, based on one or more of shape, color, and pattern of the candidate construction zone sign.

4. The method of claim 1, wherein detecting the construction zone sign comprises:

detecting a candidate construction zone sign in the image portion;

determining a first likelihood that the candidate construction zone sign relates to a construction zone, based on one or more of shape, color, and pattern of the candidate construction zone sign;

receiving, at the computing device, from a light detection and ranging (LIDAR) sensor coupled to the computing device, LIDAR-based information comprising a three-dimensional (3D) point cloud corresponding to the image portion corresponding to the candidate construction zone sign, wherein the 3D point cloud comprises a set of points based on light emitted from the LIDAR and reflected from a surface of the candidate construction zone sign; and

determining a second likelihood that the candidate construction zone sign relates to the construction zone, based on the LIDAR-based information, wherein detecting the construction zone sign is based on the first likelihood and the second likelihood.

5. The method of claim 1, wherein detecting the construction zone sign comprises:

detecting a candidate construction zone sign in the image portion;

determining a first likelihood that the candidate construction zone sign relates to a construction zone, based on one or more of shape, color, and pattern of the candidate construction zone sign;

receiving, at the computing device, from a radio detection and ranging (RADAR) sensor coupled to the computing device, RADAR-based information relating to location and one or more characteristics of the candidate construction zone sign; and

determining a second likelihood that the candidate construction zone sign relates to the construction zone, based on the RADAR-based information, wherein detecting the construction zone sign is based on the first likelihood and the second likelihood.

6. The method of claim 5, wherein the RADAR-based information includes dimensional characteristics of the candidate construction zone sign and indicates that the candidate construction zone sign is stationary.

7. The method of claim 1, wherein determining the type of the construction zone sign comprises:

determining a shape of the construction zone sign in the image portion; and

matching the shape to one or more of shapes of typical construction zone signs.

8. The method of claim 1, wherein controlling the vehicle based on the modified control strategy comprises one or more of:

(i) utilizing sensor information received from on-board or off-board sensors in making a navigation decision rather than preexisting map information, (ii) utilizing the sensor information to estimate lane boundaries rather than the preexisting map information, (iii) determining locations of construction zone markers rather than lane markers on the road to estimate and follow the lane boundaries, (iv) activating one or more sensors for detection of construction workers and making the navigation decision based on the detection, (v) following another vehicle, (vi) maintaining a predetermined safe distance with other vehicles, (vii) turning-on lights, (viii) reducing a speed of the vehicle, and (ix) stopping the vehicle.

9. A non-transitory computer readable medium having stored thereon instructions executable by a computing device of a vehicle to cause the computing device to perform functions comprising:

receiving from an image-capture device coupled to the vehicle, an image of a vicinity of a road on which the vehicle is travelling;

selecting, from the image, an image portion corresponding to a side of the road at a predetermined height;

detecting a construction zone sign in the image portion;

determining a type of the construction zone sign in the image portion;

detecting one or more construction zone cones or barrels on the road;

determining a number and locations of the one or more construction zone cones or barrels;

determining, based on the number and locations of the one or more construction zone cones or barrels, a severity of road changes;

modifying a control strategy associated with a driving behavior of the vehicle based on (i) the type of the construction zone sign, and (ii) the severity of road changes; and

controlling the vehicle based on the modified control strategy.

10. The non-transitory computer readable medium of claim 9, wherein the vehicle is in an autonomous operation mode.

11. The non-transitory computer readable medium of claim 9, wherein the function of detecting the construction zone sign comprises:

detecting a candidate construction zone sign in the image portion; and

determining that the candidate construction zone sign relates to a construction zone, based on one or more of shape, color, and pattern of the candidate construction zone sign.

12. The non-transitory computer readable medium of claim 9, wherein the function of detecting the construction zone sign comprises:

detecting a candidate construction zone sign in the image portion;

determining a first likelihood that the candidate construction zone sign relates to a construction zone, based on one or more of shape, color, and pattern of the candidate construction zone sign;

receiving, from a light detection and ranging (LIDAR) sensor coupled to the computing device, LIDAR-based information comprising a three-dimensional (3D) point cloud corresponding to the image portion corresponding to the candidate construction zone sign, wherein the 3D point cloud comprises a set of points based on light emitted from the LIDAR and reflected from a surface of the candidate construction zone sign; and

determining a second likelihood that the candidate construction zone sign relates to the construction zone, based on the LIDAR-based information, wherein the function of detecting the construction zone sign is based on the first likelihood and the second likelihood.

13. The non-transitory computer readable medium of claim 9, wherein the function of detecting the construction zone sign comprises:

detecting a candidate construction zone sign in the image portion;

determining a first likelihood that the candidate construction zone sign relates to a construction zone, based on one or more of shape, color, and pattern of the candidate construction zone sign;

receiving from a radio detection and ranging (RADAR) sensor coupled to the computing device, RADAR-based information relating to location and one or more characteristics of the candidate construction zone sign; and

determining a second likelihood that the candidate construction zone sign relates to the construction zone, based on the RADAR-based information, wherein the function of detecting the construction zone sign is based on the first likelihood and the second likelihood.

14. The non-transitory computer readable medium of claim 13, wherein the RADAR-based information includes dimensional characteristics of the candidate construction zone sign and indicates that the candidate construction zone sign is stationary.

15. The non-transitory computer readable medium of claim 9, wherein the function of determining the type of the construction zone sign comprises:

determining a shape of the construction zone sign in the image portion; and

matching the shape to one or more of shapes of typical construction zone signs.

16. A control system for a vehicle, comprising:

an image-capture device; and

a computing device in communication with the image-capture device, and configured to;

receive, from the image-capture device, an image of a vicinity of a road on which the vehicle is travelling;

select, from the image, an image portion corresponding to a side of the road at a predetermined height;

detect a construction zone sign in the image portion;

determine a type of the construction zone sign in the image portion;

detect one or more construction zone cones or barrels on the road;

determine a number and locations of the one or more construction zone cones or barrels;

determine, based on the number and locations of the one or more construction zone cones or barrels, a severity of road changes;

modify a control strategy associated with a driving behavior of the vehicle based on (i) the type of the construction zone sign, and (ii) the severity of road changes; and

control the vehicle based on the modified control strategy.

17. The control system of claim 16, wherein the computing device is further configured to control the vehicle in an autonomous operation mode.

18. The control system of claim 16, wherein, to detect the construction zone sign, the computing device is configured to:

detect a candidate construction zone sign in the image portion; and

determine that the candidate construction zone sign relates to a construction zone, based on one or more of shape, color, and pattern of the candidate construction zone sign.

19. The control system of claim 16, further comprising a light detection and ranging (LIDAR) sensor coupled to the computing device, and a radio detection and ranging (RADAR) sensor coupled to the computing device, wherein, to detect the construction zone sign, the computing device is configured to:

detect a candidate construction zone sign in the image portion;

determine a first likelihood that the candidate construction zone sign relates to a construction zone, based on one or more of shape, color, and pattern of the candidate construction zone sign;

receive one or more of;

(i) LIDAR-based information comprising a three-dimensional (3D) point cloud corresponding to the image portion corresponding to the candidate construction zone sign, wherein the 3D point cloud comprises a set of points based on light emitted from the LIDAR and reflected from a surface of the candidate construction zone sign, and (ii) RADAR-based information relating to location and one or more characteristics of the candidate construction zone sign; and

determine a second likelihood that the candidate construction zone sign relates to the construction zone, based on the one or more of the LIDAR-based information and the RADAR-based information, wherein the computing device is configured to detect the construction zone sign based on the first likelihood and the second likelihood.