Construction zone sign detection

• US 9,489,601 B2
• Filed: 10/15/2015
• Issued: 11/08/2016
• Est. Priority Date: 09/05/2012
• Status: Expired

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: X Development LLC (Alphabet Inc.)
• Inventors: Fairfield, Nathaniel, Ferguson, David Ian, Ogale, Abhijit, Wang, Matthew, Yee, Yangli Hector
• Primary Examiner(s): Beaulieu, Yonel
• Assistant Examiner(s): KLEINMAN, LAIL A
• Application Number:
  US14/883,956
• Publication Number:
  US 20160092755A1
• Time in Patent Office: 390 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: B60T 1/10 by utilising wheel movement for accumulating energy, e.g. driving air compressors
• CPC Class Codes: B60T 2201/08 Lane monitoring; Lane Keeping Systems
• CPC Class Codes: B60T 7/12 for automatic initiation; for initiation not subject to will of driver or passenger limiting speed of vehicles other than rail vehicles B60K31/00
• CPC Class Codes: B60T 7/18 operated by wayside apparatus
• CPC Class Codes: B60T 7/22 initiated by contact of vehicle, e.g. bumper, with an external object, e.g. another vehicle , or by means of contactless obstacle detectors mounted on the vehicle
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2552/00 Input parameters relating to infrastructure
• CPC Class Codes: B60W 2552/10 Number of lanes
• CPC Class Codes: B60W 2552/50 Barriers
• CPC Class Codes: B60W 2552/53 Road markings, e.g. lane marker or crosswalk
• CPC Class Codes: B60W 2554/408 Traffic behavior, e.g. swarm
• CPC Class Codes: B60W 2555/60 Traffic rules, e.g. speed limits or right of way
• CPC Class Codes: B60W 30/10 Path keeping cruise control for automatically following a preceding vehicle B60W30/165
• CPC Class Codes: B60W 30/143 Speed control B60W30/16 takes precedence
• CPC Class Codes: B60W 40/06 Road conditions
• CPC Class Codes: B60W 60/00184 related to infrastructure
• CPC Class Codes: B62D 15/025 Active steering aids, e.g. helping the driver by actively influencing the steering system after environment evaluation B62D1/28 takes precedence
• CPC Class Codes: B62D 15/0255 Automatic changing of lane, e.g. for passing another vehicle
• CPC Class Codes: B62D 15/0265 Automatic obstacle avoidance by steering
• CPC Class Codes: B62D 6/00 Arrangements for automatically controlling steering depending on driving conditions sensed and responded to, e.g. control circuits means for initiating a change in direction B62D1/00; steering valves B62D5/06; combined with means for inclining the vehicle body or wheels on bends B62D9/00
• CPC Class Codes: G01C 22/00 : Measuring distance traversed on the ground by vehicles, persons, animals or other moving solid bodies, e.g. using odometers, using pedometers
• CPC Class Codes: G01S 13/865 : Combination of radar systems with lidar systems
• CPC Class Codes: G01S 13/867 : Combination of radar systems with cameras
• CPC Class Codes: G01S 13/931 : of land vehicles
• CPC Class Codes: G01S 17/89 : for mapping or imaging
• CPC Class Codes: G01S 17/931 : of land vehicles
• CPC Class Codes: G01S 2013/9316 : combined with communication equipment with other vehicles or with base stations
• CPC Class Codes: G01S 2013/932 : using own vehicle data, e.g. ground speed, steering wheel direction
• CPC Class Codes: G01S 7/4802 : using analysis of echo signal for target characterisation; Target signature; Target cross-section
• CPC Class Codes: G01S 7/4808 : Evaluating distance, position or velocity data
• CPC Class Codes: G05D 1/024 : in combination with a laser lasers per se H01S
• CPC Class Codes: G05D 1/0246 : using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/0257 : using a radar radar systems designed for anti-collision purposes between land vehicles or between land vehicle and fixed obstacles G01S13/931
• CPC Class Codes: G05D 1/027 : comprising intertial navigation means, e.g. azimuth detector inertial navigation G01C21/16; inertial navigation combined with non-inertial navigation instruments G01C21/165
• CPC Class Codes: G05D 1/0278 : using satellite positioning signals, e.g. GPS
• CPC Class Codes: G06V 20/58 : Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/582 : of traffic signs
• CPC Class Codes: G06V 20/588 : Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
• CPC Class Codes: H04N 7/18 : Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

Methods and systems for detection of a construction zone sign are described. A computing device, configured to control the vehicle, may be configured to receive, from an image-capture device coupled to the computing device, images of a vicinity of the road on which the vehicle is travelling. Also, the computing device may be configured to determine image portions in the images that may depict sides of the road at a predetermined height range. Further, the computing device may be configured to detect a construction zone sign in the image portions, and determine a type of the construction zone sign. Accordingly, the computing device may be configured to modify a control strategy associated with a driving behavior of the vehicle; and control the vehicle based on the modified control strategy.

Claims

1. A method comprising:

receiving, at a computing device configured to control a vehicle, from an image-capture device coupled to the vehicle, an image of a vicinity of a road on which the vehicle is travelling;

selecting, from the image, using the computing device, an image portion corresponding to a side of the road at a predetermined height;

detecting, using the computing device, a construction zone sign in the image portion;

determining, using the computing device, a type of the construction zone sign;

detecting one or more construction zone objects on the road;

determining, based on a number and locations of the one or more construction zone objects, a severity of road changes; and

controlling, using the computing device, the vehicle based on (i) the type of the construction zone sign, and (ii) the severity of road changes.

2. The method of claim 1, wherein controlling the vehicle comprises controlling the vehicle in an autonomous operation mode.

3. The method of claim 1, wherein detecting the construction zone sign comprises:

detecting a candidate construction zone sign in the image portion; and

determining that the candidate construction zone sign relates to a construction zone based on one or more of shape, color, and pattern of the candidate construction zone sign.

4. The method of claim 1, further comprising:

detecting a candidate construction zone sign in the image portion;

determining a first likelihood that the candidate construction zone sign relates to a construction zone based on one or more of shape, color, and pattern of the candidate construction zone sign;

receiving, at the computing device, from a light detection and ranging (LIDAR) device coupled to the vehicle, LIDAR-based information comprising a three-dimensional (3D) point cloud corresponding to the image portion, wherein the 3D point cloud comprises a set of points based on light emitted from the LIDAR and reflected from a surface of the candidate construction zone sign; and

determining a second likelihood that the candidate construction zone sign relates to the construction zone based on the LIDAR-based information, wherein detecting the construction zone sign is based on the first likelihood and the second likelihood.

5. The method of claim 1, further comprising:

detecting a candidate construction zone sign in the image portion;

determining a first likelihood that the candidate construction zone sign relates to a construction zone, based on one or more of shape, color, and pattern of the candidate construction zone sign;

receiving, at the computing device, from a radio detection and ranging (RADAR) device coupled to the vehicle, RADAR-based information relating to location and one or more characteristics of the candidate construction zone sign; and

determining a second likelihood that the candidate construction zone sign relates to the construction zone based on the RADAR-based information, wherein detecting the construction zone sign is based on the first likelihood and the second likelihood.

6. The method of claim 5, wherein the RADAR-based information includes dimensional characteristics of the candidate construction zone sign and indicates that the candidate construction zone sign is stationary.

7. The method of claim 1, wherein determining the type of the construction zone sign comprises:

determining a shape of the construction zone sign; and

matching the shape to one or more of shapes of typical construction zone signs.

8. The method of claim 1, wherein the one or more construction zone objects comprise a cone, a barrel, a worker, a barrier, or construction equipment.

9. A non-transitory computer readable medium having stored thereon instructions that, when executed by a computing device of a vehicle, cause the computing device to perform operations comprising:

receiving, from an image-capture device coupled to the vehicle, an image of a vicinity of a road of travel of the vehicle;

selecting, from the image, an image portion corresponding to a side of the road;

detecting a construction zone sign in the image portion;

determining a type of the construction zone sign;

detecting one or more construction zone objects on the road;

determining, based on a number and locations of the one or more construction zone objects, a severity of road changes; and

controlling the vehicle based on (i) the type of the construction zone sign, and (ii) the severity of road changes.

10. The non-transitory computer readable medium of claim 9, wherein the one or more construction zone objects comprise a cone, a barrel, a worker, a barrier, or construction equipment.

11. The non-transitory computer readable medium of claim 9, wherein the operations further comprise:

identifying a lane boundary defined by the one or more constructions zone objects, wherein determining the severity of road changes is further based on the identified lane boundary.

12. The non-transitory computer readable medium of claim 9, wherein the operations further comprise:

detecting a candidate construction zone sign in the image portion;

determining a first likelihood that the candidate construction zone sign relates to a construction zone, based on one or more of shape, color, and pattern of the candidate construction zone sign;

receiving, from a light detection and ranging (LIDAR) device coupled to the vehicle, LIDAR-based information comprising a three-dimensional (3D) point cloud corresponding to the image portion, wherein the 3D point cloud comprises a set of points based on light emitted from the LIDAR and reflected from a surface of the candidate construction zone sign; and

determining a second likelihood that the candidate construction zone sign relates to the construction zone based on the LIDAR-based information, wherein detecting the construction zone sign is based on the first likelihood and the second likelihood.

13. The non-transitory computer readable medium of claim 9, wherein the operations further comprise:

detecting a candidate construction zone sign in the image portion;

determining a first likelihood that the candidate construction zone sign relates to a construction zone, based on one or more of shape, color, and pattern of the candidate construction zone sign;

receiving from a radio detection and ranging (RADAR) device coupled to the vehicle, RADAR-based information relating to location and one or more characteristics of the candidate construction zone sign; and

determining a second likelihood that the candidate construction zone sign relates to the construction zone based on the RADAR-based information, wherein detecting the construction zone sign is based on the first likelihood and the second likelihood.

14. The non-transitory computer readable medium of claim 13, wherein the RADAR-based information includes dimensional characteristics of the candidate construction zone sign and indicates that the candidate construction zone sign is stationary.

15. The non-transitory computer readable medium of claim 9, wherein determining the type of the construction zone sign comprises:

determining a shape of the construction zone sign in the image portion; and

matching the shape to one or more of shapes of typical construction zone signs.

16. A control system for a vehicle, comprising:

an image-capture device;

a computing device in communication with the image-capture device; and

data storage comprising instructions that, when executed by the computing device, cause the control system to perform operations comprising;

receiving, from an image-capture device coupled to the vehicle, an image of a vicinity of a road of travel of the vehicle,

selecting, from the image, an image portion corresponding to a side of the road;

detecting a construction zone sign in the image portion,

determining a type of the construction zone sign,

detecting one or more construction zone objects on the road,

determining, based on a number and locations of the one or more construction zone objects, a severity of road changes, and

controlling the vehicle based on (i) the type of the construction zone sign, and (ii) the severity of road changes.

17. The control system of claim 16, wherein controlling the vehicle comprises controlling the vehicle in an autonomous operation mode.

18. The control system of claim 16, wherein detecting the construction zone sign comprises:

detecting a candidate construction zone sign in the image portion; and

determining that the candidate construction zone sign relates to a construction zone based on one or more of shape, color, and pattern of the candidate construction zone sign.

19. The control system of claim 16, wherein the one or more construction zone objects comprise a cone, a barrel, a worker, a barrier, or construction equipment.

20. The control system of claim 16, wherein the operations further comprise:

identifying a lane boundary defined by the one or more constructions zone objects, wherein determining the severity of road changes is further based on the identified lane boundary.