System and method for evaluating the perception system of an autonomous vehicle

• US 10,198,619 B1
• Filed: 01/18/2018
• Issued: 02/05/2019
• Est. Priority Date: 10/05/2010
• Status: Active Grant

• Current Assignee: Waymo LLC (Alphabet Inc.)
• Original Assignee: Waymo LLC (Alphabet Inc.)
• Inventors: Zhu, Jiajun, Urmson, Christopher Paul, Haehnel, Dirk, Fairfield, Nathaniel, Smith, Russell Leigh
• Primary Examiner(s): Yang, Weiwen
• Application Number:
  US15/874,130
• Time in Patent Office: 383 Days
• Field of Search: None
• CPC Class Codes: B60R 1/23 with a predetermined field of view
• CPC Class Codes: B60R 2300/30 characterised by the type of image processing
• CPC Class Codes: B60T 17/18 Safety devices; Monitoring
• CPC Class Codes: B60T 17/221 Procedure or apparatus for checking or keeping in a correct functioning condition of brake systems hydraulic pressure systems in general F15B19/00, F15B21/04; testing structures or apparatus G01M
• CPC Class Codes: B60T 2201/022 Collision avoidance systems
• CPC Class Codes: B60T 2210/32 Vehicle surroundings
• CPC Class Codes: B60T 2270/406 Test-mode; Self-diagnosis
• CPC Class Codes: B60T 7/22 initiated by contact of vehicle, e.g. bumper, with an external object, e.g. another vehicle , or by means of contactless obstacle detectors mounted on the vehicle
• CPC Class Codes: B60T 8/00 Arrangements for adjusting wheel-braking force to meet varying vehicular or ground-surface conditions, e.g. limiting or varying distribution of braking force by changing number of effective brake cylinders in power brake systems B60T17/10
• CPC Class Codes: B60T 8/17 Using electrical or electronic regulation means to control braking detecting or indicating faulty operation B60T8/885
• CPC Class Codes: B60T 8/885 using electrical circuitry
• CPC Class Codes: B60W 2050/0292 Fail-safe or redundant systems, e.g. limp-home or backup systems
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2420/408 Radar; Laser, e.g. lidar
• CPC Class Codes: B60W 2552/05 Type of road, e.g. motorways, local streets, paved or unpaved roads
• CPC Class Codes: B60W 2555/60 Traffic rules, e.g. speed limits or right of way
• CPC Class Codes: B60W 2556/10 Historical data
• CPC Class Codes: B60W 2556/50 of positioning data, e.g. GPS [Global Positioning System] data
• CPC Class Codes: B60W 30/08 Active safety systems predicting or avoiding probable or impending collision or attempting to minimise its consequences
• CPC Class Codes: B60W 30/186 excessive wear or burn out of friction elements, e.g. clutches
• CPC Class Codes: B60W 40/06 : Road conditions
• CPC Class Codes: B60W 50/14 : Means for informing the driver, warning the driver or prompting a driver intervention
• CPC Class Codes: B62D 6/00 : Arrangements for automatically controlling steering depending on driving conditions sensed and responded to, e.g. control circuits means for initiating a change in direction B62D1/00; steering valves B62D5/06; combined with means for inclining the vehicle body or wheels on bends B62D9/00
• CPC Class Codes: G01C 21/3617 : using user history, behaviour, conditions or preferences, e.g. predicted or inferred from previous use or current movement
• CPC Class Codes: G01S 13/865 : Combination of radar systems with lidar systems
• CPC Class Codes: G01S 13/867 : Combination of radar systems with cameras
• CPC Class Codes: G01S 17/86 : Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/931 : of land vehicles
• CPC Class Codes: G05B 2219/2637 : Vehicle, car, auto, wheelchair
• CPC Class Codes: G05D 1/0055 : with safety arrangements
• CPC Class Codes: G05D 1/0088 : characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/021 : specially adapted to land vehicles
• CPC Class Codes: G05D 1/0212 : with means for defining a desired trajectory involving a plurality of land vehicles G05D1/0287
• CPC Class Codes: G05D 1/0214 : in accordance with safety or protection criteria, e.g. avoiding hazardous areas monitoring the location of vehicles within a certain area, e.g. forbidden or allowed areas, in traffic control systems for road vehicles G08G1/13
• CPC Class Codes: G05D 1/024 : in combination with a laser lasers per se H01S
• CPC Class Codes: G05D 1/0246 : using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/0257 : using a radar radar systems designed for anti-collision purposes between land vehicles or between land vehicle and fixed obstacles G01S13/931
• CPC Class Codes: G05D 1/0274 : using mapping information stored in a memory device navigation using map-matching G01C21/30
• CPC Class Codes: G05D 1/0276 : using signals provided by a source external to the vehicle involving a plurality of vehicles G05D1/0287; automatically controlling vehicle speed responsive to externally generated signals B60K31/0058
• CPC Class Codes: G05D 1/0278 : using satellite positioning signals, e.g. GPS
• CPC Class Codes: G06Q 10/02 : Reservations, e.g. for tickets, services or events
• CPC Class Codes: G06Q 30/02 : Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06Q 30/0207 : Discounts or incentives, e.g. coupons or rebates
• CPC Class Codes: G06Q 40/08 : Insurance
• CPC Class Codes: G06T 2207/10004 : Still image; Photographic image
• CPC Class Codes: G06T 2207/10028 : Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30236 : Traffic on road, railway or crossing
• CPC Class Codes: G06T 2207/30252 : Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 2207/30261 : Obstacle
• CPC Class Codes: G06T 7/20 : Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/223 : using block-matching
• CPC Class Codes: G06T 7/231 : using full search
• CPC Class Codes: G06T 7/521 : from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06T 7/74 : involving reference images or patches
• CPC Class Codes: G06V 10/255 : Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 20/56 : exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/58 : Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/588 : Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
• CPC Class Codes: G06V 20/64 : Three-dimensional objects
• CPC Class Codes: G07C 9/00563 : using personal physical data of the operator, e.g. finger prints, retinal images, voicepatterns

Abstract

A method and apparatus are provided for optimizing one or more object detection parameters used by an autonomous vehicle to detect objects in images. The autonomous vehicle may capture the images using one or more sensors. The autonomous vehicle may then determine object labels and their corresponding object label parameters for the detected objects. The captured images and the object label parameters may be communicated to an object identification server. The object identification server may request that one or more reviewers identify objects in the captured images. The object identification server may then compare the identification of objects by reviewers with the identification of objects by the autonomous vehicle. Depending on the results of the comparison, the object identification server may recommend or perform the optimization of one or more of the object detection parameters.

Claims

1. An apparatus for optimizing object detection performed by an autonomous vehicle, the apparatus comprising:

a memory operative to store;

a plurality of images captured by an autonomous vehicle using a plurality of object detection parameters;

a first plurality of object label parameters determined by the autonomous vehicle; and

a second plurality of object label parameters applied by a reviewer having reviewed the plurality of images captured by the autonomous vehicle, the first and second pluralities of object label parameters defining a shape bounding a detected object; and

a processor in communication with the memory, the processor operative to;

determine whether to optimize the plurality of object detection parameters based on a comparison of the first plurality of object label parameters with the second plurality of object label parameters; and

perform an operation on the plurality of object detection parameters based on the comparison of the first plurality of object label parameters with the second plurality of object label parameters, wherein the operation comprises;

identifying a plurality of object detection values, wherein each object detection value corresponds to at least one object detection parameter in the plurality of object detection parameters;

for each combination of the plurality of object detection values, performing an object detection routine on the plurality of images captured by the autonomous vehicle using the plurality of object detection values; and

selecting the combination of the plurality of object detection values that resulted in an optimal object detection routine.

2. The apparatus of claim 1, wherein the operation further comprises displaying the combination of the plurality of object detection values that resulted in an optimal object detection routine.

3. The apparatus of claim 1, wherein a number of times the object detection routine is performed is equal to a number of object detection parameters of the plurality of object detection parameters raised to a power of a number the plurality of the object detection values.

4. The apparatus of claim 3, wherein the number of times the object detection routine is performed equals a same number of different sets of the first plurality of object label parameters stored in memory.

5. The apparatus of claim 1, wherein the plurality of object detection values includes different sensor types used for images captured by the autonomous vehicle.

6. The apparatus of claim 1, wherein the comparison of the first plurality of object label parameters with the second plurality of object label parameters includes determining whether the first plurality of object label parameters overlaps with any portion of the second plurality of object label parameters.

7. The apparatus of claim 1, wherein the comparison of the first plurality of object label parameters with the second plurality of object label parameters includes determining whether the autonomous vehicle has met a predetermined missed object threshold corresponding to a maximum allowable quantity of the first plurality of object label parameters not applied by the autonomous vehicle compared to a threshold value.

8. An method for optimizing object detection performed by an autonomous vehicle, the method comprising:

storing, in a memory, a plurality of images captured by an autonomous vehicle using a plurality of object detection parameters,

receiving, by one or more computing devices having one or more processors, a first plurality of object label parameters determined by the autonomous vehicle and a second plurality of object label parameters applied by a reviewer having reviewed the plurality of images captured by the autonomous vehicle, the first and second pluralities of object label parameters defining a shape bounding a detected object;

determining, by the one or more computing devices, whether to optimize the plurality of object detection parameters based on a comparison of the first plurality of object label parameters with the second plurality of object label parameters;

performing, by the one or more computing devices, an operation on the plurality of object detection parameters based on the comparison of the first plurality of object label parameters with the second plurality of object label parameters;

identifying, by the one or more computing devices, a plurality of object detection values, wherein each object detection value corresponds to at least one object detection parameter in the plurality of object detection parameters;

performing, by the one or more computing devices, an object detection routine on the plurality of images captured by the autonomous vehicle using the plurality of object detection values for each combination of the plurality of object detection values; and

selecting, by the one or more computing devices, the combination of the plurality of object detection values that resulted in an optimal object detection routine.

9. The method of claim 8, further comprising displaying, by the one or more computing devices, the combination of the plurality of object detection values that resulted in an optimal object detection routine.

10. The method of claim 8, wherein a number of times the object detection routine is performed is equal to a number of object detection parameters of the plurality of object detection parameters raised to a power of a number the plurality of the object detection values.

11. The method of claim 10, wherein the number of times the object detection routine is performed equals a same number of different sets of the first plurality of object label parameters stored in memory.

12. The method of claim 8, wherein the plurality of object detection values includes different sensor types used for images captured by the autonomous vehicle.

13. The method of claim 8, wherein the comparison of the first plurality of object label parameters with the second plurality of object label parameters includes determining, by the one or more computing devices, whether the first plurality of object label parameters overlaps with any portion of the second plurality of object label parameters.

14. The method of claim 8, wherein the comparison of the first plurality of object label parameters with the second plurality of object label parameters includes determining, by the one or more computing devices, whether the autonomous vehicle has met a predetermined missed object threshold corresponding to a maximum allowable quantity of the first plurality of object label parameters not applied by the autonomous vehicle compared to a threshold value.

15. A non-transitory, tangible recording medium on which instructions are stored, the instructions, when executed by one or more processors of one or more computing devices, causes the one or more processor to perform a method for optimizing object detection performed by an autonomous vehicle, the method comprising:

storing a plurality of images captured by an autonomous vehicle using a plurality of object detection parameters,

receiving a first plurality of object label parameters determined by the autonomous vehicle and a second plurality of object label parameters applied by a reviewer having reviewed the plurality of images captured by the autonomous vehicle;

determining whether to optimize the plurality of object detection parameters based on a comparison of the first plurality of object label parameters with the second plurality of object label parameters;

performing an operation on the plurality of object detection parameters based on the comparison of the first plurality of object label parameters with the second plurality of object label parameters, the first and second pluralities of object label parameters defining a shape bounding a detected object;

identifying a plurality of object detection values, wherein each object detection value corresponds to at least one object detection parameter in the plurality of object detection parameters;

performing an object detection routine on the plurality of images captured by the autonomous vehicle using the plurality of object detection values for each combination of the plurality of object detection values; and

selecting the combination of the plurality of object detection values that resulted in an optimal object detection routine.

16. The medium of claim 15, further comprising displaying the combination of the plurality of object detection values that resulted in an optimal object detection routine.

17. The medium of claim 15, wherein a number of times the object detection routine is performed is a number of object detection parameters of the plurality of object detection parameters raised to a power of a number the plurality of the object detection values.

18. The medium of claim 17, wherein the number of times the object detection routine is performed equals a same number of different sets of the first plurality of object label parameters stored in a memory.

19. The medium of claim 15, wherein the comparison of the first plurality of object label parameters with the second plurality of object label parameters includes determining whether the first plurality of object label parameters overlaps with any portion of the second plurality of object label parameters.

20. The medium of claim 15, wherein the comparison of the first plurality of object label parameters with the second plurality of object label parameters includes determining whether the autonomous vehicle has met a predetermined missed object threshold corresponding to a maximum allowable quantity of the first plurality of object label parameters not applied by the autonomous vehicle compared to a threshold value.