System and method for camera or sensor-based parking spot detection and identification

• US 10,720,058 B2
• Filed: 09/13/2018
• Issued: 07/21/2020
• Est. Priority Date: 09/13/2018
• Status: Active Grant

• Current Assignee: Polestar Performance AB (Polestar Automotive Holding UK Plc)
• Original Assignee: Volvo Car Corporation (AB Volvo)
• Inventors: Ding, Sihao, Wallin, Andreas
• Primary Examiner(s): Previl, Daniel
• Application Number:
  US16/129,871
• Publication Number:
  US 20200090519A1
• Time in Patent Office: 677 Days
• Field of Search: 3409322, 340428, 340438, 3404573, 340464, 340466, 340517, 340525, 34053913, 34053922, 34053926
• CPC Class Codes: B60R 2300/607 from a bird's eye viewpoint
• CPC Class Codes: B60R 2300/806 for aiding parking
• CPC Class Codes: B62D 15/0275 by overlaying a vehicle path based on present steering angle over an image without processing that image
• CPC Class Codes: G06T 2207/30264 Parking
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/586 of parking space
• CPC Class Codes: G08G 1/168 Driving aids for parking, e.g. acoustic or visual feedback on parking space

Abstract

The present invention provides an on-board vehicle system and method for camera or sensor-based parking spot detection and identification. Advantageously, this system and method utilizes a standard front (or side or rear) camera or sensor image to detect and identify one or more parking spots at a distance via vector or like representation using a deep neural network trained with data annotated using an annotation tool, without first transforming the standard camera or sensor image(s) to a bird'"'"'s-eye-view (BEV) or the like. The system and method can be incorporated in a driver-assist (DA) or autonomous driving (AD) system.

Claims

1. A system for detecting and identifying a parking spot, comprising:

a camera or sensor operable for obtaining an image comprising a parking spot, wherein the camera or sensor comprises one or more of a front camera or sensor, a side camera or sensor, and a rear camera or sensor mounted on a vehicle, and wherein the image comprises one or more of a front image, a side image, and a rear image showing a corresponding view from the vehicle; and

a processor executing an algorithm operable for segmenting the parking spot in the image and overlaying a vector representation connecting points associated with front corners of the parking spot and generating a representation of the parking spot, wherein the representation comprises information related to one or more of a location, a size, an orientation, and a classification of the parking spot.

2. The system of claim 1, wherein the camera or sensor comprises multiple of the front camera or sensor, the side camera or sensor and the rear camera or sensor, and wherein the image comprises multiple of the front image, the side image, and the rear image stitched together.

3. The system of claim 1, wherein the processor is disposed one of on-board the vehicle and remote from the vehicle in a network cloud.

4. The system of claim 1, wherein the algorithm is operable for executing a stage comprising one or more of an image pre-processing stage, a network operations stage, and a post-processing stage that comprises a decoding step that interprets output of the network operations stage and a non-maximum suppression step, and wherein the network comprises an artificial intelligence network trained using a plurality of training images that are annotated using an annotation tool.

5. The system of claim 4, wherein the annotation tool is operable for receiving selected points of interest on the plurality of images from an annotator, segmenting the training images based on the selected points of interest, annotating the plurality of training images, and saving the results as a j son file or the like, wherein the selected points of interest utilize one or more classes of markers, and wherein the plurality of training images are annotated using one or more classes of representations based on the one or more classes of markers.

6. The system of claim 1, wherein the algorithm utilizes a generative algorithm operable for modifying the representation such that it is consistent with surrounding representations generated from the image.

7. The system of claim 1, wherein the classification of the parking spot comprises one or more of unoccupied, occupied, and parking spot type.

8. The system of claim 1, further comprising a display operable for displaying the representation to a driver of the vehicle overlaid on one of the image and another image.

9. The system of claim 1, wherein the representation is one or more of:

communicated to and used by one or more of a driver-assist system and an autonomous driving system of the vehicle to maneuver the vehicle into the parking spot when the representation indicates that the parking spot is unoccupied, communicated to a cloud network and shared with other vehicles, and used to generate a map of a parking area.

10. The system of claim 1, wherein the camera or sensor is operable for obtaining the image at least 30 meters away.

11. The system of claim 1, wherein the system is used in conjunction with one or more other sensors or systems of the vehicle operable for assessing the position of the vehicle in the surrounding environment.

12. A method for detecting and identifying a parking spot, comprising:

obtaining an image comprising a parking spot using a camera or sensor, wherein the camera or sensor comprises one or more of a front camera or sensor, a side camera or sensor, and a rear camera or sensor mounted on a vehicle, and wherein the image comprises one or more of a front image, a side image, and a rear image showing a corresponding view from the vehicle; and

segmenting the parking spot in the image and overlaying a vector representation connecting points associated with front corners of the parking spot and generating a representation of the parking spot using a processor executing an algorithm, wherein the representation comprises information related to one or more of a location, a size, an orientation, and a classification of the parking spot.

13. The method of claim 12, wherein the camera or sensor comprises multiple of the front camera or sensor, the side camera or sensor and the rear camera or sensor, and wherein the image comprises multiple of the front image, the side image, and the rear image stitched together.

14. The method of claim 12, wherein the processor is disposed one of on-board the vehicle and remote from the vehicle in a network cloud.

15. The method of claim 12, wherein the algorithm is operable for executing a stage comprising one or more of an image pre-processing stage, a network operations stage, and a post-processing stage that comprises a decoding step that interprets output of the network operations stage and a non-maximum suppression step, and wherein the network comprises an artificial intelligence network trained using a plurality of training images that are annotated using an annotation tool.

16. The method of claim 15, wherein the annotation tool is operable for receiving selected points of interest on the plurality of images from an annotator, segmenting the training images based on the selected points of interest, annotating the plurality of training images, and saving the results as a j son file or the like, wherein the selected points of interest utilize one or more classes of markers, and wherein the plurality of training images are annotated using one or more classes of representations based on the one or more classes of markers.

17. The method of claim 12, wherein the algorithm utilizes a generative algorithm operable for modifying the representation such that it is consistent with surrounding representations generated from the image.

18. The method of claim 12, wherein the classification of the parking spot comprises one or more of unoccupied, occupied, and parking spot type.

19. The method of claim 12, further comprising displaying the representation to a driver of the vehicle overlaid on one of the image and another image using a display.

20. The method of claim 12, wherein the representation is one or more of:

communicated to and used by one or more of a driver-assist system and an autonomous driving system of the vehicle to maneuver the vehicle into the parking spot when the representation indicates that the parking spot is unoccupied, communicated to a cloud network and shared with other vehicles, and used to generate a map of a parking area.

21. The method of claim 12, wherein the camera or sensor is operable for obtaining the image at least 30 meters away.

22. The method of claim 12, wherein the method is used in conjunction with one or more other sensors or systems of the vehicle operable for assessing the position of the vehicle in the surrounding environment.

23. A vehicle comprising a system for detecting and identifying a parking spot, the vehicle comprising:

a camera or sensor operable for obtaining an image comprising a parking spot, wherein the camera or sensor comprises one or more of a front camera or sensor, a side camera or sensor, and a rear camera or sensor mounted on the vehicle, and wherein the image comprises one or more of a front image, a side image, and a rear image showing a corresponding view from the vehicle;

one of an on-board processor and a communications link to a remote processor executing an algorithm operable for segmenting the parking spot in the image and overlaying a vector representation connecting points associated with front corners of the parking spot and generating a representation of the parking spot, wherein the representation comprises information related to one or more of a location, a size, an orientation, and a classification of the parking spot; and

a display operable for displaying the representation to a driver of the vehicle overlaid on one of the image and another image.

24. The vehicle of claim 23, wherein the camera or sensor comprises multiple of the front camera or sensor, the side camera or sensor and the rear camera or sensor, and wherein the image comprises multiple of the front image, the side image, and the rear image stitched together.

25. The vehicle of claim 23, wherein the algorithm comprises an artificial intelligence network trained using a plurality of training images that are annotated using an annotation tool, and wherein the algorithm utilizes a generative algorithm operable for modifying the representation such that it is consistent with surrounding representations generated from the image.

26. The vehicle of claim 23, further comprising one or more of a driver-assist system and an autonomous driving system operable for receiving the representation and maneuvering the vehicle into the parking spot when the representation indicates that the parking spot is unoccupied.