Method and apparatus for providing trailer information

• US 10,332,002 B2
• Filed: 03/27/2017
• Issued: 06/25/2019
• Est. Priority Date: 03/27/2017
• Status: Active Grant

• Current Assignee: GM Global Technology Operations LLC (General Motors Company), The Board of Trustees of Michigan State University (Michigan State University)
• Original Assignee: GM Global Technology Operations LLC (General Motors Company)
• Inventors: Bliss, Michael, Zhang, Yunfei, Liu, Xiaoming, Atoum, Yousef, Roth, Joseph
• Primary Examiner(s): Osinski, Michael S
• Application Number:
  US15/470,159
• Publication Number:
  US 20180272941A1
• Time in Patent Office: 820 Days
• CPC Class Codes: B60D 1/36 for facilitating connection, e.g. hitch catchers , visual guide means, signalling aids B60D1/465 takes precedence; vehicle signalling in general B60Q; optical arrangements specially adapted for viewing trailer-hitches B60R1/003
• CPC Class Codes: B60D 1/62 involving supply lines, electric circuits, or the like
• CPC Class Codes: B60R 2300/808 for facilitating docking to a trailer
• CPC Class Codes: B62D 15/0295 by overlaying a vehicle path based on present steering angle over an image without processing that image
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle

Abstract

A method and apparatus for providing trailer information are provided. The method includes detecting a p coupler of the trailer in the image of the rear-facing camera; detecting a position of the coupler in the received image; and determining a distance between the detected position of the coupler of the trailer and a hitch of vehicle. The method may be use to display information about a trailer coupler or guide a vehicle to line up a vehicle hitch to a trailer coupler.

Claims

1. A method for providing trailer coupler information, the method comprising:

receiving an image taken by a rear-facing camera;

identifying a coupler of a trailer in the received image;

detecting a position of the identified coupler of the trailer in the received image;

determining a distance between the detected position of the identified coupler of the trailer and a hitch of a vehicle; and

displaying at least one from among information on the determined distance, information on the detected position of the identified coupler of the trailer in the received image, and information to guide the vehicle to the identified coupler,

wherein the identifying the coupler of the trailer in the received image comprises detecting coordinates of the coupler by using a convolutional neural network for identifying a coupler, and

wherein the detecting the position of the identified coupler of the trailer in the received image comprises;

generating N random patches in the received image; and

testing the N random patches using a plurality of convolutional neural networks to estimate the position of the identified coupler of the trailer in the received image,

where N is a positive integer 2 or greater.

2. The method of claim 1, wherein the plurality of convolutional neural networks comprises a first convolutional neural network corresponding to a first distance that is closest to the rear-facing camera, a second convolutional neural network corresponding to a second distance that is further away from the rear-facing camera than the first distance, and a third convolutional neural network corresponding to a third distance that is in between the first distance and second distance.

3. The method of claim 2, wherein the detecting the position of the identified coupler of the trailer in the image further comprises:

estimating contour points of the identified coupler in the received image;

determining a geometry of the identified coupler;

based on the contour points and the geometry, determining a height of the identified coupler.

4. The method of claim 3, wherein the estimating contour points of the identified coupler comprises determining two-dimensional coordinates of edges of the identified coupler using a convolutional neural network for detecting a contour of a coupler.

5. The method of claim 4, further comprising determining an absolute position of the identified coupler based on the height and two-dimensional coordinates of the edges of the identified coupler.

6. The method of claim 1, further comprising:

receiving a user input to adjust the position of the identified coupler of the trailer in the received image; and

based on the user input, re-detecting the position of the identified coupler of the trailer in the received image.

7. The method of claim 1, further comprising:

controlling to guide the vehicle to the identified coupler based on at least one from among the information on the determined distance and the information on the detected position of the identified coupler of the trailer in the received image.

8. A non-transitory medium comprising computer instructions executable to cause a processing device to perform the method of claim 1.

9. An apparatus for providing trailer coupler information, the apparatus comprising:

at least one memory comprising computer executable instructions; and

at least one processor configured to read and execute the computer executable instructions, the computer executable instructions causing the at least one processor to;

receive an image taken by a rear-facing camera;

identify a coupler of a trailer in the received image;

detect a position of the identified coupler of the trailer in the received image;

determine a distance between the detected position of the identified coupler of the trailer and a hitch of a vehicle; and

control to display at least one from among information on the determined distance, information on the detected position of the identified coupler of the trailer in the received image, and information to guide the vehicle to the identified coupler,

wherein the computer executable instructions cause the at least one processor to identify the coupler of the trailer in the received image by detecting coordinates of the coupler by using a convolutional neural network for identifying a coupler, and

wherein the computer executable instructions cause the at least one processor to detect the position of the identified coupler of the trailer in the received image by generating N random patches in the received image and testing the N random patches using a plurality of convolutional neural networks to estimate the position of the identified coupler of the trailer in the received image,

where N is a positive integer 2 or greater.

10. The apparatus of claim 9, wherein the plurality of convolutional neural networks comprises a first convolutional neural network corresponding to a first distance that is closest to the rear-facing camera, a second convolutional neural network corresponding to a second distance that is further away from the rear-facing camera than the first distance, and a third convolutional neural network corresponding to a third distance that is in between the first distance and second distance.

11. The apparatus of claim 10, wherein the computer executable instructions cause the at least one processor to detect the position of the identified coupler of the trailer in the received image by estimating contour points of the identified coupler in the image, determining a geometry of the identified coupler, and based on the contour points and the geometry, determining a height of the identified coupler.

12. The apparatus of claim 11, wherein the computer executable instructions cause the at least one processor to estimate contour points of the identified coupler by determining two-dimensional coordinates of edges of the identified coupler using a convolutional neural network for detecting a contour of a coupler.

13. The apparatus of claim 12, wherein the computer executable instructions cause the at least one processor to determine an absolute position of the identified coupler based on the height and two-dimensional coordinates of the edges of the identified coupler.

14. The apparatus of claim 9, wherein the computer executable instructions further cause the at least one processor to receive a user input to adjust the position of the identified coupler of the trailer in the received image;

and based on the user input, re-detect the position of the identified coupler of the trailer in the received image.

15. The apparatus of claim 14, wherein the computer executable instructions cause the at least one processor to control to guide the vehicle to the identified coupler based on at least one from among the information on the determined distance and the information on the detected position of the identified coupler of the trailer in the received image.

16. A non-transitory computer readable medium comprising computer executable instructions executable by a processor to perform a method for providing trailer information, the method comprising:

detecting a position of a coupler of a trailer in an image taken by a rear-facing camera by using a convolutional neural network; and

determining a distance between the detected position of the coupler of the trailer and a hitch of vehicle,

wherein the detecting the position of the coupler of the trailer in the image taken by the rear-facing camera comprises;

detecting coordinates of the coupler of the trailer in the image by using the convolutional neural network for identifying the coupler;

generating N random patches in the image; and

testing the N random patches using a plurality of convolutional neural networks to estimate the position of the coupler of the trailer in the image,

where N is a positive integer 2 or greater.

17. The non-transitory medium of claim 16, wherein the method further comprises performing at least one from among:

controlling to guide the vehicle to the coupler of the trailer in the image based on at least one from among the information on the determined distance and the information on the detected position of the coupler of the trailer; and

displaying at least one from among information on the determined distance, information the detected position of the coupler of the trailer, and information to guide a vehicle to the coupler.