State and position prediction of observed vehicles using optical tracking of wheel rotation

• US 10,706,563 B2
• Filed: 05/15/2018
• Issued: 07/07/2020
• Est. Priority Date: 05/15/2018
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Kushleyev, Aleksandr, Chaves, Stephen Marc, Turpin, Matthew Hyatt
• Primary Examiner(s): Woldemariam, Aklilu K
• Application Number:
  US15/980,300
• Publication Number:
  US 20190355132A1
• Time in Patent Office: 784 Days
• Field of Search: 382104, 348 36, 348148, 348149, 701 29, 701 41
• CPC Class Codes: G01C 21/3602 Input other than that of destination using image analysis, e.g. detection of road signs, lanes, buildings, real preceding vehicles using a camera
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06T 7/62 of area, perimeter, diameter or volume
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 2201/08 Detecting or categorising vehicles

Abstract

Various embodiments may include methods of using image data to estimate motion of a vehicle observed within camera images, such as images captured by a vehicle navigation system of a host vehicle. Various embodiments may include a camera capturing a sequence of images including the observed vehicle, and a processor performing image processing to identify a wheel of the observed vehicle, and determining a rate of rotation of the wheel based on changes in orientation of the wheel between at least two images within the sequence of images. The processor may further determine a speed of the observed vehicle based on the wheel'"'"'s rate of rotation and diameter. The processor may further determine a direction of travel and/or turning rate of the observed vehicle by determining relative angles of wheels of the observed vehicle in at least one image.

Claims

1. A method of using image data to estimate motion of an observed vehicle, comprising:

capturing, by a camera, a sequence of images including the observed vehicle;

performing, by a processor, image processing to identify a wheel of the observed vehicle within the sequence of images; and

determining, by the processor, a rate of rotation of the wheel based on changes in orientation of the wheel between at least two images within the sequence of images, comprising;

performing image processing to identify a feature of the wheel present in a first image of the wheel and a second image of the wheel;

performing image processing to identify a center of the wheel in each of the first and second images of the wheel;

determining an amount of positional change of the identified feature of the wheel between the first and second images of the wheel;

determining an angle of rotation of the identified feature of the wheel between the first and second images of the wheel based on the amount of positional change of the identified feature of the wheel between the first and second images of the wheel and the center of the wheel in each of the first and second images of the wheel;

determining an amount of time elapsed between the first image and the second image; and

determining the rate of rotation of the wheel based on the determining the angle of rotation of the identified feature and the amount of time elapsed between the first and second images of the wheel.

2. The method of claim 1, further comprising:

performing image processing of at least one image to determining a diameter of the wheel; and

determining a speed of the observed vehicle based on the determined rate of rotation of the wheel and the diameter of the wheel.

3. The method of claim 2, further comprising:

augmenting a vehicle navigation system of a host vehicle using the determined speed of the observed vehicle.

6. The method of claim 1, further comprising determining whether the observed vehicle is turning based on image processing of a front wheel and a rear wheel of the observed vehicle within the sequence of images.

7. The method of claim 1, further comprising determining an angle of turn of the observed vehicle based on image processing of a front wheel and a rear wheel of the observed vehicle within the sequence of images.

8. The method of claim 7, wherein performing image processing to identify the wheel of the observed vehicle within the sequence of images comprises performing, by the processor, image processing of at least one image to identify the front wheel and the rear wheel of the observed vehicle, the method further comprising:

image processing of the sequence of images to determine a relative angle of the front wheel with respect to a camera capturing the sequence of images;

image processing of at least one image to determine a relative angle of the rear wheel with respect to the camera;

image processing of at least one image to determine a distance between the front wheel and the rear wheel; and

determining a rate of turn of the observed vehicle based on the distance between the front wheel and the rear wheel, a linear speed of the observed vehicle based on the determined rate of rotation, and a difference between the relative angle of the front wheel and the relative angle of the rear wheel.

9. The method of claim 7, further comprising:

augmenting a vehicle navigation system of a host vehicle using the determining angle of turn of the observed vehicle.

4. A method of using image data to estimate motion of an observed vehicle, comprising:

capturing, by a camera, a sequence of images including the observed vehicle;

performing, by a processor, image processing to identify a wheel of the observed vehicle within the sequence of images; and

determining a direction of travel of the observed vehicle based on image processing of the wheel within the sequence of images, comprising;

performing image processing of at least one image to determine an observed horizontal diameter of the wheel;

performing image processing of at least one image to determine an observed vertical diameter of the wheel; and

determining a relative angle of the wheel with respect to the camera based at least upon a ratio of the observed horizontal diameter to the observed vertical diameter.

5. The method of claim 4, further comprising:

augmenting a vehicle navigation system of a host vehicle using the determined direction of travel of the observed vehicle.

24. The method of claim 4, wherein the determining the direction of travel of the observed vehicle based on image processing of the wheel within the sequence of images further comprises:

calculating the ratio of the observed horizontal diameter to the observed vertical diameter;

determining a viewing angle to the wheel from the camera capturing the sequence of images; and

determining the relative angle of the wheel with respect to the camera based at least upon the determined viewing angle to the wheel from the camera.

10. A vehicle navigation system, comprising:

a camera configured to capture images external to a vehicle; and

a processor configured to receive a sequence of images from the camera and configured with processor-executable instructions to;

perform image processing to identify a wheel of an observed vehicle within the sequence of images; and

determine a rate of rotation of the wheel based on changes in orientation of the wheel between at least two images within the sequence of images, comprising;

identifying a feature of the wheel present in a first image of the wheel and a second image of the wheel;

performing image processing to identify a center of the wheel in each of the first and second images of the wheel;

determining an amount of positional change of the identified feature of the wheel between the first and second images of the wheel;

determining an angle of rotation of the identified feature of the wheel between the first and second images of the wheel based on the amount of positional change of the identified feature of the wheel between the first and second images of the wheel and the center of the wheel in each of the first and second images of the wheel;

determining an amount of time elapsed between the first image and the second image; and

determining the rate of rotation of the wheel based on the determining the angle of rotation of the identified feature and the amount of time elapsed between the first and second images of the wheel.

11. The vehicle navigation system of claim 10, wherein the processor is further configured with processor-executable instructions to:

perform image processing to determine a diameter of the wheel; and

determine a speed of the observed vehicle based on the determined rate of rotation of the wheel and the diameter of the wheel.

13. The vehicle navigation system of claim 10, wherein the processor is further configured with processor-executable instructions to determine whether the observed vehicle is turning based on image processing of a front wheel and a rear wheel of the observed vehicle within the sequence of images.

14. The vehicle navigation system of claim 10, wherein the processor is further configured with processor-executable instructions to determine an angle of turn of the observed vehicle based on image processing of a front wheel and a rear wheel of the observed vehicle within the sequence of images.

15. The vehicle navigation system of claim 14, wherein the processor is further configured with processor-executable instructions to:

perform image processing on at least one image to identify the front wheel and the rear wheel of the observed vehicle;

perform image processing on at least one image to determine a relative angle of the front wheel with respect to the camera;

perform image processing on at least one image to determine a relative angle of the rear wheel with respect to the camera;

perform image processing on at least one image to determine a distance between the front wheel and the rear wheel; and

determine a rate of turn of the observed vehicle based on the distance between the front wheel and the rear wheel, a linear speed of the observed vehicle based on the determined rate of rotation, and a difference between the relative angle of the front wheel and the relative angle of the rear wheel.

12. A vehicle navigation system, comprising:

a camera configured to capture images external to a vehicle; and

a processor configured to receive a sequence of images from the camera and configured with processor-executable instructions to;

perform image processing to identify a wheel of an observed vehicle within the sequence of images; and

determine a direction of travel of the observed vehicle based on image processing of the wheel within the sequence of images, comprising;

performing image processing of at least one image to determine an observed horizontal diameter of the wheel;

performing image processing of at least one image to determine an observed vertical diameter of the wheel; and

determining a relative angle of the wheel with respect to the camera based at least upon a ratio of the observed horizontal diameter to the observed vertical diameter.

25. The vehicle navigation system of claim 12, wherein the processor is further configured with processor-executable instructions to determine the direction of travel of the observed vehicle based on image processing of the wheel within the sequence of images by:

calculating the ratio of the observed horizontal diameter to the observed vertical diameter;

determining a viewing angle to the wheel from the camera capturing the sequence of images; and

determining the relative angle of the wheel with respect to the camera based at least upon the determined viewing angle to the wheel from the camera.

16. A processing device for use in a vehicle, the processing device configured to:

receive a sequence of images from a camera on the vehicle configured to obtain images external to the vehicle;

perform image processing to identify a wheel of an observed vehicle within the sequence of images; and

determine a rate of rotation of the wheel based on changes in orientation of the wheel between at least two images within the sequence of images, comprising;

identifying a feature of the wheel present in a first image of the wheel and a second image of the wheel;

performing image processing to identify a center of the wheel in each of the first and second images of the wheel;

determining an amount of positional change of the identified feature of the wheel between the first and second images of the wheel;

determining an angle of rotation of the identified feature of the wheel between the first and second images of the wheel based on the amount of positional change of the identified feature of the wheel between the first and second images of the wheel and the center of the wheel in each of the first and second images of the wheel;

determining an amount of time elapsed between the first image and the second image; and

determining the rate of rotation of the wheel based on the determining the angle of rotation of the identified feature and the amount of time elapsed between the first and second images of the wheel.

17. The processing device of claim 16, wherein the processing device is further configured to:

perform image processing to determine a diameter of the wheel; and

determine a speed of the observed vehicle based on the determined rate of rotation of the wheel and the diameter of the wheel.

19. The processing device of claim 16, wherein the processing device is further configured to determine whether the observed vehicle is turning based on image processing of a front wheel and a rear wheel of the observed vehicle within the sequence of images.

20. The processing device of claim 16, wherein the processing device is further configured to determine an angle of turn of the observed vehicle based on image processing of a front wheel and a rear wheel of the observed vehicle within the sequence of images.

21. The processing device of claim 20, wherein the processing device is further configured to:

perform image processing on at least one image to identify the front wheel and the rear wheel of the observed vehicle;

perform image processing on at least one image to determine a relative angle of the front wheel with respect to the camera;

perform image processing on at least one image to determine a relative angle of the rear wheel with respect to the camera;

perform image processing on at least one image to determine a distance between the front wheel and the rear wheel; and

determine a rate of turn of the observed vehicle based on the distance between the front wheel and the rear wheel, a linear speed of the observed vehicle based on the determined rate of rotation, and a difference between the relative angle of the front wheel and the relative angle of the rear wheel.

18. A processing device for use in a vehicle, the processing device configured to:

receive a sequence of images from a camera on the vehicle configured to obtain images external to the vehicle;

perform image processing to identify a wheel of an observed vehicle within the sequence of images; and

determine a direction of travel of the observed vehicle based on image processing of the wheel within the sequence of images, comprising;

performing image processing of at least one image to determine an observed horizontal diameter of the wheel;

performing image processing of at least one image to determine an observed vertical diameter of the wheel; and

determining a relative angle of the wheel with respect to the camera based at least upon a ratio of the observed horizontal diameter to the observed vertical diameter.

26. The processing device of claim 18, wherein the processing device is further configured with processor-executable instructions to determine the direction of travel of the observed vehicle based on image processing of the wheel within the sequence of images by:

calculating the ratio of the observed horizontal diameter to the observed vertical diameter;

determining a viewing angle to the wheel from the camera capturing the sequence of images; and

determining the relative angle of the wheel with respect to the camera based at least upon the determined viewing angle to the wheel from the camera.

22. A non-transitory, processor-readable medium having stored thereon processor-executable instructions configured to cause a processor of a vehicle navigation system to perform operations comprising:

capturing, by a camera, a sequence of images including an observed vehicle;

performing, by a processor, image processing to identify a wheel of the observed vehicle within the sequence of images; and

determining, by the processor, a rate of rotation of the wheel based on changes in orientation of the wheel between at least two images within the sequence of images, comprising;

performing image processing to identify a feature of the wheel present in a first image of the wheel and a second image of the wheel;

performing image processing to identify a center of the wheel in each of the first and second images of the wheel;

determining an amount of positional change of the identified feature of the wheel between the first and second images of the wheel;

determining an angle of rotation of the identified feature of the wheel between the first and second images of the wheel based on the amount of positional change of the identified feature of the wheel between the first and second images of the wheel and the center of the wheel in each of the first and second images of the wheel;

determining an amount of time elapsed between the first image and the second image; and

determining the rate of rotation of the wheel based on the determining the angle of rotation of the identified feature and the amount of time elapsed between the first and second images of the wheel.

23. The non-transitory, processor-readable medium of claim 22, wherein the stored processor-executable instructions are configured to cause a processor of a vehicle navigation system to perform operations further comprising:

performing image processing of at least one image to determining a diameter of the wheel; and

determining a speed of the observed vehicle based on the determined rate of rotation of the wheel and the diameter of the wheel.