DYNAMIC GUIDELINE OVERLAY WITH IMAGE CROPPING

• US 20130321629A1
• Filed: 05/31/2012
• Published: 12/05/2013
• Est. Priority Date: 05/31/2012
• Status: Active Grant

• Current Assignee: GM Global Technology Operations LLC (General Motors Company)
• Original Assignee: GM Global Technology Operations LLC (General Motors Company)
• Inventors: Zhang, Wende, Litkouhi, Bakhtiar Brian, Lybecker, Kent S.
• Granted Patent: US 9,738,223 B2
• Time in Patent Office: Days
• US Class Current: 348/148
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 2300/302 combining image information with GPS information or vehicle data, e.g. vehicle speed, gyro, steering angle data
• CPC Class Codes: B60R 2300/305 merging camera image with lines or icons
• CPC Class Codes: B60R 2300/306 using a re-scaling of images
• CPC Class Codes: B60R 2300/406 using wireless transmission
• CPC Class Codes: B60R 2300/50 characterised by the display information being shared, e.g. external display, data transfer to other traffic participants or centralised traffic controller
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 3/00 Geometric image transformations in the plane of the image
• CPC Class Codes: G06T 3/14 Transformations for image registration, e.g. adjusting or mapping for alignment of images
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: H04N 17/002 for television cameras

Abstract

A method for dynamically registering a graphic upon a cropped image obtained from a camera device includes capturing an original image obtained from the camera device. Intrinsic calibration information for the camera device, extrinsic information for the camera device and vehicle information are monitored. The cropped image is generated based on cropping parameters to exclude undesirable content from the original image. The intrinsic calibration information is adjusted based on a relationship to the cropping parameters. The graphic is dynamically registered upon the cropped image based on the adjusted intrinsic calibration information for the camera device, the monitored extrinsic information for the camera device and the monitored vehicle information.

Claims

1. Method for dynamically registering a graphic upon a cropped image obtained from a camera device, comprising:

capturing an original image obtained from the camera device;

monitoring intrinsic calibration information for the camera device, extrinsic information for the camera device and vehicle information;

generating the cropped image based on cropping parameters to exclude undesirable content from the original image;

adjusting the intrinsic calibration information based on a relationship to the cropping parameters; and

dynamically registering the graphic upon the cropped image based on the adjusted intrinsic calibration information for the camera device, the monitored extrinsic information for the camera device and the monitored vehicle information.

2. The method of claim 1 wherein the camera device comprises at least one of a rearward camera device, a frontward camera device, and a side view camera device.

3. The method of claim 2 wherein the dynamically registered graphic comprises a dynamic guideline overlay feature representing a desired path and positional information behind the vehicle.

4. The method of claim 1 wherein the cropping parameters are configured to reduce at least one of a portion of the original image along a first axis and a portion of the original image along a second axis orthogonal to the first axis.

5. The method of claim 4 wherein the first axis is associated with image width and the second axis is associated with image height.

6. The method of claim 4 wherein the cropping parameters are predetermined based on dimensions and design of a vehicle upon which the camera device is positioned.

7. The method of claim 4 wherein the cropping parameters include a user defined area of interest within the original image and exclude a user defined area outside the area of interest.

8. The method of claim 4 wherein the intrinsic calibration information for the camera device comprises focal length parameters for the camera device, and wherein adjusting the intrinsic calibration information based on the relationship to the cropping parameters comprises:

updating a first focal length parameter perpendicular to the first axis based on a mathematical relationship between the first focal length parameter, said cropping parameters configured to reduce the portion of the original image along the first axis; and

updating a second focal length parameter perpendicular to the second axis based on a mathematical relationship between the second focal length parameter, said cropping parameters configured to reduce the portion of the original image along the second axis.

9. The method of claim 8 wherein the intrinsic calibration information for the camera device further comprises optical center parameters, and wherein adjusting the intrinsic calibration information based on the relationship to the cropping parameters further comprises:

updating a first optical center parameter with respect to the first axis based on a mathematical relationship between the first optical center parameter, said cropping parameters configured to reduce the portion of the original image along the first axis; and

updating a second optical center parameter with respect to the second axis based on a mathematical relationship between the second optical center parameter, said cropping parameters configured to reduce the portion of the original image along the second axis.

10. The method of claim 4 wherein the intrinsic calibration information for the camera device comprises a plurality of pixels associated with the original image, and wherein adjusting the intrinsic calibration information based on the relationship to the cropping parameters comprises:

applying a two-dimensional grid having a third axis and a fourth axis over the original image, the fourth axis orthogonal to the third axis;

manually labeling the plurality of pixels with respect to the two-dimensional grid applied over the original image, each pixel having a respective coordinate with respect to the third and fourth axes; and

transferring the plurality of pixels associated with the original image to the cropped image, comprising

re-calibrating a location of each of the plurality of pixels in the cropped image with respect to the first axis based on a mathematical relationship between the plurality of pixels with respect to the first axis, said cropping parameters configured to reduce the portion of the original image along the first axis, and

re-calibrating a location of each of the plurality of pixels in the cropped image with respect to the second axis based on a mathematical relationship between the plurality of pixels with respect to the second axis, said cropping parameters configured to reduce the portion of the original image along the second axis.

11. The method of claim 10 wherein the third axis corresponds to a world coordinate horizontal axis and the fourth axis corresponds to a world coordinate vertical axis.

12. The method of claim 10 wherein the intrinsic calibration information further comprises focal length parameters and optical center parameters, said method further comprising:

monitoring radial distortion parameters for the camera device;

monitoring the re-calibrated plurality of pixels with respect to the third axis and the re-calibrated plurality of pixels with respect to the fourth axis;

determining expected pixel errors in the cropped image with respect to the first and second axes based on the monitored focal length parameters, the optical center parameters, the radial distortion parameters, the extrinsic information, each re-calibrated pixel location in the cropped image with respect to the first axis and each re-calibrated pixel location in the cropped image with respect to the second axis;

comparing the expected pixel errors with respect to the first axis to a first threshold;

comparing the expected pixel errors with respect to the second axis to a second threshold; and

determining a calibration failure if one of the expected pixel errors with respect to the first axis is greater than the first threshold and the expected pixel errors with respect to the second axis is greater than the second threshold.

13. The method of claim 7 further comprising:

dynamically updating the cropping parameters in response to a user input adjusting the area of interest; and

dynamically adjusting the intrinsic calibration information in response to the dynamically updated cropping parameters.

14. The method of claim 13 wherein the user input adjusting the area of interest comprises the user input adjusting a zoom range factor to one of increase and decrease the desired area of interest.

15. The method of claim 7 wherein the graphic comprises an overlay graphic representing at least one of text, symbols and shape of a display sign when the desired area of interest encompasses the display sign, said overlay graphic being dynamically registered over the display sign.

16. The method of claim 7 further comprising:

monitoring informational inputs related to the area of interest;

analyzing the informational inputs to detect a feature of interest; and

determining the graphic for dynamic registration upon the cropped image based upon the feature of interest.

17. The method of claim 16 wherein the feature of interest comprises an upcoming maneuver point and the graphic comprises a navigation status graphic representing the upcoming maneuver point.

18. The method of claim 16 wherein:

the feature of interest comprises at least one of a pedestrian, a lane marking, and a vehicle; and

the graphic comprises a highlight graphic representing the feature of interest.

19. The method of claim 1 further comprising:

displaying the cropped image and the dynamically registered graphic upon the cropped image upon a graphic projection display, said graphic projection display comprising one of a touch screen display, a human machine interface display, a rear-view mirror display, a side-view mirror display and a display monitor.

20. Method for registering a graphic upon a cropped image obtained from a camera device, comprising:

capturing an original rearward image having a point of view behind a vehicle obtained from a rearward wide field of view camera device positioned proximate to a rearward end of the vehicle;

monitoring intrinsic calibration information for the camera device comprising focal length parameters and optimal center parameters;

monitoring radial distortion parameters for the for the camera device;

monitoring extrinsic information for the camera device comprising positional parameters and orientation parameters;

monitoring vehicle information comprising vehicle speed, orientation and steering angle parameters;

generating the cropped image based on predetermined cropping parameters to exclude undesirable content from the original image, the predetermined cropping parameters configured to reduce a portion of the original rearward image;

updating the intrinsic calibration information based on a relationship to the predetermined cropping parameters comprising

updating a first focal length parameter along a first axis based on a mathematical relationship between the first focal length parameter, said predetermined cropping parameters configured to reduce the portion of the original image along the first axis, and

updating a second focal length parameter along a second axis based on a mathematical relationship between the second focal length parameter, said predetermined cropping parameters configured to reduce the portion of the original image along the second axis,

updating a first optical center parameter along the first axis based on a mathematical relationship between the first optical center parameter, said predetermined cropping parameters configured to reduce the portion of the original image along the first axis, and

updating a second optical center parameter along the second axis based on a mathematical relationship between the second optical center parameter, said predetermined cropping parameters configured to reduce the portion of the original image along the second axis;

dynamically registering the graphic upon the cropped image based on the adjusted intrinsic calibration information for the camera device, the monitored radial distortion parameters for the camera device, the monitored extrinsic information for the camera device and the monitored vehicle information; and

displaying the cropped image and the dynamically registered graphic upon the cropped image upon a graphic projection display, said graphic projection display comprising one of a touch screen display, a human machine interface display, a rear-view mirror display, a side-view mirror display and a display monitor.