Systems and methods for driver assistance

• US 10,614,617 B2
• Filed: 04/18/2018
• Issued: 04/07/2020
• Est. Priority Date: 04/21/2017
• Status: Active Grant

• Current Assignee: Harman International Industries Incorporated (Samsung Electronics Co. Ltd.)
• Original Assignee: Harman International Industries Incorporated (Samsung Electronics Co. Ltd.)
• Inventors: Fortmann, Gustavo Adolfo
• Primary Examiner(s): Zalalee, Sultana M
• Application Number:
  US15/956,644
• Publication Number:
  US 20180308275A1
• Time in Patent Office: 720 Days
• CPC Class Codes: G05D 1/0094 involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/586 of parking space

Abstract

Embodiments are disclosed for generating a virtual view that is at least partially decoupled from a view of one or more physical cameras. An example method includes capturing images from at least one physical camera, generating a virtual image of a virtual camera from the captured images of at least one physical camera, the virtual image having either a different viewpoint, a different orientation, or a different viewpoint and orientation than the images captured by at least one physical camera, and automatically maintaining the virtual position of the virtual camera, relative to a reference point, when the at least one physical camera moves relative to the reference point.

Claims

1. A method, comprising:

capturing images from at least one physical camera coupled to a movable vehicle;

generating a virtual image of a virtual camera from the captured images from the at least one physical camera, the virtual image having either a different viewpoint, a different orientation, or a different viewpoint and orientation than the images captured from the at least one physical camera, the virtual image displayed on a display distinct from a windshield of the vehicle; and

automatically maintaining a virtual position of the virtual camera, relative to a reference point, when the at least one physical camera moves relative to the reference point, while displaying the virtual image on the display, wherein the virtual image further includes a reference line between reference points determined to provide a guide for maintaining an orientation of the virtual image, the virtual image maintaining the reference line in a parallel position to itself across each frame of the virtual image.

2. The method of claim 1, wherein automatically maintaining the virtual position of the virtual camera relative to the reference point is based on movement of the at least one physical camera relative to the reference point.

3. The method of claim 1, wherein an object or a part of the object perceived by the at least one physical camera is used in order to maintain the virtual position of the virtual camera relative to the reference point, wherein the at least one physical camera is fixedly mounted to the vehicle, and wherein maintaining the virtual position of the virtual camera further comprises maintaining an orientation of the virtual camera.

4. The method of claim 1, wherein automatically maintaining the virtual position of the virtual camera relative to the reference point is performed during a parallel park operation.

5. The method of claim 1, wherein automatically maintaining the virtual position of the virtual camera relative to the reference point, when the at least one physical camera moves relative to the reference point, comprises automatically maintaining the virtual position of the virtual camera fixed in 3-D grounded space relative to earth while the at least one physical camera and the vehicle move together relative to the earth.

6. The method of claim 1, further comprising detecting a trigger for decoupling the virtual camera from a vehicle-locked position, and generating the virtual image responsive to detecting the trigger.

7. The method of claim 6, wherein the trigger is a first trigger, the method further comprising detecting a second trigger, and, responsive to detecting the second trigger, returning the virtual camera to the vehicle-locked position and displaying a virtual view of the vehicle from a virtual perspective of the virtual camera in the vehicle-locked position.

8. The method of claim 7, wherein the second trigger comprises determining that a world-locked position is no longer able to be generated from the images captured from the at least one physical camera due to movement of the vehicle away from the world-locked position.

9. The method of claim 7, wherein the second trigger comprises determining that a speed of the vehicle exceeds a threshold.

10. The method of claim 6, wherein the trigger includes detecting an object within a vicinity of the vehicle.

11. The method of claim 6, wherein the trigger includes detecting an upcoming turn maneuver in a direction of travel of the vehicle.

12. The method of claim 6, wherein the trigger includes detecting a change in movement of the vehicle in a selected degree of freedom that is above a threshold.

13. The method of claim 6, wherein the trigger comprises user input.

14. A virtual view system comprising:

at least one physical camera positioned on a vehicle;

a display device;

an image processing unit coupled to the at least one physical camera; and

a storage device storing instructions executable by the image processing unit to;

capture, via the at least one physical camera, image data from an associated physical perspective relative to the vehicle;

generate, via the image processing unit using the image data from the at least one physical camera, a view of the vehicle from a virtual position of a virtual camera, the virtual camera capturing the view from a virtual perspective that extends from the virtual position in a direction of interest, and the virtual camera being decoupled from the at least one physical camera under a first condition;

generate, via the image processing unit using the image data from the at least one physical camera, a reference line between reference points determined to provide a guide for maintaining an orientation of the view from the virtual perspective, the view maintaining the reference line in a parallel position to itself across each frame of the view, and

display, via the display device, the view from the virtual perspective, the view appearing static as the at least one physical camera moves with the vehicle while under the first condition,

wherein the view from the virtual perspective further includes a reference line between reference points determined to provide a guide for maintaining an orientation of the view from the virtual perspective, the view from the virtual perspective maintaining the reference line in a parallel position to itself across each frame of the view from the virtual perspective, and

wherein the virtual camera is coupled to the at least one physical camera under a second condition, the view appearing to move with the vehicle under the second condition.

15. The virtual view system of claim 14, wherein the instructions further include instructions to detect a trigger for decoupling the virtual camera from the at least one physical camera in the first condition and switching to the second condition, wherein the trigger is a first trigger, the method further comprising detecting a second trigger, and, responsive to detecting the second trigger, returning the virtual camera to be coupled to the at least one physical camera, wherein the second trigger comprises determining movement of the vehicle away from a world-locked position.

16. The virtual view system of claim 14, wherein the at least one physical camera is a first physical camera of a plurality of physical cameras positioned on the vehicle, each physical camera of the plurality of physical cameras having a field of view that at least partially overlaps with another physical camera of the plurality of physical cameras.

17. The virtual view system of claim 14, wherein the display device is integrated or positioned in the vehicle.

18. The virtual view system of claim 14, wherein the display device is remote from the vehicle.

19. A virtual view system comprising:

a physical camera mounted on an object;

a display device coupled below a windshield;

a processor; and

a storage device storing instructions executable by the processor to;

capture, via the physical camera, image data from a plurality of perspectives relative to the object;

receive, from an image processing unit using the image data from the physical camera, a virtual view of the object from a virtual perspective of a virtual camera, the virtual camera having a virtual position from which the virtual perspective is captured, the virtual position remaining static relative to movement of the physical camera under a first condition and the virtual position moving in a coordinated manner with the plurality of physical cameras under a second condition;

identify, via the processor, reference points in the image data relative to the object, wherein the reference points are relevant to a detected driving maneuver;

calculate, via the processor, a reference line between the identified reference points determined to provide a guide for maintaining an orientation of the virtual view,

display, via the display device, the virtual view of the object, the virtual view appearing static in at least a first degree of freedom under the first condition and the virtual view appearing to move in at least the first degree of freedom with the object under the second condition, and

display within the virtual view of the object, the reference line, wherein the virtual view maintains the reference line in a parallel position to itself across each frame of the virtual view.

20. The virtual view system of claim 19, wherein the image processing unit is included in a cloud computing system and wherein the virtual view is generated remotely from the virtual view system.

21. The virtual view system of claim 19, wherein the image processing unit is included in the object and wherein the virtual view is generated locally to the virtual view system.

22. The virtual view system of claim 19, wherein the object is a first part of a vehicle, said first part able to move independently from at least a second part of the vehicle, wherein the physical camera is mounted on the first part of the vehicle and the virtual camera remains static with regard to the second part of the vehicle.