System and method to adjust the field of view displayed on an electronic mirror using real-time, physical cues from the driver in a vehicle

• US 10,116,873 B1
• Filed: 11/09/2015
• Issued: 10/30/2018
• Est. Priority Date: 11/09/2015
• Status: Active Grant

• Current Assignee: Ambarella International LP (Ambarella Incorporated)
• Original Assignee: Ambarella Incorporated
• Inventors: Campbell, Jeffery
• Primary Examiner(s): Rao, Andy
• Assistant Examiner(s): Jiang, Zaihan
• Application Number:
  US14/935,688
• Time in Patent Office: 1,086 Days
• Field of Search: 348 14, 348 38
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 1/26 to the rear of the vehicle
• CPC Class Codes: B60R 1/28 with an adjustable field of view
• CPC Class Codes: B60R 1/30 providing vision in the non-visible spectrum, e.g. night or infrared vision
• CPC Class Codes: B60R 2300/70 characterised by an event-triggered choice to display a specific image among a selection of captured images
• CPC Class Codes: G02B 5/08 Mirrors vehicle mirrors involving special optical features B60R1/08
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 2207/30268 Vehicle interior
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/59 inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/18 Eye characteristics, e.g. of the iris
• CPC Class Codes: H04N 23/51 Housings
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/63 by using electronic viewfinders
• CPC Class Codes: H04N 23/69 Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
• CPC Class Codes: H04N 23/698 : for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 5/2628 : Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation

Abstract

An apparatus comprising a first sensor, a second sensor and a processor. The first sensor may be configured to generate a first video signal based on a targeted view from a vehicle. The second sensor may be configured to generate a second video signal based on a targeted view of a driver. The processor may be configured to (A) receive the first video signal, (B) receive the second video signal, (C) determine a field of view to present to the driver, (D) generate a third video signal and (E) present the third video signal to a display showing the field of view. The field of view is determined based on (i) a body position of the driver extracted from the second video signal and (ii) the first video signal.

Claims

1. An apparatus comprising:

a first sensor configured to generate a first video signal based on a targeted view from a vehicle;

a second sensor configured to generate a second video signal based on a targeted view of a driver; and

a processor configured to (A) receive said first video signal, (B) receive said second video signal, (C) determine a field of view to present to said driver, (D) generate a third video signal and (E) present said third video signal to an electronic mirror configured to show said field of view, wherein (a) said field of view is determined based on (i) a body position of said driver extracted from said second video signal by determining a distance from said second sensor and (ii) said first video signal, (b) said distance from said second sensor is based on a comparison of a number of pixels of a known object in a first video frame showing an interior of said vehicle without said driver and a second video frame of said interior of said vehicle with said driver, (c) said field of view displayed on said electronic mirror is configured to emulate a view from a reflective mirror as seen from a point of view of said driver and (d) said electronic mirror implements at least one of a rear view mirror and a side view mirror for said vehicle.

2. The apparatus according to claim 1, wherein said body position of said driver is extracted from said second video signal by said processor using video analytics based on machine learning.

3. The apparatus according to claim 1, wherein said first sensor is configured to capture video data from a perspective of said vehicle.

4. The apparatus according to claim 3, wherein said perspective of said vehicle is a view directed to the rear of said vehicle.

5. The apparatus according to claim 1, wherein said second sensor is implemented using an in-cabin camera.

6. The apparatus according to claim 1, wherein said second sensor is configured to monitor said driver by detecting at least one of a head, face and gaze of said driver.

7. The apparatus according to claim 6, wherein said monitoring of said driver is performed using data from said second sensor and one or more other sensors directed at said driver.

8. The apparatus according to claim 1, wherein said processor is configured to (i) determine whether said driver is attempting to look outside a current field of view output to said electronic mirror and (ii) adjust said field of view output to said electronic mirror, if said processor determines said driver is attempting to look outside said current field of view output to said electronic mirror.

9. The apparatus according to claim 1, wherein (i) said first sensor is implemented as part of a camera and (ii) said processor is configured to cause said camera to alter a direction of view of said camera based on said field of view of said driver.

10. The apparatus according to claim 1, wherein said (i) said first sensor is configured to capture a wide field of view, (ii) said processor is configured to transform said first video signal to fit said electronic mirror and (iii) said third video signal is a subset of said transformed first video signal.

11. The apparatus according to claim 10, wherein said processor is configured to adjust said field of view by performing at least one of a digital pan, a digital tilt and a digital zoom of said transformed first video signal.

12. The apparatus according to claim 10, wherein said transform comprises at least one of scaling, dewarping, removing visual artifacts, and cropping.

13. The apparatus according to claim 1, wherein (A) said body position of said driver is further determined based on an analysis of one or more video frames from one or more other in-cabin cameras capturing said driver, and (B) said analysis determines (a) a location of the head of said driver, (b) a relative size of the head of said driver and (c) a rotation of the head of said driver.

14. The apparatus according to claim 1, wherein said field of view displayed on said electronic mirror is configured to improve upon a view from a reflective mirror by implementing at least one of (i) amplifying a zoom level of said field of view in response to said body position of said driver, (ii) showing a high dynamic range (HDR) of luminosity, (iii) providing night vision, (iv) providing more viewing area and (v) highlighting detected objects.

15. The apparatus according to claim 1, wherein a notification is generated to indicate information about said field of view currently shown on said electronic mirror.

16. The apparatus according to claim 1, wherein said apparatus implements adjusting said field of view displayed on said electronic mirror using real-time physical cues from said driver in said vehicle.

17. The apparatus according to claim 1, wherein (i) said first sensor is implemented in a first enclosure, (ii) and said second sensor is implemented in a second enclosure and (iii) said processor is implemented in either said first enclosure or said second enclosure.

19. The apparatus according to claim 1, wherein (i) said distance is determined based on video analysis by said processor to determine location coordinates for eyes of said driver, (ii) said video analysis is used to determine with respect to said second sensor (a) a first location coordinate, (b) a second location coordinate and (c) a depth coordinate for said eyes of said driver and (iii) said depth coordinate is determined based on said comparison.

18. An apparatus comprising:

an interface configured to receive (i) data from a first video signal based on a targeted view from a vehicle captured by a first sensor and (ii) data from a second video signal based on a targeted view of a driver captured by a second sensor; and

a processor configured to (A) determine a field of view to present to said driver, (B) generate a third video signal and (C) present said third video signal to an electronic mirror configured to show said field of view, wherein (a) said field of view is determined based on (i) a body position of said driver extracted from said second video signal by determining a distance of said driver from said second sensor and (ii) said data from said first video signal, (b) said distance from said second sensor is based on a comparison of a number of pixels of a known object in a first video frame showing an interior of said vehicle without said driver and a second video frame of said interior of said vehicle with said driver, (c) said field of view displayed on said electronic mirror is configured to emulate a view from a reflective mirror as seen from a point of view of said driver and (d) said electronic mirror implements at least one of a rear view mirror and a side view mirror for said vehicle.