Apparatuses, systems and methods for acquiring images of occupants inside a vehicle

• US 9,886,637 B1
• Filed: 01/13/2016
• Issued: 02/06/2018
• Est. Priority Date: 01/13/2015
• Status: Active Grant

• Current Assignee: State Farm Mutual Automobile Insurance Company (State Farm)
• Original Assignee: State Farm Mutual Automobile Insurance Company (State Farm)
• Inventors: Chan, Aaron Scott, Sanchez, Kenneth Jason
• Primary Examiner(s): Gilliard, Delomia L
• Application Number:
  US14/994,299
• Time in Patent Office: 755 Days
• Field of Search: None
• CPC Class Codes: B60R 21/01542 detecting passenger motion
• CPC Class Codes: B60R 21/01552 detecting position of specific human body parts, e.g. face, eyes or hands
• CPC Class Codes: B60R 2300/30 characterised by the type of image processing
• CPC Class Codes: B60R 2300/8006 for monitoring and displaying scenes of vehicle interior, e.g. for monitoring passengers or cargo
• CPC Class Codes: B60W 2040/0872 Driver physiology
• CPC Class Codes: B60W 2050/0095 Automatic control mode change
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2420/54 Audio sensitive means, e.g. ultrasound
• CPC Class Codes: B60W 2540/22 Psychological state; Stress level or workload
• CPC Class Codes: B60W 2540/223 Posture, e.g. hand, foot, or seat position, turned or inclined
• CPC Class Codes: B60W 2540/225 Direction of gaze
• CPC Class Codes: B60W 2540/229 Attention level, e.g. attentive to driving, reading or sleeping
• CPC Class Codes: B60W 2540/26 Incapacity
• CPC Class Codes: B60W 40/08 related to drivers or passengers
• CPC Class Codes: B60W 40/09 Driving style or behaviour
• CPC Class Codes: B60W 50/08 Interaction between the driver and the control system
• CPC Class Codes: B60W 50/14 Means for informing the driver, warning the driver or prompting a driver intervention
• CPC Class Codes: B60W 60/0053 from vehicle to occupant
• CPC Class Codes: B60W 60/0059 Estimation of the risk associated with autonomous or manual driving, e.g. situation too complex, sensor failure or driver incapacity
• CPC Class Codes: B60Y 2400/30 Sensors
• CPC Class Codes: B60Y 2400/3015 : Optical cameras
• CPC Class Codes: B60Y 2400/3017 : Radars
• CPC Class Codes: G01S 19/42 : Determining position
• CPC Class Codes: G05D 1/0061 : for transition from automatic pilot to manual pilot and vice versa
• CPC Class Codes: G06F 16/51 : Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 18/22 : Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 18/24 : Classification techniques
• CPC Class Codes: G06F 18/2413 : based on distances to training or reference patterns
• CPC Class Codes: G06T 15/005 : General purpose rendering architectures
• CPC Class Codes: G06T 2207/30196 : Human being; Person
• CPC Class Codes: G06T 2207/30268 : Vehicle interior
• CPC Class Codes: G06T 7/337 : involving reference images or patches
• CPC Class Codes: G06T 7/70 : Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/73 : using feature-based methods
• CPC Class Codes: G06V 20/59 : inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
• CPC Class Codes: G06V 20/593 : Recognising seat occupancy
• CPC Class Codes: G06V 20/597 : Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: G06V 20/64 : Three-dimensional objects
• CPC Class Codes: G06V 40/10 : Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/11 : Hand-related biometrics; Hand pose recognition
• CPC Class Codes: H04N 19/17 : the unit being an image region, e.g. an object
• CPC Class Codes: H04N 19/423 : characterised by memory arrangements H04N19/433 takes precedence
• CPC Class Codes: H04N 23/45 : for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images

Abstract

The present disclosure is directed to apparatuses, systems and methods for acquiring images of occupants inside a vehicle. More particularly, the present disclosure is directed to apparatuses, systems and methods for acquiring images of occupants inside a vehicle using a vehicle in-cabin device that automatically determines a location of the device inside the vehicle.

Claims

1. A vehicle in-cabin device for generating data representative of at least one skeletal diagram of at least one occupant within an associated vehicle, the vehicle in-cabin device comprising:

a processor and a memory;

at least one sensor for generating vehicle interior data, wherein the vehicle interior data is representative of a location of the vehicle in-cabin device within the vehicle;

a vehicle in-cabin device location determination module stored on the memory that, when executed by the processor, causes the processor to generate vehicle in-cabin device location data based on the vehicle interior data, wherein the vehicle in-cabin device location data is representative of a location and orientation of the vehicle in-cabin device within the vehicle interior; and

an adjustment algorithm stored on the memory that, when executed by the processor, causes the processor to adjust real-time images based on the vehicle in-cabin device location data such that the real-time images are correlated with previously classified image data.

2. The vehicle in-cabin device as in claim 1, wherein the at least one sensor is selected from:

at least one digital image sensor, at least one ultra-sonic sensor, at least one radar-sensor, at least one infrared light sensor, or at least one laser light sensor.

3. The vehicle in-cabin vehicle device as in claim 1, wherein the vehicle interior data is representative of a three-dimensional location of the vehicle in-cabin device within the vehicle interior.

4. The vehicle in-cabin vehicle device as in claim 1, wherein the vehicle interior data is representative of an orientation of the vehicle in-cabin device selected from:

a vehicle in-cabin device tilt value, a vehicle in-cabin device pan value, or a vehicle in-cabin device roll value.

5. The vehicle in-cabin device as in claim 1, wherein the vehicle interior data is representative of a three-dimensional location of the vehicle in-cabin device within the vehicle interior and an orientation of the vehicle in-cabin device selected from:

a vehicle in-cabin device tilt value, a vehicle in-cabin device pan value, or a vehicle in-cabin device roll value.

6. The vehicle in-cabin device as in claim 1, further comprising:

a compass for generating compass data, wherein the compass data is representative of an orientation of the vehicle in-cabin device within the vehicle interior.

7. The vehicle in-cabin device as in claim 1, wherein the at least one sensor includes a first digital image sensor and a second digital image sensor, and the vehicle interior data is representative of a three-dimensional location of the vehicle in-cabin device within the vehicle interior.

8. A computer-implemented method for automatically registering a location of a vehicle in-cabin device within an interior of a vehicle, the method comprising:

positioning the vehicle in-cabin device within the interior of the vehicle;

receiving, at a processor of a computing device, vehicle in-cabin device registration location data from a sensor of the vehicle in-cabin device, wherein the vehicle in-cabin device registration location data is representative of at least one of;

a two-dimensional location of the vehicle in-cabin device within the vehicle or a three-dimensional location of the vehicle in-cabin device within the vehicle and at least one of;

a vehicle in-cabin device tilt value, a vehicle in-cabin device pan value, or a vehicle in-cabin device roll value; and

adjusting real-time images based on the vehicle in-cabin device registration location data such that the real-time images are correlated with previously classified image data.

9. The method of claim 8, further comprising:

receiving, at the processor of the computing device, vehicle in-cabin device reference data, wherein the vehicle in-cabin device reference data is representative of known vehicle in-cabin device locations and orientations.

10. The method of claim 9, further comprising:

comparing, using the processor of the computing device, the vehicle in-cabin device registration location data and the vehicle in-cabin device reference data.

11. The method of claim 8, further comprising:

receiving, at a processor of the computing device, vehicle in-cabin device reference image data, wherein the vehicle in-cabin device reference data is representative of known vehicle in-cabin device locations and orientations.

12. The method of claim 8, further comprising:

extracting features from vehicle in-cabin device reference data, wherein the features are representative of items within the vehicle.

13. The method of claim 12, wherein the vehicle in-cabin device registration location data is based on the features.

14. A non-transitory computer-readable medium storing computer-readable instruction that, when executed by a processor, cause the processor to automatically register a location of a vehicle in-cabin device within an interior of a vehicle, the non-transitory computer-readable medium comprising:

a vehicle in-cabin device registration location data receiving module that, when executed by the processor, causes the processor to receive vehicle in-cabin device registration location data from a sensor of the vehicle in-cabin device, wherein the vehicle in-cabin device registration location data is representative of at least one of;

a two-dimensional location of the vehicle in-cabin device within the vehicle or a three-dimensional location of the vehicle in-cabin device within the vehicle and at least one of;

a vehicle in-cabin device tilt value, a vehicle in-cabin device pan value, or a vehicle in-cabin device roll value; and

an adjustment algorithm that, when executed by the processor, causes the processor to adjust real-time images based on the vehicle in-cabin device registration location data such that the real-time images are correlated with previously classified image data.

15. The non-transitory computer-readable medium of claim 14, further comprising:

a vehicle in-cabin device reference data receiving module that, when executed by a processor, causes the processor to receive vehicle in-cabin device reference data, wherein the vehicle in-cabin device reference data is representative of known vehicle in-cabin device locations and orientations.

16. The non-transitory computer-readable medium of claim 15, further comprising:

a comparing module that, when executed by a processor, causes the processor to compare the vehicle in-cabin device registration location data and the vehicle in-cabin device reference data.

17. The non-transitory computer-readable medium of claim 14, further comprising:

a vehicle in-cabin device reference image data receiving module that, when executed by the processor, causes the processor to receive vehicle in-cabin device reference image data, wherein the vehicle in-cabin device reference data is representative of known vehicle in-cabin device locations and orientations.

18. The non-transitory computer-readable medium of claim 14, further comprising:

a feature extracting module that, when executed by the processor, causes the processor to extract features from vehicle in-cabin device reference data, wherein the features are representative of items within the vehicle.

19. The non-transitory computer-readable medium of claim 18, wherein the vehicle in-cabin device registration location data is based on the features.

20. The non-transitory computer-readable medium of claim 19, wherein the vehicle in-cabin device reference image data is representative of known features.