Automatic image content analysis method and system

• US 9,443,152 B2
• Filed: 05/03/2012
• Issued: 09/13/2016
• Est. Priority Date: 05/03/2011
• Status: Active Grant

• Current Assignee: iOnRoad Technologies Limited (Samsung Electronics Co. Ltd.)
• Original Assignee: iOnRoad Technologies Limited (Samsung Electronics Co. Ltd.)
• Inventors: Atsmon, Alon, Atsmon, Dan, Dayan, Ram, Bass, Hilla
• Primary Examiner(s): Perungavoor, Sath V
• Assistant Examiner(s): AYNALEM, NATHNAEL B
• Application Number:
  US14/115,302
• Publication Number:
  US 20140111647A1
• Time in Patent Office: 1,594 Days
• Field of Search: 348/148
• US Class Current: 1/1
• CPC Class Codes: B60R 11/04 Mounting of cameras operative during drive; Arrangement of controls thereof relative to the vehicle
• CPC Class Codes: G01C 21/34 Route searching; Route guidance
• CPC Class Codes: G01C 21/362 received from an external device or application, e.g. PDA, mobile phone or calendar application
• CPC Class Codes: G01C 21/3647 Guidance involving output of stored or live camera images or video streams
• CPC Class Codes: G01C 21/3697 Output of additional, non-guidance related information, e.g. low fuel level G01C21/3679 takes precedence
• CPC Class Codes: G06F 16/29 Geographical information databases
• CPC Class Codes: G06Q 30/0261 based on user location
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/582 of traffic signs
• CPC Class Codes: G06V 20/584 of vehicle lights or traffic lights
• CPC Class Codes: G06V 20/586 of parking space
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
• CPC Class Codes: G06V 20/59 inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G08G 1/005 including pedestrian guidance indicator
• CPC Class Codes: G08G 1/0112 from the vehicle, e.g. floating car data [FCD]
• CPC Class Codes: G08G 1/0133 for classifying traffic situation
• CPC Class Codes: G08G 1/0141 for traffic information dissemination
• CPC Class Codes: G08G 1/0175 by photographing vehicles, e.g. when violating traffic rules
• CPC Class Codes: G08G 1/04 using optical or ultrasonic detectors
• CPC Class Codes: G08G 1/09623 : Systems involving the acquisition of information from passive traffic signs by means mounted on the vehicle G08G1/0967 takes precedence
• CPC Class Codes: G08G 1/09626 : where the origin of the information is within the own vehicle, e.g. a local storage device, digital map
• CPC Class Codes: G08G 1/096716 : where the received information does not generate an automatic action on the vehicle control
• CPC Class Codes: G08G 1/09675 : where a selection from the received information takes place in the vehicle
• CPC Class Codes: G08G 1/096775 : where the origin of the information is a central station
• CPC Class Codes: G08G 1/14 : indicating individual free spaces in parking areas
• CPC Class Codes: G08G 1/143 : inside the vehicles
• CPC Class Codes: G08G 1/144 : on portable or mobile units, e.g. personal digital assistant [PDA]
• CPC Class Codes: G08G 1/147 : where the parking area is within an open public zone, e.g. city centre
• CPC Class Codes: G08G 1/161 : Decentralised systems, e.g. inter-vehicle communication
• CPC Class Codes: G08G 1/166 : for active traffic, e.g. moving vehicles, pedestrians, bikes
• CPC Class Codes: G08G 1/167 : Driving aids for lane monitoring, lane changing, e.g. blind spot detection
• CPC Class Codes: G08G 1/202 : Dispatching vehicles on the basis of a location, e.g. taxi dispatching
• CPC Class Codes: G08G 1/205 : Indicating the location of the monitored vehicles as destination, e.g. accidents, stolen, rental
• CPC Class Codes: H04N 21/2146 : located in mass transportation means, e.g. aircraft, train or bus flight-deck installations for entertainment or communications B64D11/0015; arrangements specially adapted for transportation systems in broadcast systems H04H20/62; moving wireless networks H04W84/005
• CPC Class Codes: H04N 23/51 : Housings
• CPC Class Codes: H04N 7/181 : for receiving images from a plurality of remote sources
• CPC Class Codes: H04N 7/185 : from a mobile camera, e.g. for remote control

Abstract

A system and method for analyzing and wirelessly transmitting information in real-time, comprising capturing snapshots and videos, by utilizing for example a smartphone. The images are analyzed in conjunction with other system data to provide the user: 3 dimensional images, GPS coordinates, safety alerts, etc. In particular, the system is mountable within a vehicle interior to provide: road conditions, dangerous driving situations, the make/model/manufacturer of surrounding cars, GPS coordinates, and the identity of local businesses. The user may also share the information via wireless transmissions with surrounding users possessing the same system, such as for real-time video gaming systems. And the system may also be detached from the vehicle for use in other scenarios, such as searching a merchant database for matching items and wirelessly sending the information to other system users.

Claims

1. A handheld device comprising,

at least one camera;

a display;

at least two sensors;

a housing comprising said at least one camera, said housing being sized and shaped for mounting and detachment within an interior of a vehicle passenger compartment;

wherein said at least one camera and a first sensor of said at least two sensors simultaneously record a first plurality of images of an external environment outside of a vehicle and at least one signal of an internal passenger compartment environment inside of the vehicle;

wherein said first plurality of images comprises at least one of a plurality of snapshots and a plurality of videos;

a computer readable medium containing computer executable instructions; and

a processor adapted to execute said computer executable instructions for;

identifying a deviation from a predefined driving direction by an analysis of said first plurality of images;

identifying a signal recorded in said vehicle internal passenger compartment by a second sensor of said at least two sensors after said identifying said deviation;

analyzing said deviation and said signal to identify a reaction of a driver of said vehicle to said deviation in real time; and

instructing said display to present an alert according to said analyzing.

2. The handheld device of claim 1, further comprising a database to store said first plurality of images, said at least one signal and results of said analyzing, wherein the database is located on the device or on a remote system server.

3. The handheld device of claim 1, wherein said display plays audio results of said analyzing.

4. The handheld device of claim 1, wherein said computer readable medium is an application executed on said handheld device.

5. The handheld device of claim 1, wherein said instructions comprise instructions for graphically displaying driving instructions projected inside said vehicle, wherein said driving instructions depict street lane markings and directional arrows.

6. The handheld device of claim 5, further comprising said driving instructions depicting businesses in proximity to said street lane markings;

wherein said display comprises means for a user to indicate their preference for a business.

7. The handheld device of claim 6, wherein said display presents multiple users'"'"' preferences on a map.

8. The handheld device of claim 1, further comprising a module which detects when said housing is mounted within said vehicle;

wherein said at least one camera and said first sensor are instructed to perform said simultaneous recording in response to said detecting.

9. The handheld device of claim 1, wherein said computer readable medium comprises instructions for generating a report comprising results of said analyzing, wherein said generating comprises analyzing said first plurality of images and said at least one signal to calculate parameters selected from a group consisting of:

drive times, stop times, safety points, crash alerts, headway alerts, and lane departures;

generating said report according to said parameters; and

presenting said report using said display;

wherein said report comprises;

a color coded map marking a driving route and information extracted from said parameters.

10. The handheld device of claim 4, wherein said computer readable medium comprises:

instructions for analyzing said first plurality of images and said at least one signal to determine when said vehicle has parked;

instructions for recording at least one of said first plurality of images to image the vehicle'"'"'s parking location; and

instructions for displaying said at least one of said first plurality of images on said handheld device.

11. The handheld device of claim 1, wherein said analysis of said first plurality of images determines when at least one dangerous event for said vehicle occurs;

wherein said at least one dangerous event is a member of a group consisting of at least collision, maintaining distance, lane deviation, incorrect in-lane location, and potential hazards.

12. The handheld device of claim 11, wherein said display presents an alert in response to a detection of said at least one dangerous event.

13. The handheld device of claim 1, wherein said processor applies said analyzing on both said first plurality of images and said at least one signal to detect driving;

wherein said computer readable medium contains computer executable instructions to direct incoming phone calls based on said driving detection.

14. The handheld device of claim 1, wherein said computer readable medium contains computer executable instructions for:

identifying a driver'"'"'s needs based on driver manual input, real-time in-vehicle data, and automatic pattern recognition data;

recognizing the driver'"'"'s geographic location;

determining a plurality of local businesses within a pre-designated range matching said driver'"'"'s needs, wherein locations are derived from a visual database and/or a geographic database; and

notifying the driver of the match between said plurality of local businesses and the driver'"'"'s needs.

15. The handheld device of claim 14, wherein said identifying comprises acquiring real-time vehicle fluid level and calculating said driver'"'"'s needs according to said vehicle fluid;

wherein said plurality of businesses comprises a plurality of gas stations.

16. The handheld device of claim 14, wherein said recognizing comprises identifying a routine driving stop of said vehicle, and notifying the driver about said routine driving stop.

17. The handheld device of claim 14, wherein said recognizing comprises notifying the driver of a possible purchase from a local business.

18. The handheld device of claim 1, wherein said first plurality of images is captured to image multiple viewpoints around a vehicle exterior of said vehicle.

19. The handheld device of claim 1, wherein said analysis of said first plurality of images comprises an analysis to extract information about vehicles around the driver based on image processing.

20. The handheld device of claim 1, further comprising a unit which wirelessly transmits results of said analyzing to surrounding vehicles.

21. The handheld device of claim 20, wherein said analyzing comprises real-time image, audio, and video recognition, wherein:

said image recognition comprises a recognition of a member selected from a group consisting of faces, traveling routes and places, road signs, surrounding vehicles, objects and text;

said audio recognition comprises a recognition of a member selected from a group consisting of voices, sounds, and songs; and

,

said video recognition comprises a recognition of a member selected from a group consisting of music channels, movies, and television series.

22. The handheld device of claim 1, wherein said analyzing comprises a calculation of a normalized driving scoring of said driver.

23. The handheld device of claim 1, wherein said deviation is from a middle of a lane, wherein said second sensor is a microphone, and wherein said signal is an audio signal of an activated turn blinker signaling a turn.

24. A computerized image processing method, comprising,

using at least one camera and at least a first sensor to record simultaneously a first plurality of images of an external environment outside of a vehicle and at least one signal of an internal passenger compartment environment inside of said vehicle;

identifying, using a processor, a deviation from a predefined driving direction by an analysis of;

identifying, using said processor, a signal recorded in said vehicle internal passenger compartment by at least a second, different sensor after said identifying said deviation;

analyzing, using said processor, said deviation and said signal to identify a reaction of a driver of said vehicle to said deviation in real time; and

,

instructing a display to present an alert according to said analyzing.