SYSTEM OF VEHICLES EQUIPPED WITH IMAGING EQUIPMENT FOR HIGH-DEFINITION NEAR REAL-TIME MAP GENERATION

• US 20180349715A1
• Filed: 05/31/2018
• Published: 12/06/2018
• Est. Priority Date: 05/31/2017
• Status: Active Grant

• Current Assignee: Woven By Toyota US Incorporated
• Original Assignee: Carmera, Inc. (Toyota Motor Corporation)
• Inventors: GUPTA, Ro, DAY, Justin, NICHOLL, Chase, HENSTELL, Max, STAMOS, Ioannis, BOYLE, David, SORRELGREEN, Ethan, DINH, Quynh
• Granted Patent: US 11,802,968 B2
• Time in Patent Office: Days
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 2300/207 using multi-purpose displays, e.g. camera image and navigation or video on same display
• CPC Class Codes: B60R 2300/301 combining image information with other obstacle sensor information, e.g. using RADAR/LIDAR/SONAR sensors for estimating risk of collision
• CPC Class Codes: B60R 2300/302 combining image information with GPS information or vehicle data, e.g. vehicle speed, gyro, steering angle data
• CPC Class Codes: B60R 2300/406 using wireless transmission
• CPC Class Codes: B60R 2300/50 characterised by the display information being shared, e.g. external display, data transfer to other traffic participants or centralised traffic controller
• CPC Class Codes: G01C 21/3841 Data obtained from two or more sources, e.g. probe vehicles
• CPC Class Codes: G01C 21/3885 Transmission of map data to client devices; Reception of map data by client devices
• CPC Class Codes: G01C 21/3893 Transmission of map data from distributed sources, e.g. from roadside stations
• CPC Class Codes: G01S 17/42 Simultaneous measurement of distance and other co-ordinates indirect measurement G01S17/46
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/88 Lidar systems specially adapted for specific applications
• CPC Class Codes: G01S 17/89 for mapping or imaging
• CPC Class Codes: G06T 2207/30236 Traffic on road, railway or crossing
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06V 10/17 using hand-held instruments
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/584 of vehicle lights or traffic lights
• CPC Class Codes: G08G 1/0112 from the vehicle, e.g. floating car data [FCD]
• CPC Class Codes: G08G 1/0129 for creating historical data or processing based on historical data
• CPC Class Codes: G08G 1/0133 : for classifying traffic situation
• CPC Class Codes: G08G 1/0141 : for traffic information dissemination
• CPC Class Codes: G08G 1/20 : Monitoring the location of vehicles belonging to a group, e.g. fleet of vehicles, countable or determined number of vehicles
• CPC Class Codes: H04W 4/44 : for communication between vehicles and infrastructures, e.g. vehicle-to-cloud [V2C] or vehicle-to-home [V2H]

Abstract

Described are street level intelligence platforms, systems, and methods that can include a fleet of swarm vehicles having imaging devices. Images captured by the imaging devices can be used to produce and/or be integrated into maps of the area to produce high-definition maps in near real-time. Such maps may provide enhanced street level intelligence useful for fleet management, navigation, traffic monitoring, and/or so forth.

Claims

1. A non-transitory processor readable medium storing code configured to be executed by a processor, the code comprising code to cause the processor to:

capture, from a camera of a smartphone disposed with a vehicle, a video of a streetscape;

perform a first pass of computer vision analysis on the video of the streetscape to identify a plurality of candidate high-priority events;

perform a second pass of computer vision analysis on the plurality of candidates of high-priority events to identify a high-priority event, the second pass of computer vision analysis consuming more computational resources than the first pass of computer vision analysis such that the second pass of computer vision analysis cannot be performed on the video in real time; and

transmit, over a wireless data network and to a remote analysis service, an indication of the high-priority event, such that the remote analysis service can integrate the high-priority event into a map of the streetscape, the remote analysis service consuming more computational resources to integrate the high-priority event into the map than are available at the vehicle.

2. The non-transitory processor readable medium of claim 1, the code further comprising code to cause the processor to:

receive, from the remote analysis service, the map of the streetscape including the high-priority event.

3. The non-transitory processor readable medium of claim 1, wherein the code to cause the processor to perform the first pass of computer vision analysis includes code to cause the first pass of computer vision analysis to be performed in real time on the video of the streetscape.

4. The non-transitory processor readable medium of claim 1, wherein the processor is located within a housing of the smartphone.

5. The non-transitory processor readable medium of claim 1, the code further comprising code to cause the processor to:

detect that the vehicle has been turned on;

boot an operating system of the smartphone based on detecting that the vehicle has been turned on;

configure the smartphone such that a call functionality is disabled; and

detect that the vehicle has left a home base, the video of the streetscape captured triggered by the detection that the vehicle has left the home base.

6. The non-transitory processor readable medium of claim 1, the code further comprising code to cause the processor to:

detect that the vehicle has been turned on;

boot an operating system of the smartphone based on detecting that the vehicle has been turned on;

configure the smartphone such that a call functionality is disabled; and

configure the smartphone such that the smartphone periodically contacts the remote analysis service for instructions.

7. The non-transitory processor readable medium of claim 1, the code further comprising code to cause the processor to:

detect that the vehicle has returned to a home base;

cease capture of video based on detecting that the vehicle has returned to the home base;

connect to a WiFi network based on detecting that the vehicle has returned to the home base; and

upload video captured from the camera to the remote analysis service via the WiFi network.

8. The non-transitory processor readable medium of claim 1, the code further comprising code to cause the processor to:

detect that the vehicle has returned to a home base;

connect to a WiFi network based on detecting that the vehicle has returned to the home base;

uploading video captured from the camera to the remote analysis service via the WiFi network;

detect that the vehicle has been turned off;

enter a power saving state after video captured from the camera has been uploaded to the remote analysis service, based on detecting that the vehicle has been turned off.

10. The non-transitory processor readable medium of claim 1, the code further comprising code to cause the processor to:

capture a still image from the camera;

divide the still image into a plurality of regions;

calculate a Laplacian variance for at least a subset of regions from the plurality of regions; and

compare the Laplacian variances to a predetermined threshold and store the still image if Laplacian variance for a majority of the subset of regions is less than the predetermined threshold or discard the still image if at least one Laplacian variance calculated for at least one region from the plurality of regions is greater than the predetermined threshold.

11. The non-transitory processor readable medium of claim 1, wherein performing the first pass of computer vision analysis includes scanning the video in real time for orange features to identify candidate traffic cones.

12. The non-transitory processor readable medium of claim 1, wherein:

performing the first pass of computer vision analysis includes scanning the video in real time for orange features to identify candidate traffic cones; and

performing the second pass of computer vision analysis includes confirming that the candidate traffic cones are actual traffic cones.

13. The non-transitory processor readable medium of claim 1, wherein:

performing the first pass of computer vision analysis includes scanning the video in real time for orange features to identify candidate traffic cones;

performing the second pass of computer vision analysis includes confirming that the candidate traffic cones are actual traffic cones; and

the map of the streetscape includes an indication of a lane closure corresponding to a presence of the actual traffic cones.

14. The non-transitory processor readable medium of claim 1, wherein:

performing the first pass of computer vision analysis includes scanning the video in real time for flashing lights to identify candidate emergency vehicles; and

performing the second pass of computer vision analysis includes identifying stationary emergency vehicles.

15. A system, comprising:

a plurality of vehicle-mounted smartphones including a cellular data radio and a WiFi radio, each vehicle-mounted smartphone from the plurality of vehicle-mounted smartphones configured to;

continuously capture video of streetscapes at a rate that exceeds average daily bandwidth of a network accessible via the cellular data radio for that vehicle-mounted smartphone,

locally store at least eight hours of video,

identify a high-priority feature in the video of the streetscape and send a portion of the video containing the high-priority feature to a video-and-map-integration device via that vehicle-mounted smartphone'"'"'s cellular data radio, and

connect to a WiFi network when parked at a home location and to transfer locally stored video to the video-and-map-integration device via that vehicle-mounted smartphones'"'"' WiFi radio; and

the video-and-map-integration device configured to;

integrate high-priority features into a map,

send a signal representing the map containing the high-priority features to the plurality of vehicle-mounted smartphones over a cellular data network,

integrate video received from the plurality of vehicle-mounted smartphones via WiFi radios into the map such that the map is updated at least daily, and

send a signal representing the map containing content from the video received via WiFi radios to at least one of (1) a map-viewer device, (2) a vehicle-mounted smartphone from the plurality of vehicle-mounted smartphones, or (3) a navigation device.

16. The system of claim 15, wherein the video received from the plurality of vehicle-mounted smartphones via WiFi radios has a resolution of at least 1280×

720 at at least 30 frames per second.

17. The system of claim 15, wherein each vehicle-mounted smartphone from the plurality of vehicle-mounted smartphones is communicatively coupled to a vehicle-mounted LiDAR device, and each vehicle mounted smartphone from the plurality of vehicle-mounted smartphones is configured to associate data from the LiDAR device with recorded video.

18. The system of claim 15, wherein each vehicle-mounted smartphone from the plurality of vehicle-mounted is configured to associate recorded video with Global Positioning System (GPS) coordinates received from a GPS receiver.

19. The system of claim 15, wherein the video and map integration device is configured to:

receive, from a first vehicle-mounted smartphone mounted to a first vehicle, an indication of a high-priority feature at a location, the first vehicle-mounted smartphone being from the plurality of vehicle-mounted smartphones;

poll the plurality of vehicle-mounted smartphones to identify a second vehicle-mounted smartphone mounted to a second vehicle that passed the location within a predetermined time of the first vehicle passing the location;

receive, from the second vehicle-mounted smartphone, video of the location; and

determine a time when the high-priority feature appeared based on the video of the location received from the second vehicle-mounted smartphone and the indication of the high-priority feature received from the first vehicle-mounted smartphone.

20. The system of claim 15, wherein the predetermined time is less than 48 hours.

21. A method, comprising:

receiving, from a plurality of vehicle-mounted smartphones, video depicting a plurality of street segments;

bucketize the video such that each bucket is associated with a video segment capturing one street segment from the plurality of street segments by one vehicle-mounted smartphone from the plurality of vehicle-mounted smartphones;

identify (i) a selected street segment and (ii) a plurality of street segments adjacent to the selected street segment from the plurality of street segments, no video of the selected street segment received from any of the plurality of vehicle-mounted smartphones during a time interval;

for each of the plurality of street segments adjacent to the selected street segments, perform image recognition on each bucket associated with that street segment adjacent to the selected street segment and the time interval to calculate a pedestrian count; and

estimate a pedestrian count for the selected street based on an average of the pedestrian counts for the time interval for each of the plurality of street segments adjacent to the selected street segment.

22. The method of claim 21, further comprising:

normalize the pedestrian count for the time interval and for each of the plurality of street segments adjacent to the selected street segment based on the number of video frames and the number of vehicle-mounted smartphones that captured video of that street segment adjacent to the selected street segment during the time interval, the estimate of the pedestrian count for the selected street based on an average of the normalized pedestrian counts for the time interval for each of the plurality of street segments adjacent to the selected street segment.

23. The method of claim 21, wherein video depicting a particular street segment from the plurality of street segments is received irregularly.