System and method for providing content in autonomous vehicles based on perception dynamically determined at real-time

• US 10,511,878 B2
• Filed: 05/30/2018
• Issued: 12/17/2019
• Est. Priority Date: 06/30/2016
• Status: Active Grant

• Current Assignee: Baidu USA LLC (Baidu Incorporated)
• Original Assignee: Baidu USA LLC (Baidu Incorporated)
• Inventors: Li, Liyun, Liu, Shaoshan, Wu, Shuang, Peng, James, Wang, Jingao
• Primary Examiner(s): Chokshi, Pinkal R
• Application Number:
  US15/993,123
• Publication Number:
  US 20180278982A1
• Time in Patent Office: 566 Days
• CPC Class Codes: B60R 11/04 Mounting of cameras operative during drive; Arrangement of controls thereof relative to the vehicle
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: H04L 67/12 specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04N 21/251 Learning process for intelligent management, e.g. learning user preferences for recommending movies details of learning user preferences for the retrieval of video data in a video database G06F16/739; computer systems using learning methods G06N3/08
• CPC Class Codes: H04N 21/2668 Creating a channel for a dedicated end-user group, e.g. insertion of targeted commercials based on end-user profiles information retrieval from the Internet by querying with filtering and personalisation G06F16/9535; arrangements for replacing or switching information during the broadcast H04H20/10; push services over packet-switching network H04L12/1859; adaptation of message content in packet-switching networks H04L51/063
• CPC Class Codes: H04N 21/41422 located in transportation means, e.g. personal vehicle arrangements specially adapted for transportation systems in broadcast systems H04H20/62
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/44218 Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45
• CPC Class Codes: H04N 21/44222 Analytics of user selections, e.g. selection of programs or purchase activity monitoring of user selections in data processing systems G06F11/34; arrangements for monitoring the user's behaviour or opinions in broadcast systems H04H60/33
• CPC Class Codes: H04N 21/4532 involving end-user characteristics, e.g. viewer profile, preferences monitoring of user activities for profile generation for accessing a video database G06F16/739; user profiles in network data switching protocols H04L67/306; processing of user preferences or user profiles in wireless networks H04W8/18
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

In one embodiment, an image analysis is performed on an image captured using a camera mounted on an autonomous vehicle, the image representing an exterior environment of an autonomous vehicle. Localization information surrounding the autonomous vehicle is obtained at a point in time. A perception of an audience external to the autonomous vehicle is determined based on the image analysis and the localization information. One or more content items are received from one or more content servers over a network in response to the perception of the audience. A first content item selected from the one or more content items is displayed on a display device mounted on an exterior surface of the autonomous vehicle.

Claims

1. A computer-implemented method for content delivery to an autonomous vehicle, the method comprising:

performing an image analysis on an image captured using a camera mounted on an autonomous vehicle, the image representing an exterior environment of an autonomous vehicle;

obtaining localization information surrounding the autonomous vehicle at a point in time;

determining a perception associated with exterior environment of the autonomous vehicle based on the image analysis and the localization information;

receiving a plurality of content items from one or more content servers over a network in response to the perception associated with the exterior environment of the autonomous vehicle;

ranking the plurality of content items using a predetermined ranking algorithm to select a first content item from the plurality of content items;

superimposing the first content item onto the image captured by the camera, generating an augmented image augmented with the first content item; and

displaying the augmented image on a display device mounted on an interior surface of the autonomous vehicle, wherein the display device includes a touch screen to allow a passenger to interact with one or more choices presented by the first content item, wherein the display device is configured to appear as an ordinary window of the autonomous vehicle, and wherein when the passenger looks at the display device, an object of the augmented image is displayed as a virtual object in a virtual reality manner as if the passenger looked at a physical object through the ordinary window of the autonomous vehicle.

2. The method of claim 1, further comprising transmitting perception information describing the perception to of the one or more content servers to exchange content items provided by the content servers.

3. The method of claim 2, wherein the perception information comprises one or more keywords describing the perception, and wherein the content servers are cloud servers communicatively coupled to the autonomous vehicle over a cloud network.

4. The method of claim 1, wherein the content items are ranked based on a user profile of the passenger.

5. The method of claim 1, wherein the display device is mounted on the same exterior surface of the autonomous vehicle on which the cameras is mounted.

6. The method of claim 1, wherein the camera is a video camera that captures a stream of video frames in real time, and wherein the display device displays a stream of augmented video frames having the first content item augmented thereon.

7. The method of claim 1, further comprising:

caching the first content item in a local content database for subsequent usage; and

periodically updating content items stored in the local content database with the content servers.

8. A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations of providing content to autonomous vehicle, the operations including

performing an image analysis on an image captured using a camera mounted on an autonomous vehicle, the image representing an exterior environment of an autonomous vehicle;

obtaining localization information surrounding the autonomous vehicle at a point in time;

determining a perception associated with exterior environment of the autonomous vehicle based on the image analysis and the localization information;

receiving a plurality of content items from one or more content servers over a network in response to the perception associated with the exterior environment of the autonomous vehicle;

ranking the plurality of content items using a predetermined ranking algorithm to select a first content item from the plurality of content items;

superimposing the first content item onto the image captured by the camera, generating an augmented image augmented with the first content item; and

displaying the augmented image on a display device mounted on an interior surface of the autonomous vehicle, wherein the display device includes a touch screen to allow a passenger to interact with one or more choices presented by the first content item, wherein the display device is configured to appear as an ordinary window of the autonomous vehicle, and wherein when the passenger looks at the display device, an object of the augmented image is displayed as a virtual object in a virtual reality manner as if the passenger looked at a physical object through the ordinary window of the autonomous vehicle.

9. The machine-readable medium of claim 8, wherein the operations further comprise transmitting perception information describing the perception to of the one or more content servers to exchange content items provided by the content servers.

10. The machine-readable medium of claim 9, wherein the perception information comprises one or more keywords describing the perception, and wherein the content servers are cloud servers communicatively coupled to the autonomous vehicle over a cloud network.

11. The machine-readable medium of claim 8, wherein the content items are ranked based on a user profile of the passenger.

12. The machine-readable medium of claim 8, wherein the display device is mounted on the same exterior surface of the autonomous vehicle on which the cameras is mounted.

13. The machine-readable medium of claim 8, wherein the camera is a video camera that captures a stream of video frames in real time, and wherein the display device displays a stream of augmented video frames having the first content item augmented thereon.

14. The machine-readable medium of claim 8, wherein the operations further comprise:

caching the first content item in a local content database for subsequent usage; and

periodically updating content items stored in the local content database with the content servers.

15. A data processing system, comprising:

a processor; and

a memory coupled to the processor for storing instructions, which when executed by the processor, cause the processor to perform operations of providing content in an autonomous vehicle, the operations including

performing an image analysis on an image captured using a camera mounted on an autonomous vehicle, the image representing an exterior environment of an autonomous vehicle,

obtaining localization information surrounding the autonomous vehicle at a point in time,

determining a perception associated with exterior environment of the autonomous vehicle based on the image analysis and the localization information,

receiving a plurality of content items from one or more content servers over a network in response to the perception associated with the exterior environment of the autonomous vehicle,

ranking the plurality of content items using a predetermined ranking algorithm to select a first content item from the plurality of content items,

superimposing the first content item onto the image captured by the camera, generating an augmented image augmented with the first content item, and

displaying the augmented image on a display device mounted on an interior surface of the autonomous vehicle, wherein the display device includes a touch screen to allow a passenger to interact with one or more choices presented by the first content item, wherein the display device is configured to appear as an ordinary window of the autonomous vehicle, and wherein when the passenger looks at the display device, an object of the augmented image is displayed as a virtual object in a virtual reality manner as if the passenger looked at a physical object through the ordinary window of the autonomous vehicle.

16. The system of claim 15, wherein the operations further comprise transmitting perception information describing the perception to of the one or more content servers to exchange content items provided by the content servers.

17. The system of claim 16, wherein the perception information comprises one or more keywords describing the perception, and wherein the content servers are cloud servers communicatively coupled to the autonomous vehicle over a cloud network.

18. The system of claim 15, wherein the content items are ranked based on a user profile of the passenger.

19. The system of claim 15, wherein the display device is mounted on the same exterior surface of the autonomous vehicle on which the cameras is mounted.

20. The system of claim 15, wherein the camera is a video camera that captures a stream of video frames in real time, and wherein the display device displays a stream of augmented video frames having the first content item augmented thereon.

21. The system of claim 15, wherein the operations further comprise:

caching the first content item in a local content database for subsequent usage; and

periodically updating content items stored in the local content database with the content servers.