System and method for providing augmented virtual reality content in autonomous vehicles

• US 10,366,290 B2
• Filed: 05/11/2016
• Issued: 07/30/2019
• Est. Priority Date: 05/11/2016
• Status: Active Grant

• Current Assignee: Baidu USA LLC (Baidu Incorporated)
• Original Assignee: Baidu USA LLC (Baidu Incorporated)
• Inventors: Wang, Quan, Ma, Biao, Liu, Shaoshan, Peng, James
• Primary Examiner(s): Amini, Javid A
• Application Number:
  US15/152,015
• Publication Number:
  US 20170330034A1
• Time in Patent Office: 1,175 Days
• Field of Search: 345633
• CPC Class Codes: B60K 2360/175 Autonomous driving
• CPC Class Codes: B60K 2360/176 Camera images
• CPC Class Codes: B60K 2360/177 Augmented reality
• CPC Class Codes: B60K 2360/21 using cameras
• CPC Class Codes: B60K 2360/592 Data transfer involving external databases
• CPC Class Codes: B60K 2360/741 Instruments adapted for user detection
• CPC Class Codes: B60K 35/00 Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
• CPC Class Codes: B60K 35/28 characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
• CPC Class Codes: B60K 35/65 Instruments specially adapted for specific vehicle types or users, e.g. for left- or right-hand drive
• CPC Class Codes: B60K 35/654 the user being the driver
• CPC Class Codes: B60K 35/85 Arrangements for transferring vehicle- or driver-related data
• CPC Class Codes: B60W 50/00 Details of control systems for road vehicle drive control not related to the control of a particular sub-unit , e.g. process diagnostic or vehicle driver interfaces
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: H04L 67/12 specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04L 67/306 User profiles
• CPC Class Codes: H04L 67/52 : specially adapted for the location of the user terminal

Abstract

In one embodiment, a server receives a request from a first autonomous vehicle for content delivery. In response to the request, a vision analysis is performed on an image obtained from the request to determine three-dimensional (3D) positioning information of the image. A list of content items are identified based on current vehicle information of the first autonomous vehicle in view of a user profile of a user ridding the first autonomous vehicle. A first content item selected from the list of content items is augmented onto the image based on the 3D positioning information of the image, generating an augmented image. The augmented image is transmitted to the first autonomous vehicle, where the augmented image is to be displayed on a display device within the autonomous vehicle in a virtual reality manner.

Claims

1. A computer-implemented method for content delivery to autonomous vehicles, the method comprising:

receiving at a server a request from a first autonomous vehicle for content delivery over a network, the request including an image that was captured by a camera of the first autonomous vehicle while the first autonomous vehicle was driving on a road, the image capturing an object external to the first autonomous vehicle;

in response to the request, performing a vision analysis on an image obtained from the request to determine three-dimensional (3D) positioning information of the image, including determining a position of the object within the image and a viewing angle of the object with respect to a location of the camera;

identifying a list of a plurality of content items to augment based on current vehicle information of the first autonomous vehicle in view of one or more user profiles of one or more users to be entertained while riding the first autonomous vehicle;

augmenting a first content item selected based on user preferences from the list of content items onto the image by superimposing the first content item onto the image based on the position and viewing angle of the object to generate an augmented image, wherein the first content item includes a content item previously cached by a content database maintained by the autonomous vehicle; and

transmitting the augmented image to the first autonomous vehicle over the network, wherein the augmented image is to be displayed on a display device within the autonomous vehicle in a virtual reality manner.

2. The method of claim 1, wherein the server provides content delivery to a plurality of autonomous vehicle over the network.

3. The method of claim 1, wherein the image was captured in real-time by one or more cameras of the first autonomous vehicle representing a surrounding view of the first autonomous vehicle.

4. The method of claim 3, wherein the image is one of a plurality of images streamed in sequence from the first autonomous vehicle, and wherein the augmented image is one of a plurality of augmented images streamed from the server to the first autonomous vehicle.

5. The method of claim 1, wherein the current vehicle information includes location information of the first autonomous vehicle obtained from the request.

6. The method of claim 1, wherein the selected content item is selected from the list of content items based on a user preference determined based on the user profile, wherein the user profile is retrieved based on a user identifier obtained from the request.

7. The method of claim 1, further comprising performing an image recognition on the image to recognize a first object external to the autonomous vehicle, wherein the content item augmented onto the image includes first metadata describing the first object.

8. The method of claim 7, wherein the content item further includes second metadata describing a second object nearby, including indicating a distance between the first object and the second object.

9. The method of claim 8, wherein the second object is not shown in the image, but is within a predetermined proximity of the first autonomous vehicle based on map and point of interest (MPOI) information obtained based on location and route information of the first autonomous vehicle.

10. A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations of content delivery to autonomous vehicles, the operations comprising:

receiving a request from a first autonomous vehicle for content delivery over a network, the request including an image that was captured by a camera of the first autonomous vehicle while the first autonomous vehicle was driving on a road, the image capturing an object external to the first autonomous vehicle;

in response to the request, performing a vision analysis on an image obtained from the request to determine three-dimensional (3D) positioning information of the image, including determining a position of the object within the image and a viewing angle of the object with respect to a location of the camera;

identifying a list of a plurality of content items to augment based on current vehicle information of the first autonomous vehicle in view of one or more user profiles of a one or more users to be entertained while riding the first autonomous vehicle;

augmenting a first content item selected based on user preferences from the list of content items onto the image by superimposing the first content item onto the image based on the position and viewing angle of the object to generate an augmented image, wherein the first content item includes a content item previously cached by a content database maintained by the autonomous vehicle; and

transmitting the augmented image to the first autonomous vehicle over the network, wherein the augmented image is to be displayed on a display device within the autonomous vehicle in a virtual reality manner.

11. The non-transitory machine-readable medium of claim 10, wherein the server provides content delivery to a plurality of autonomous vehicle over the network.

12. The non-transitory machine-readable medium of claim 10, wherein the image was captured in real-time by one or more cameras of the first autonomous vehicle representing a surrounding view of the first autonomous vehicle.

13. The non-transitory machine-readable medium of claim 12, wherein the image is one of a plurality of images streamed in sequence from the first autonomous vehicle, and wherein the augmented image is one of a plurality of augmented images streamed from the server to the first autonomous vehicle.

14. The non-transitory machine-readable medium of claim 10, wherein the vehicle information includes location information of the first autonomous vehicle obtained from the request.

15. The non-transitory machine-readable medium of claim 10, wherein the selected content item is selected from the list of content items based on a user preference determined based on the user profile, wherein the user profile is retrieved based on a user identifier obtained from the request.

16. The non-transitory machine-readable medium of claim 10, wherein the operations further comprise performing an image recognition on the image to recognize a first object external to the autonomous vehicle, wherein the content item augmented onto the image includes first metadata describing the first object.

17. The non-transitory machine-readable medium of claim 16, wherein the content item further includes second metadata describing a second object nearby, including indicating a distance between the first object and the second object.

18. The non-transitory machine-readable medium of claim 17, wherein the second object is not shown in the image, but is within a predetermined proximity of the first autonomous vehicle based on map and point of interest (MPOI) information obtained based on location and route information of the first autonomous vehicle.

21. The system of claim 18, wherein the image was captured in real-time by one or more cameras of the first autonomous vehicle representing a surrounding view of the first autonomous vehicle.

22. The system of claim 21, wherein the image is one of a plurality of images streamed in sequence from the first autonomous vehicle, and wherein the augmented image is one of a plurality of augmented images streamed from the server to the first autonomous vehicle.

23. The system of claim 18, wherein the current vehicle information includes location information of the first autonomous vehicle obtained from the request.

24. The system of claim 18, wherein the selected content item is selected from the list of content items based on a user preference determined based on the user profile, wherein the user profile is retrieved based on a user identifier obtained from the request.

25. The system of claim 18, wherein the operations further comprise performing an image recognition on the image to recognize a first object external to the autonomous vehicle, wherein the content item augmented onto the image includes first metadata describing the first object.

19. A data processing system operating as a server, comprising:

a processor; and

a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to perform operations, the operations including

receiving a request from a first autonomous vehicle for content delivery over a network, the request including an image that was captured by a camera of the first autonomous vehicle while the first autonomous vehicle was driving on a road, the image capturing an object external to the first autonomous vehicle;

in response to the request, performing a vision analysis on an image obtained from the request to determine three-dimensional (3D) positioning information of the image, including determining a position of the object within the image and a viewing angle of the object with respect to a location of the camera,

identifying a list of a plurality of content items to augment based on current vehicle information of the first autonomous vehicle in view of one or Wore user profiles of one or more users to be entertained while riding the first autonomous vehicle,

augmenting a first content item selected based on user preferences from the list of content items onto the image by superimposing the first content item onto the image based on the position and viewing angle of the object to generate an augmented image, wherein the first content item includes a content item previously cached by a content database maintained by the autonomous vehicle, and

transmitting the augmented image to the first autonomous vehicle over the network, wherein the augmented image is to be displayed on a display device within the autonomous vehicle in a virtual reality manner.

20. The system of claim 19, wherein the server provides content delivery to a plurality of autonomous vehicle over the network.