Mutual augmented reality experience for users in a network system

• US 10,740,615 B2
• Filed: 11/20/2018
• Issued: 08/11/2020
• Est. Priority Date: 11/20/2018
• Status: Active Grant

• Current Assignee: Uber Technologies, Inc.
• Original Assignee: Uber Technologies, Inc.
• Inventors: Rogan, Aaron Matthew, Leung, Wes, Belmonte, Nicolas Garcia, Sadana, Ramik
• Primary Examiner(s): Patel, Shivang I
• Application Number:
  US16/197,243
• Publication Number:
  US 20200160054A1
• Time in Patent Office: 630 Days
• Field of Search: 345633
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/653 by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04W 4/023 using mutual or relative location information between multiple location based services [LBS] targets or of distance thresholds
• CPC Class Codes: H04W 4/40 for vehicles, e.g. vehicle-to-pedestrians [V2P]
• CPC Class Codes: H04W 4/44 for communication between vehicles and infrastructures, e.g. vehicle-to-cloud [V2C] or vehicle-to-home [V2H]

Abstract

A network system, such as a transport management system, generates a mutual augmented reality (AR) experience for a user and a provider associated with a service. Responsive to receiving a service request, a service management module matches the user with an available provider and monitors the location of the user and provider client devices as the user and provider travel to the pickup location. When the devices are within a threshold distance of each other, an image recognition module monitors live video streams on the devices for the vehicle and the user. Responsive to the vehicle and user entering the field of view of the devices, an AR control module selects computer-generated AR elements and instructs the devices to visually augment the video streams to identify the user and provider to each other and to allow the user and provider to communicate and share data with each other.

Claims

1. A computer-implemented method for providing a mutual augmented reality experience, the method comprising:

receiving periodic location information from a client device associated with a provider of a transportation service and a client device associated with a user of the service;

determining that the provider client device and the user client device are within a threshold distance of each other;

responsive to the provider client device being within the threshold distance of the user client device, instructing each of the provider client device and the user client device to trigger a mutual augmented reality session by;

initiating a live video feed captured by a camera on the respective device; and

augmenting the live video feed with augmented reality elements associated with the transportation service, the augmented reality elements including a first augmented reality element;

receiving an indication of user-selection or provider-selection of the first augmented reality element at the respective client device; and

opening, responsive to receiving the indication, a communication channel between the provider client device and the user client device, wherein the communication channel is configured to enable the user and the provider to exchange messages and augmented reality elements between their respective client devices.

2. The method of claim 1, wherein the augmented reality elements include an identification of a vehicle associated with the provider.

3. The method of claim 1, wherein the augmented reality elements include an identification of a pickup location associated with the transportation service.

4. The method of claim 1, further comprising:

using image recognition to monitor the live video feed on the user client device;

comparing characteristics of an approaching vehicle with provider information to determine a vehicle score; and

responsive to the vehicle score exceeding a vehicle score threshold, determining that the approaching vehicle is the provider'"'"'s vehicle.

5. The method of claim 1, wherein the augmented reality elements include one or more of images, video, text, and graphics.

6. The method of claim 1, wherein the augmented reality elements include an icon allowing the user or provider to initiate a messaging session.

7. The method of claim 1, further comprising:

receiving user input from a first one of the user client device and the provider client device, the user input comprising an instruction to display one or more additional augmented reality elements; and

sending the one or more additional augmented reality elements for display on the first one of the user client device and the provider client device.

8. The method of claim 1, further comprising:

receiving user input from a first one of the user client device and the provider client device, the user input comprising an instruction to display one or more additional augmented reality elements; and

sending the one or more additional augmented reality elements for display on a second one of the user client device and the provider client device.

9. A non-transitory computer-readable storage medium storing computer-executable instructions that, in response to executing, cause a device comprising a processor to perform operations, comprising:

receiving periodic location information from a client device associated with a provider of a transportation service and a client device associated with a user of the service;

determining that the provider client device and the user client device are within a threshold distance of each other;

responsive to the provider client device being within the threshold distance of the user client device, instructing each of the provider client device and the user client device to trigger a mutual augmented reality session by;

initiating a live video feed captured by a camera on the respective device; and

augmenting the live video feed with augmented reality elements associated with the transportation service, the augmented reality elements including a first augmented reality element;

receiving an indication of user-selection or provider-selection of the first augmented reality element at the respective client device; and

opening, responsive to receiving the indication, a communication channel between the provider client device and the user client device, wherein the communication channel is configured to enable the user and the provider to exchange messages and augmented reality elements between their respective client devices.

10. The non-transitory computer-readable storage medium of claim 9, wherein the augmented reality elements include an identification of a vehicle associated with the provider.

11. The non-transitory computer-readable storage medium of claim 9, wherein the augmented reality elements include an identification of a pickup location associated with the transportation service.

12. The non-transitory computer-readable storage medium of claim 9, further comprising:

using image recognition to monitor the live video feed on the user client device;

comparing characteristics of an approaching vehicle with provider information to determine a vehicle score; and

responsive to the vehicle score exceeding a vehicle score threshold, determining that the approaching vehicle is the provider'"'"'s vehicle.

13. The non-transitory computer-readable storage medium of claim 9, wherein the augmented reality elements include one or more of images, video, text, and graphics.

14. The non-transitory computer-readable storage medium of claim 9, wherein the augmented reality elements include an icon allowing the user or provider to initiate a messaging session.

15. The non-transitory computer-readable storage medium of claim 9, further comprising:

receiving user input from a first one of the user client device and the provider client device, the user input comprising an instruction to display one or more additional augmented reality elements; and

sending the one or more additional augmented reality elements for display on the first one of the user client device and the provider client device.

16. The non-transitory computer-readable storage medium of claim 9, further comprising:

receiving user input from a first one of the user client device and the provider client device, the user input comprising an instruction to display one or more additional augmented reality elements; and

sending the one or more additional augmented reality elements for display on a second one of the user client device and the provider client device.

17. A computer system comprising:

one or more computer processors for executing computer program instructions; and

a non-transitory computer-readable storage medium storing instructions executable by the one or more computer processors to perform steps comprising;

receiving periodic location information from a client device associated with a provider of a transportation service and a client device associated with a user of the service;

determining that the provider client device and the user client device are within a threshold distance of each other;

responsive to the provider client device being within the threshold distance of the user client device, instructing each of the provider client device and the user client device to trigger a mutual augmented reality session by;

initiating a live video feed captured by a camera on the respective device; and

augmenting the live video feed with augmented reality elements associated with the transportation service, the augmented reality elements including a first augmented reality element;

receiving an indication of user-selection or provider-selection of the first augmented reality element at the respective client device; and

opening, responsive to receiving the indication, a communication channel between the provider client device and the user client device, wherein the communication channel is configured to enable the user and the provider to exchange messages and augmented reality elements between their respective client devices.

18. The computer system of claim 17, wherein the augmented reality elements include an identification of a vehicle associated with the provider.

19. The computer system of claim 17, wherein the augmented reality elements include an identification of a pickup location associated with the transportation service.

20. The computer system of claim 17, further comprising:

using image recognition to monitor the live video feed on the user client device;

comparing characteristics of an approaching vehicle with provider information to determine a vehicle score; and

responsive to the vehicle score exceeding a vehicle score threshold, determining that the approaching vehicle is the provider'"'"'s vehicle.