Generating augmented reality images using sensor and location data

• US 10,573,084 B2
• Filed: 10/08/2018
• Issued: 02/25/2020
• Est. Priority Date: 06/15/2010
• Status: Active Grant

• Current Assignee: Live Nation Entertainment Incorporated
• Original Assignee: Live Nation Entertainment Incorporated
• Inventors: Callaghan, James Paul
• Primary Examiner(s): Yang, Ryan R
• Application Number:
  US16/154,216
• Publication Number:
  US 20190108684A1
• Time in Patent Office: 505 Days
• CPC Class Codes: A61M 2005/004 Magazines with multiple needles directly inserted into an injection or infusion device, e.g. revolver-like magazines
• CPC Class Codes: A61M 5/20 Automatic syringes, e.g. with automatically actuated piston rod, with automatic needle injection, filling automatically A61M5/142 , A61M5/46 take precedence; hypodermic projectiles F42B12/54
• CPC Class Codes: A61M 5/31595 Pre-defined multi-dose administration by repeated overcoming of means blocking the free advancing movement of piston rod, e.g. by tearing or de-blocking
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06Q 10/02 Reservations, e.g. for tickets, services or events
• CPC Class Codes: G06Q 30/06 Buying, selling or leasing transactions
• CPC Class Codes: G06Q 30/0643 Graphical representation of items or shoppers
• CPC Class Codes: G06T 1/00 General purpose image data processing
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: H04W 4/02 Services making use of location information

Abstract

Embodiments relate to using sensor data and location data from a device to generate augmented reality images. A mobile device pose can be determined (a geographic position, direction and a three dimensional orientation of the device) within a location. A type of destination in the location can be identified and multiple destinations can be identified, with the mobile device receiving queue information about the identified destinations from a server. A first image can be captured. Based on the queue information, one of the identified destinations can be selected. The geographic position of each identified destination can be identified, and these positions can be combined with the mobile device pose to generate a second image. Finally, an augmented reality image can be generated by combining the first image and the second image, the augmented reality image identifying the selected one destination.

Claims

1. A computer-implemented method comprising:

determining a geographic position of a mobile device, the mobile device being within a bounded geographic area, the bounded geographic area having a plurality of destinations, each destination being of a type of a plurality of types of destination;

determining a three-dimensional (3D) orientation of the mobile device;

identifying a type of destination of the plurality of types of destinations;

transmitting a first communication including an identification of the identified type of destination;

identifying a plurality of destinations associated with the identified type of destination;

receiving, by the mobile device, a second communication, the second communication including queue-length information associated with the identified plurality of destinations;

identifying a geographic position for each of the identified plurality of destinations;

capturing a first image using a camera of the mobile device;

generating a second image based on one or more parameters;

generating an augmented reality image based on the first image and the second image; and

displaying the augmented reality image.

2. The computer-implemented method of claim 1, wherein the one or more parameters includes at least one from amongst a group consisting of:

the geographic position of the mobile device,

the three-dimensional orientation of an augmented reality device,

the queue-length information associated with the identified plurality of destinations, and

the geographic position at least one of the identified plurality of destinations.

3. The computer-implemented method of claim 1, wherein receiving queue-length information associated with the identified plurality of destinations comprises receiving an estimated queue length for each of the identified plurality of destinations, and the method further comprises:

performing a comparison to select a destination of the identified plurality of destinations, the comparison being based on the estimated queue-lengths; and

selecting a destination of the plurality of destinations, the selection based on the estimated queue-lengths, wherein generating a digital image based on queue-length information associated with the plurality of destinations comprises generating a digital image having the selected destination visually highlighted.

4. The computer-implemented method of claim 2, wherein determining the geographic position of the augmented reality device comprises:

identifying a latitude and a longitude measurement from a global positioning system (GPS) sensor in the augmented reality device;

identifying a map associated with the latitude and longitude measurements; and

combining the latitude and longitude measurements with the map to determine the geographic position of the augmented reality device.

5. The computer-implemented method of claim 2, wherein determining the 3D orientation of the augmented reality device comprises determining a 3D orientation of the augmented reality device using a gyroscope of the augmented reality device.

6. The computer-implemented method of claim 5, wherein determining the three-dimensional orientation of the augmented reality device further comprises determining the three-dimensional orientation of the augmented reality device using a compass of the augmented reality device.

7. The computer-implemented method of claim 2, further comprising determining movement information associated with the augmented reality device, wherein generating a digital image further comprises generating a digital image based on the determined movement information.

8. A system, comprising:

one or more processors; and

a non-transitory computer-readable storage medium containing instructions which, when executed on the one or more processors, cause the one or more processors to perform operations including;

determining a geographic position of a mobile device, the mobile device being within a bounded geographic area, the bounded geographic area having a plurality of destinations, each destination being of a type of a plurality of types of destination;

determining a three-dimensional (3D) orientation of the mobile device;

identifying a type of destination of the plurality of types of destinations;

transmitting a first communication including an identification of the identified type of destination;

identifying a plurality of destinations associated with the identified type of destination;

receiving, by the mobile device, a second communication, the second communication including queue-length information associated with the identified plurality of destinations;

identifying a geographic position for each of the identified plurality of destinations;

capturing a first image using a camera of the mobile device;

generating a second image based on one or more parameters;

generating an augmented reality image based on the first image and the second image; and

displaying the augmented reality image.

9. The system of claim 8, wherein the one or more parameters includes at least one from amongst a group consisting of:

the geographic position of the mobile device,

the three-dimensional orientation of an augmented reality device,

the queue-length information associated with the identified plurality of destinations, and

the geographic position at least one of the identified plurality of destinations.

10. The system of claim 8, wherein receiving the queue-length information associated with the identified plurality of destinations comprises receiving an estimated queue length for each of the identified plurality of destinations, and the operations further comprise:

performing a comparison to select a destination of the identified plurality of destinations, the comparison being based on the estimated queue-lengths; and

selecting a destination of the plurality of destinations, the selection based on the estimated queue-lengths, wherein generating a digital image based on queue-length information associated with the plurality of destinations comprises generating a digital image having the selected destination visually highlighted.

11. The system of claim 9, wherein determining the geographic position of the augmented reality device comprises:

identifying a latitude and a longitude measurement from a global positioning system (GPS) sensor in the augmented reality device;

identifying a map associated with the latitude and longitude measurements; and

combining the latitude and longitude measurements with the map to determine the geographic position of the augmented reality device.

12. The system of claim 9, wherein determining the 3D orientation of the augmented reality device comprises determining a 3D orientation of the augmented reality device using a gyroscope of the augmented reality device.

13. The system of claim 9, wherein determining the three-dimensional orientation of the augmented reality device further comprises determining the three-dimensional orientation of the augmented reality device using a compass of the augmented reality device.

14. The system of claim 9, further comprising determining movement information associated with the augmented reality device, wherein generating a digital image further comprises generating a digital image based on the determined movement information.

15. A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, including instructions configured to cause a data processing apparatus to perform operations including:

determining a geographic position of a mobile device, the mobile device being within a bounded geographic area, the bounded geographic area having a plurality of destinations, each destination being of a type of a plurality of types of destination;

determining a three-dimensional (3D) orientation of the mobile device;

identifying a type of destination of the plurality of types of destinations;

transmitting a first communication including an identification of the identified type of destination;

identifying a plurality of destinations associated with the identified type of destination;

receiving, by the mobile device, a second communication, the second communication including queue-length information associated with the identified plurality of destinations;

identifying a geographic position for each of the identified plurality of destinations;

capturing a first image using a camera of the mobile device;

generating a second image based on one or more parameters;

generating an augmented reality image based on the first image and the second image; and

displaying the augmented reality image.

16. The computer-program product of claim 15, wherein the one or more parameters includes at least one from amongst a group consisting of:

the geographic position of the mobile device,

the three-dimensional orientation of an augmented reality device,

the queue-length information associated with the identified plurality of destinations, and

the geographic position at least one of the identified plurality of destinations.

17. The computer-program product of claim 15, wherein receiving the queue-length information associated with the identified plurality of destinations comprises receiving an estimated queue length for each of the identified plurality of destinations, and the operations further comprise:

performing a comparison to select a destination of the identified plurality of destinations, the comparison being based on the estimated queue-lengths; and

selecting a destination of the plurality of destinations, the selection based on the estimated queue-lengths, wherein generating a digital image based on queue-length information associated with the plurality of destinations comprises generating a digital image having the selected destination visually highlighted.

18. The computer-program product of claim 16, wherein determining the geographic position of the augmented reality device comprises:

identifying a latitude and a longitude measurement from a global positioning system (GPS) sensor in the augmented reality device;

identifying a map associated with the latitude and longitude measurements; and

combining the latitude and longitude measurements with the map to determine the geographic position of the augmented reality device.

19. The computer-program product of claim 16, wherein determining the 3D orientation of the augmented reality device comprises determining a 3D orientation of the augmented reality device using a gyroscope of the augmented reality device.

20. The computer-program product of claim 16, wherein determining the three-dimensional orientation of the augmented reality device further comprises determining the three-dimensional orientation of the augmented reality device using a compass of the augmented reality device.