Intelligent method of determining trigger items in augmented reality environments

• US 9,530,251 B2
• Filed: 04/15/2014
• Issued: 12/27/2016
• Est. Priority Date: 05/10/2012
• Status: Active Grant

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Aurasma Limited (HP Inc.)
• Inventors: Loxam, James, Blanchflower, Sean Mark
• Primary Examiner(s): Zimmerman, Mark
• Assistant Examiner(s): Nguyen, Vu
• Application Number:
  US14/253,791
• Publication Number:
  US 20140225924A1
• Time in Patent Office: 987 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09G 2340/12 Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels
• CPC Class Codes: G09G 5/14 Display of multiple viewports

Abstract

The systems and methods may allow mobile computing devices to identify real world trigger items and to cause augmented reality scenarios associated with a real world trigger item to be presented on a display of the mobile computing device. A lookup scheme may identify indexed feature points of known candidate trigger items stored in a database. A number of evaluations occur between the indexed feature points to the feature points found in the real world trigger item. A determination that the feature points of a candidate trigger item matches the feature points of the real world trigger item is made when a confidence threshold is met or exceeded based on at least the look up scheme, weighted factor evaluation, variable threshold evaluation and potentially a geometric consistency evaluation to declare a positive match.

Claims

1. A computer-implemented method comprising:

processing a stream of frames of a scene captured using a camera of a mobile computing device when operated by a user;

performing image recognition to recognize, from one or more frames of the stream of frames, a trigger item corresponding to any one of a symbol, person, or object depicted in the scene; and

in response to performing image recognition, (i) determining an augmented reality scenario that includes an augmented reality content item generated by a second user of another computing device using an image of a symbol, person or object that is deemed to sufficiently match the trigger item based on a predetermined confidence factor, and (ii) presenting, on a display, the augmented reality scenario with the trigger item, as the trigger item is depicted in the stream of frames of the scene.

2. The computer-implemented method of claim 1, wherein performing image recognition includes performing the image recognition on a plurality of frames of the stream of frames.

3. The computer-implemented method of claim 2, wherein performing image recognition includes determining a plurality of feature points of the trigger item from multiple frames of the stream of frames.

4. The computer-implemented method of claim 1, further comprising transmitting at least a set of frames from the stream of frames from the mobile computing device of a user to a remote server system, the remote server system performing image processing operations for at least some of the image recognition on at least the set of frames to detect the trigger item by comparing the one or more characteristics of the trigger item with characteristics of known items in a database of the remote server system.

5. The computer-implemented method of claim 4, wherein determining the augmented reality scenario includes receiving, on the mobile computing device, the augmented reality content item for the augmented reality scenario from the remote server system.

6. The method of claim 5, wherein the remote server system cooperates with multiple instances of an augmented reality application that each reside on different computing devices, including an instance of the augmented reality application that resides on the mobile computing device.

7. The method of claim 1, wherein the trigger item corresponds to one or more of a billboard or a poster, and wherein the augmented reality content corresponds to a video.

8. A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to:

process a stream of frames of a scene detected by a camera of a mobile computing device in real-time, the mobile computing device being operated by a user;

perform image recognition to recognize, from one or more frames of the stream of frames, a trigger item corresponding to any one of a symbol, person, or object depicted in the scene; and

in response to performing image recognition, (i) determine an augmented reality scenario that includes an augmented reality content item generated by a second user of another computing device using an image of a symbol, person or object that is deemed to sufficiently match the trigger item based on a predetermined confidence factor, and (ii) provide for display, the augmented reality scenario with the trigger item as the trigger item is depicted in the stream of frames of the scene.

9. The non-transitory computer-readable medium of claim 8, wherein the instructions are executed on a mobile computing device.

10. The non-transitory computer-readable medium of claim 9, wherein the processor performs the image recognition by determining a plurality of feature points of the trigger item from the one or more frames of the stream of frames.

11. The non-transitory computer-readable medium of claim 9, wherein the instructions cause the mobile computing device to determine the augmented reality scenario by receiving the augmented reality content item for the augmented reality scenario from a remote server system.

12. The non-transitory computer-readable medium of claim 8, wherein the instructions cause the mobile computing device to detect the trigger item by transmitting at least a set of frames from the stream of frames to a remote server system, and to receive from the remote server system an augmented reality content scenario determined by the remote server system based upon an image recognition performed on at least the set of frames at the remote server system.

13. The non-transitory computer-readable medium of claim 8, wherein the instructions are executed on a remote server system, and wherein the remote server system cooperates with multiple instances of an augmented reality application that each resides on different computing devices, including an instance of the augmented reality application that resides on the mobile computing device.

14. The non-transitory computer-readable medium of claim 8, wherein the trigger item corresponds to one or more of an image, a billboard, or a poster, and wherein the augmented reality content corresponds to a video.

15. A mobile computing device, comprising:

a camera;

a display;

one or more memory resources storing instructions; and

a processor, coupled to the camera, the display, and the one or more memory resources, to execute the instructions, wherein the instructions, when executed by the processor, cause the mobile computing device to;

process a stream of frames of a scene captured using the camera in real-time, the mobile computing device being operated by a user;

initiate performance of image recognition to recognize, from analysis of one or more frames of the stream of frames, a trigger item in at symbol, person, or object depicted in the scene; and

in response to the image recognition being performed, (i) determine an augmented reality scenario that includes an augmented reality content item generated by a second user of another computing device using an image of a symbol, person or object that is deemed to sufficiently match the trigger item based on a predetermined confidence factor, and (ii) present, on the display, the augmented reality scenario with the trigger item as the trigger item is depicted in the stream of frames of the scene in real-time.

16. The mobile computing device of claim 15, wherein the instructions cause the processor to perform the image recognition on the one or more frames of the stream of frames in order to recognize the trigger item.

17. The mobile computing device of claim 16, wherein the instructions cause the processor to perform the image recognition by determining a plurality of feature points of the trigger item from the one or more frames of the stream of frames.

18. The mobile computing device of claim 15, further comprising:

a network interface; and

wherein the instructions cause the processor to initiate the image recognition by transmitting at least the one or more frames of the stream of frames to a remote server system via the network interface, the remote server system performing one or more image processing operations on at least the set of frames to detect the trigger item by comparing one or more characteristics of the trigger item with characteristics of known items in a database of the remote server system.

19. The mobile computing device of claim 18, wherein the instructions cause the mobile computing device to receive the augmented reality content item for the augmented reality scenario from the remote server system.

20. The mobile computing device of claim 19, wherein the remote server system cooperates with multiple instances of an augmented reality application that each resides on different computing devices, including an instance of the augmented reality application that resides on the mobile computing device.