Intelligent method of determining trigger items in augmented reality environments

• US 9,430,876 B1
• Filed: 05/10/2012
• Issued: 08/30/2016
• Est. Priority Date: 05/10/2012
• Status: Active Grant

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Aurasma Limited (HP Inc.)
• Inventors: Loxam, James, Blanchflower, Sean Mark
• Primary Examiner(s): Zimmerman, Mark
• Assistant Examiner(s): Nguyen, Vu
• Application Number:
  US13/468,246
• Time in Patent Office: 1,573 Days
• Field of Search: 345/633
• US Class Current: 1/1
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09G 2340/12 Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels
• CPC Class Codes: G09G 5/14 Display of multiple viewports

Abstract

The systems and methods may allow mobile computing devices to identify real world trigger items and to cause augmented reality scenarios associated with a real world trigger item to be presented on a display of the mobile computing device. A lookup scheme may identify indexed feature points of known candidate trigger items stored in a database. A number of evaluations occur between the indexed feature points to the feature points found in the real world trigger item. A determination that the feature points of a candidate trigger item matches the feature points of the real world trigger item is made when a confidence threshold is met or exceeded based on at least the look up scheme, weighted factor evaluation, variable threshold evaluation and potentially a geometric consistency evaluation to declare a positive match.

Claims

1. A computer-aided method for enabling an augmented reality application resident on a computing device to identify a real world trigger item, the method comprising:

capturing, by a processor, an image of a real world trigger item from a series of frames of a video based on geographical information associated, with a location where the video was captured;

identifying indexed feature points of a known candidate trigger item stored in a database that match feature points of the real world trigger item;

performing a geometric consistency evaluation of the feature points of the real world trigger item, including comparing an overall shape of a composite of individual tiles of the real world trigger item to an overall shape of each of the matching feature points of the known candidate trigger item;

performing a weighted factor evaluation of the feature points of the real world trigger item, including identifying a required set of feature points for the real world trigger item and determining that the matching feature points of the known candidate trigger item include the required set of feature points;

determining, by the processor, a percentage of the matching feature points of the known candidate trigger item relative to a total number of indexed feature points of the known candidate trigger item;

determining whether the determined percentage is equal to or greater than a percentage threshold;

in response to a determination that the determined percentage is equal to or greater than the percentage threshold, selecting an augmented reality scenario associated with the real world trigger; and

presenting, by the processor, the augmented reality scenario onto a display of the computing device by overlaying augmented reality content onto the series of frames of the video.

2. The method of claim 1, further comprising:

eliminating those candidate trigger items as potential matches to the real world trigger item based on the candidate trigger items not possessing feature points in the required set.

3. The method of claim 1, further comprising:

based on one or more of the feature points of the real world trigger item not being geometrically consistent with the indexed feature points of the known candidate trigger item, capturing an image of another real world trigger item from the series of frames of the video and repeating the use of the lookup scheme to find the indexed feature points of the known candidate trigger e previously stored in the database to determine another match.

4. The method of claim 1, wherein when the percentage threshold is not met or exceeded, capturing the image of the real world trigger item from additional frames of the video and repeating the look up scheme, weighted factor evaluation, and determination of a percentage to declare a positive match.

5. The method of claim 1, wherein the indexed feature points of the known candidate trigger item previously stored in the database are identified based on geographic information associated with the captured image of the real world trigger item.

6. A non-transitory computer readable medium, which stores instructions executable by a processor to cause a computing device to:

capture an image of a real world trigger item from a series of frames of a video based on geographical metadata associated with a location where the video was captured;

identify indexed feature points of a known candidate trigger item to d in a database that match feature points of the real world trigger item;

perform a geometric consistency evaluation of the feature points of the real world trigger item, including compare an overall shape of a composite of individual tiles of the real world trigger item to an overall shape of each of the matching feature points of the known candidate trigger item;

identify a required set of feature points for the real world trigger item,

determine that the matching feature points of the known candidate trigger item include the required set of feature points;

determine a percentage based on a number of the matching feature points of the known candidate trigger item relative to a total number of indexed feature points of the known candidate trigger item;

determine whether the determined percentage is equal to or greater than a percentage threshold;

in response to a determination that the determined percentage is equal to or greater than the percentage threshold, select an augmented reality scenario associated with the real world trigger; and

overlay augmented reality content from the augmented reality scenario on a display of the computing device.

7. A computing device comprising

at least one processor;

and

a memory storing machine readable instructions to be executed by the at least one processor to cause the computing device to;

index and store feature points associated with known candidate trigger items in one or more databases;

capture an image of a real world trigger item from a series of frames of a video based on global positioning system information associated with a location where the video was captured;

identify indexed feature points of a first known candidate trigger item stored in the one or more databases that match feature points of the real world trigger item;

perform a geometric consistency evaluation, including compare an overall shape of a composite of individual tiles of the real world trigger item to an overall shape of each of the matching feature points of the known candidate trigger item;

identify a required set of feature points for the real world trigger item;

determine that the matching feature points of the first known candidate trigger item include the required set of feature points;

determine a percentage based on a number of the matching feature points of the first known candidate trigger item relative to a total number of indexed feature points of the first known candidate trigger item;

determine whether the determined percentage is equal to or greater than a percentage threshold; and

in response to a determination that the determined percentage is equal to or greater than the percentage threshold, provide augmented reality content overlaid onto the series of frames of the video in the computing device.

8. The computing device of claim 7, wherein in response to a determination that the determined percentage is less than the percentage threshold, the machine readable instructions are to cause the computing device to identify indexed feature points of a second known candidate trigger item stored in the one or more databases that match feature points of the real world trigger item and repeat the determining of a percentage to find a positive match.