Image matching to augment reality

• US 9,001,252 B2
• Filed: 11/02/2009
• Issued: 04/07/2015
• Est. Priority Date: 11/02/2009
• Status: Active Grant

• Current Assignee: Huawei Device Company Limited (Huawei Investment & Holding Co., Ltd.)
• Original Assignee: Empire Technology Development LLC (Allied Inventors Management, LLC)
• Inventors: Hannaford, Blake
• Primary Examiner(s): Ye, Lin
• Assistant Examiner(s): YODER III, CHRISS S
• Application Number:
  US12/611,083
• Publication Number:
  US 20110102605A1
• Time in Patent Office: 1,982 Days
• Field of Search: 348/113, 348/115, 348/116, 348/119, 348/161, 348/169, 348/207.1, 348/208.2, 348/222.1, 348/231.3, 348/231.5, 348/333.01, 348/333.03, 348/333.12, 348/208.14, 348/333.02, 348/231.4
• US Class Current: 348/333.01
• CPC Class Codes: G06F 16/9537 Spatial or temporal dependent retrieval, e.g. spatiotemporal queries
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 7/00 Image analysis
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: H04L 67/289 Intermediate processing functionally located close to the data consumer application, e.g. in same machine, in same home or in same sub-network
• CPC Class Codes: H04L 67/52 specially adapted for the location of the user terminal
• CPC Class Codes: H04L 67/561 Adding application-functional data or data for application control, e.g. adding metadata
• CPC Class Codes: H04N 1/00244 with a server, e.g. an internet server fax-servers or the like for store and forward H04N1/324
• CPC Class Codes: H04N 23/661 Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04W 4/02 Services making use of location information
• CPC Class Codes: H04W 4/029 Location-based management or tracking services

Abstract

Technology for augmenting reality using image matching is described. The technology can receive a captured image, determine location information relating to the received image, and transmit the location information to a server computing device. The technology may then receive one or more tags from the server computing device, wherein each tag can include one or more patches and supplementary information corresponding to at least one of the patches and each patch can be associated with the location information. The technology can then determine using an image matching process whether at least one of the received one or more patches appears in the received image, and if a patch appears in the received image, render the supplementary information corresponding to the patch.

Claims

1. A method performed by a computing system for augmenting reality, comprising:

receiving a captured image;

determining location information relating to the received captured image;

transmitting the determined location information to a server computing device;

receiving one or more tags from the server computing device, each tag including one or more patches and supplementary information corresponding to at least one of the patches, each patch including a patch image and location information that specifies a location that is near a location specified by the determined location information;

determining, using a template matching process, whether at least one of the received one or more patch images appears in the received captured image; and

in response to a determination that a first patch image appears in the received image, rendering the supplementary information corresponding to the patch associated with the first patch image.

2. The method of claim 1, wherein the captured image is received from a digital camera associated with the computing system.

3. The method of claim 1, wherein the determining the location information is based on information received from a GPS signal.

4. The method of claim 1, wherein the location information transforms a physical condition sensed by an accelerometer into an electrical signal.

5. The method of claim 1, wherein the supplementary information is associated with additional information that specifies how to render the supplementary information.

6. The method of claim 1, wherein the template matching process is at least partially performed by a hardware-based video compression processor.

7. The method of claim 1, wherein the rendering includes superimposing an image specified by the supplementary information.

8. The method of claim 1, wherein the rendering includes playing an audible sound specified by the supplementary information.

9. The method of claim 1, further comprising:

identifying a confidence level for the patch image, and

discarding the patch image in response to the patch image having an identified confidence level below a specified threshold confidence level.

10. A computer-readable memory storing computer-executable instructions that, when executed, perform a method for augmenting reality, the method comprising:

receiving a captured digital image that is digitized by an image capturing device;

determining location information for the image capturing device at the time the captured digital image is digitized;

transmitting the determined location information;

receiving one or more tags, each tag including one or more patches, each patch including a patch image and location information that specifies a location that is near a location specified by the determined location information;

determining using a template matching process whether at least one of the tags includes a patch image that appears in the captured digital image; and

in response to a determination that a first patch image appears in the captured digital image, rendering supplementary information corresponding to the patch associated with the first patch image.

11. The computer-readable memory of claim 10, further comprising storing the captured digital image, the first patch image, and the supplementary information in a data structure, wherein the supplementary information specifies information for the rendering in response to the determination that the first patch image appears in the captured digital image.

12. The computer-readable memory of claim 11, wherein the supplementary information includes an image to superimpose on the digital image.

13. The computer-readable memory of claim 11, wherein the supplementary information includes an audible sound.

14. The computer-readable memory of claim 10, wherein at least one of the one or more tags comprises the supplementary information.

15. The computer-readable memory of claim 14, wherein the supplementary information includes additional information specifying rendering information.

16. The computer-readable memory of claim 10, wherein the determined location information is transmitted using a wireless data communications protocol supported by a mobile telephone service.

17. A system for augmenting reality, comprising:

a processor and memory;

an image capturing component;

a hardware component configured to detect position information in response to an image being captured by the image capturing component;

a transceiver configured to transmit the detected position information and to receive at least one tag, wherein a tag comprises a template image, location information that specifies a location that is near a location specified by the detected location information, and associated information; and

an imaging component configured to employ a template matching routine to determine whether the template image appears within the captured image and to render supplementary information responsive to a determination that the template image appears within the captured image.

18. The system of claim 17, wherein the image capturing component includes a digital camera associated with a client computing device with which the processor and memory are associated.

19. The system of claim 17, wherein the hardware component includes at least one of a GPS receiver, an accelerometer, or a compass.

20. The system of claim 17, wherein the imaging component is further configured to function in association with a second hardware component configured to implement video compression specification.

21. The system of claim 17, wherein the supplementary information is rendered by superimposition of an image on the captured image.