Multimodal augmented reality for location mobile information service

• US 8,688,443 B2
• Filed: 12/23/2009
• Issued: 04/01/2014
• Est. Priority Date: 12/23/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Inventors: Ehlen, Patrick, Johnston, Michael, ZAJAC, Remi
• Primary Examiner(s): Godbold, Douglas
• Application Number:
  US12/645,589
• Publication Number:
  US 20110153310A1
• Time in Patent Office: 1,560 Days
• Field of Search: 704/231, 704/235, 704270-272
• US Class Current: 704/231
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: H04L 67/04 specially adapted for terminals or networks with limited capabilities; specially adapted for terminal portability
• CPC Class Codes: H04L 67/52 specially adapted for the location of the user terminal
• CPC Class Codes: H04M 2250/10 including a GPS signal receiver
• CPC Class Codes: H04M 3/42348 Location-based services which utilize the location information of a target

Abstract

In one or more embodiments, one or more methods and/or systems described can perform producing a lattice of object hypotheses based on multiple reference objects from image information; receiving input speech information that includes a request for information associated with at least one reference object of the multiple reference objects; producing a lattice of speech hypotheses based on at least a first possible description included in the speech information; producing a lattice of scored semantic hypotheses based on at least the lattice of object hypotheses and the lattice of speech hypotheses; determining that a single semantic interpretation score of the lattice of scored semantic hypotheses exceeds a predetermined value; and providing requested information associated with the at least the first reference object of the plurality of reference objects.

Claims

1. A method comprising:

producing a lattice of object hypotheses based at least in part on a plurality of reference objects identified in an image corresponding to a geographic location, wherein each of the plurality of reference objects is related to a physical object at the geographic location;

receiving input speech information that includes a request for information associated with at least a first reference object of the plurality of reference objects;

producing a lattice of speech hypotheses based on at least a first possible description included in the speech information;

producing a lattice of scored semantic hypotheses based on at least the lattice of object hypotheses and the lattice of speech hypotheses;

determining that a single semantic interpretation score of the lattice of scored semantic hypotheses exceeds a predetermined value; and

providing requested information associated with the at least the first reference object of the plurality of reference objects.

2. The method of claim 1, further comprising:

determining that two or more semantic interpretation scores of the lattice of scored semantic hypotheses exceeds the predetermined value;

receiving clarification from a user; and

providing requested information associated with at least a second reference object of the plurality of reference objects.

3. The method of claim 1, wherein a particular physical object related to the first reference object is a building, wherein the first reference object is associated with a particular business entity identified based on the particular physical object, and wherein the request information includes one or more telephone numbers, one or more directions, one or more menus, or a combination thereof associated with the particular business entity.

4. The method of claim 3, further comprising providing the one or more directions by providing output speech information that includes the one or more directions, providing image overlay information that indicates the one or more directions on the image, or both.

5. The method of claim 1, further comprising receiving the image via a packet network, an Internet, a cellular telephone network, a satellite telephone network, or a combination thereof.

6. The method of claim 1, wherein producing the lattice of scored semantic hypotheses based on at least the lattice of object hypotheses and the lattice of speech hypotheses includes producing the lattice of scored semantic hypotheses based on at least the lattice of object hypotheses, the lattice of speech hypotheses, and a finite state machine.

7. The method of claim 1, wherein a particular physical object related to the first reference object is a sign, a lettering, or a street, wherein the first reference object is associated with a particular business entity identified based on the particular physical object, and wherein the requested information includes one or more reviews, one or more ratings, or both associated with the particular business entity.

8. A system comprising:

a processor; and

a memory device coupled to the processor, the memory device including instructions that, when executed by the processor, cause the processor to perform a method comprising;

producing a lattice of object hypotheses based at least in part on a plurality of reference objects identified in an image corresponding to a geographic location, wherein each of the plurality of reference objects is related to a physical object at the geographic location;

receiving input speech information that includes a request for information associated with at least a first reference object of the plurality of reference objects;

producing a lattice of speech hypotheses based on at least a first possible description included in the speech information;

producing a lattice of scored semantic hypotheses based on at least the lattice of object hypotheses and the lattice of speech hypotheses;

determining that a single semantic interpretation score of the lattice of scored semantic hypotheses exceeds a predetermined value; and

providing requested information associated with the at least the first reference object of the plurality of reference objects.

9. The system of claim 8, wherein the method further comprises:

determining that two or more semantic interpretation scores of the lattice of scored semantic hypotheses exceeds the predetermined value;

receiving clarification from a user; and

providing requested information associated with at least a second reference object of the plurality of reference objects.

10. The system of claim 8, wherein the method further comprises receiving the image from an electronic image sensor.

11. The system of claim 8, wherein the method further comprises receiving the image via a packet network, an Internet, a cellular telephone network, a satellite telephone network, or a combination thereof.

12. The system of claim 8, wherein producing the lattice of scored semantic hypotheses based on at least the lattice of object hypotheses and the lattice of speech hypotheses includes producing the lattice of scored semantic hypotheses based on at least the lattice of object hypotheses, the lattice of speech hypotheses, and a finite state machine.

13. The system of claim 8, wherein a particular physical object related to the first reference object is a building, a sign, a lettering, or a street.

14. The system of claim 8, wherein the request information includes one or more directions, and wherein the method further comprises providing the one or more directions by providing output speech information that includes the one or more directions, providing image overlay information that indicates the one or more directions on the image, or both.

15. A computer readable storage device including instructions that, when executed by a processor, cause the processor to perform a method comprising:

producing a lattice of object hypotheses based at least in part on a plurality of reference objects identified in an image corresponding to a geographic location, wherein each of the plurality of reference objects is related to a physical object at the geographic location;

receiving input speech information that includes a request for information associated with at least a first reference object of the plurality of reference objects;

producing a lattice of speech hypotheses based on at least a first possible description included in the speech information;

producing a lattice of scored semantic hypotheses based on at least the lattice of object hypotheses and the lattice of speech hypotheses;

determining that a single semantic interpretation score of the lattice of scored semantic hypotheses exceeds a predetermined value; and

providing requested information associated with the at least the first reference object of the plurality of reference objects.

16. The computer readable storage device of claim 15, wherein the method further comprises:

determining that two or more semantic interpretation scores of the lattice of scored semantic hypotheses exceeds the predetermined value;

receiving clarification from a user; and

providing requested information associated with at least a second reference object of the plurality of reference objects.

17. The computer readable storage device of claim 15, wherein the method further comprises receiving the image from an electronic image sensor.

18. The computer readable storage device of claim 17, wherein the method further comprises receiving the image via a packet network, an Internet, a cellular telephone network, a satellite telephone network, or a combination thereof.

19. The computer readable storage device of claim 15, wherein producing the lattice of scored semantic hypotheses based on at least the lattice of object hypotheses and the lattice of speech hypotheses includes producing the lattice of scored semantic hypotheses based on at least the lattice of object hypotheses, the lattice of speech hypotheses, and a finite state machine.

20. The computer readable storage device of claim 15, wherein the first reference object is associated with a particular business entity, and wherein the requested information includes one or more telephone numbers, one or more directions, one or more review, one or more ratings, one or more menus, or a combination thereof associated with the particular business entity.