MULTIMODAL AUGMENTED REALITY FOR LOCATION MOBILE INFORMATION SERVICE

• US 20110153310A1
• Filed: 12/23/2009
• Published: 06/23/2011
• Est. Priority Date: 12/23/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: AT&T Mobility II LLC (AT&T, Inc.)
• Inventors: Zajac, Remi, Ehlen, Patrick, Johnston, Michael
• Granted Patent: US 8,688,443 B2
• Time in Patent Office: Days
• US Class Current: 704/9
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: H04L 67/04 specially adapted for terminals or networks with limited capabilities; specially adapted for terminal portability
• CPC Class Codes: H04L 67/52 specially adapted for the location of the user terminal
• CPC Class Codes: H04M 2250/10 including a GPS signal receiver
• CPC Class Codes: H04M 3/42348 Location-based services which utilize the location information of a target

Abstract

In one or more embodiments, one or more methods and/or systems described can perform producing a lattice of object hypotheses based on multiple reference objects from image information; receiving input speech information that includes a request for information associated with at least one reference object of the multiple reference objects; producing a lattice of speech hypotheses based on at least a first possible description included in the speech information; producing a lattice of scored semantic hypotheses based on at least the lattice of object hypotheses and the lattice of speech hypotheses; determining that a single semantic interpretation score of the lattice of scored semantic hypotheses exceeds a predetermined value; and providing requested information associated with the at least the first reference object of the plurality of reference objects.

Claims

1. A method, comprising:

producing a lattice of object hypotheses based on a plurality of reference objects from image information;

receiving input speech information that includes a request for information associated with at least a first reference object of the plurality of reference objects;

producing a lattice of speech hypotheses based on at least a first possible description included in the speech information;

producing a lattice of scored semantic hypotheses based on at least the lattice of object hypotheses and the lattice of speech hypotheses;

determining that a single semantic interpretation score of the lattice of scored semantic hypotheses exceeds a predetermined value; and

providing requested information associated with the at least the first reference object of the plurality of reference objects.

2. The method of claim 1, further comprising:

determining that two or more semantic interpretation scores of the lattice of scored semantic hypotheses exceeds the predetermined value;

receiving clarification from a user; and

providing requested information associated with at least a second reference object of the plurality of reference objects.

3. The method of claim 1, further comprising:

receiving the image information from an electronic image sensor.

5. The method of claim 1, wherein producing the lattice of scored semantic hypotheses based on at least the lattice of object hypotheses and the lattice of speech hypotheses includes producing the lattice of scored semantic hypotheses based on at least the lattice of object hypotheses, the lattice of speech hypotheses, and a finite state machine.

6. The method of claim 1, wherein the requested information includes one or more of a telephone number, one or more directions, a review, a rating, and a menu.

7. The method of claim 6, further comprising:

providing the one or more directions by providing at least one of output speech information that includes the one or more directions and image overlay information that indicates the one or more directions on an image.

4. The method of claim 4, further comprising:

receiving the image information via at least one of a packet network, an Internet, a cellular telephone network, and a satellite telephone network.

8. A system, comprising:

a processor; and

a memory medium coupled to the processor;

wherein the memory medium includes instructions, which when executed by the processor, cause the system to perform;

producing a lattice of object hypotheses based on a plurality of reference objects from image information;

receiving input speech information that includes a request for information associated with at least a first reference object of the plurality of reference objects;

producing a lattice of speech hypotheses based on at least a first possible description included in the speech information;

producing a lattice of scored semantic hypotheses based on at least the lattice of object hypotheses and the lattice of speech hypotheses;

determining that a single semantic interpretation score of the lattice of scored semantic hypotheses exceeds a predetermined value; and

providing requested information associated with the at least the first reference object of the plurality of reference objects.

9. The system of claim 8, wherein the memory medium further includes instructions, which when executed by the processor, cause the system to further perform:

determining that two or more semantic interpretation scores of the lattice of scored semantic hypotheses exceeds the predetermined value;

receiving clarification from a user; and

providing requested information associated with at least a second reference object of the plurality of reference objects.

10. The system of claim 8, wherein the memory medium further includes instructions, which when executed by the processor, cause the system to further perform:

receiving the image information from an electronic image sensor.

11. The system of claim 10, wherein the memory medium further includes instructions, which when executed by the processor, cause the system to further perform:

receiving the image information via at least one of a packet network, an Internet, a cellular telephone network, and a satellite telephone network.

12. The system of claim 8, wherein producing the lattice of scored semantic hypotheses based on at least the lattice of object hypotheses and the lattice of speech hypotheses includes producing the lattice of scored semantic hypotheses based on at least the lattice of object hypotheses, the lattice of speech hypotheses, and a finite state machine.

13. The system of claim 8, wherein the requested information includes one or more of a telephone number, one or more directions, a review, a rating, and a menu.

14. The system of claim 13, wherein the memory medium further includes instructions, which when executed by the processor, cause the system to further perform:

providing the one or more directions by providing at least one of output speech information that includes the one or more directions and image overlay information that indicates the one or more directions on an image.

15. A computer readable memory medium comprising instructions, which when executed on a processing system, cause the processing system to perform:

producing a lattice of object hypotheses based on a plurality of reference objects from image information;

receiving input speech information that includes a request for information associated with at least a first reference object of the plurality of reference objects;

producing a lattice of speech hypotheses based on at least a first possible description included in the speech information;

producing a lattice of scored semantic hypotheses based on at least the lattice of object hypotheses and the lattice of speech hypotheses;

determining that a single semantic interpretation score of the lattice of scored semantic hypotheses exceeds a predetermined value; and

providing requested information associated with the at least the first reference object of the plurality of reference objects.

16. The computer readable memory medium of claim 15, wherein the memory medium further includes instructions, which when executed on the processing system, cause the processing system to further perform:

determining that two or more semantic interpretation scores of the lattice of scored semantic hypotheses exceeds the predetermined value;

receiving clarification from a user; and

providing requested information associated with at least a second reference object of the plurality of reference objects.

17. The computer readable memory medium of claim 15, wherein the memory medium further includes instructions, which when executed on the processing system, cause the processing system to further perform:

receiving the image information from an electronic image sensor.

18. The computer readable memory medium of claim 17, wherein the memory medium further includes instructions, which when executed on the processing system, cause the processing system to further perform:

receiving the image information via at least one of a packet network, an Internet, a cellular telephone network, and a satellite telephone network.

19. The computer readable memory medium of claim 15, wherein producing the lattice of scored semantic hypotheses based on at least the lattice of object hypotheses and the lattice of speech hypotheses includes producing the lattice of scored semantic hypotheses based on at least the lattice of object hypotheses, the lattice of speech hypotheses, and a finite state machine.

20. The computer readable memory medium of claim 15, wherein the requested information includes one or more of a telephone number, one or more directions, a review, a rating, and a menu.