METHOD AND APPARATUS FOR CONTEXTUAL QUERY BASED ON VISUAL ELEMENTS AND USER INPUT IN AUGMENTED REALITY AT A DEVICE

• US 20150317836A1
• Filed: 05/05/2014
• Published: 11/05/2015
• Est. Priority Date: 05/05/2014
• Status: Active Grant

• Current Assignee: HERE Global B.V. (Nokia Corporation)
• Original Assignee: HERE Global B.V. (Nokia Corporation)
• Inventors: BEAUREPAIRE, Jerome, FIALHO, Andre, KRAUS, Daniel
• Granted Patent: US 9,558,716 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 2203/04802 3D-info-object: information is displayed on the internal or external surface of a three dimensional manipulable object, e.g. on the faces of a cube that can be rotated by the user
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09G 2340/12 Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 2370/10 Use of a protocol of communication by packets in interfaces along the display data pipeline
• CPC Class Codes: G09G 5/026 Control of mixing and/or overlay of colours in general G09G5/022 and G09G5/024 take precedence
• CPC Class Codes: G09G 5/14 Display of multiple viewports

Abstract

An approach is provided for enabling a contextual query based on visual elements and user input in augmented reality at a device. The approach includes a method for receiving at least one input specifying content information, wherein the input is received via at least one user interface presenting image data. The method further includes processing and/or facilitating a processing of the image data to cause, at least in part, an identification of one or more visual elements. The method also includes causing, at least in part, a construction of at least one query based, at least in part, on the content information and the one or more visual elements.

Claims

1. A method comprising:

receiving at least one input specifying content information, wherein the input is received via at least one user interface presenting image data;

processing and/or facilitating a processing of the image data to cause, at least in part, an identification of one or more visual elements; and

causing, at least in part, a construction of at least one query based, at least in part, on the content information and the one or more visual elements.

2. A method of claim 1, wherein the at least one user interface includes, at least in part, at least one augmented reality user interface.

3. A method of claim 1, further comprising:

determining at least one screen location of the at least one input with respect to the at least one user interface,

wherein the identification of the one or more visual elements is further based, at least in part, on the at least one screen location.

4. A method of claim 3, wherein the identification of the one or more visual elements is based, at least in part, on an overlapping of the at least one screen location with at least one other screen location associated with the one or more visual elements.

5. A method of claim 1, further comprising:

determining contextual information associated with at least one device presenting the at least one user interface, at least one user of the at least one device, the content information, the image data, the one or more visual elements, or a combination thereof,

wherein the construction of the at least one query is further based, at least in part, on the contextual information.

6. A method of claim 5, wherein the content information is determined from a user tactile input, a user audio input, a user gaze information, a user activity, one or more applications at the at least one device, or a combination thereof.

7. A method of claim 5, further comprising:

determining the image data via one or more sensors at the at least one device, one or more applications at the at least one device, from a storage associated with the at least one device, or a combination thereof.

8. A method of claim 1, further comprising:

determining one or more services, one or more points of interest, or a combination thereof associated with the one or more visual elements, the image data, the content information, or a combination thereof,

wherein the construction of the at least one query is further based, at least in part, on the contextual information.

9. A method of claim 1, further comprising:

processing and/or facilitating a processing of the at least one query to determine one or more query results; and

causing, at least in part, a presentation of the one or more query results as at least one overlay element on the one or more visual elements, one or more other visual elements present in the at least one user interface, or a combination thereof.

10. A method of claim 1, further comprising:

causing, at least in part, an initiating of one or more requests for one or more services based, at least in part, on the content information, the one or more visual elements, or a combination thereof.

11. An apparatus comprising:

at least one processor; and

at least one memory including computer program code for one or more programs,

the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to perform at least the following,

receive at least one input specifying content information, wherein the input is received via at least one user interface presenting image data;

process and/or facilitate a processing of the image data to cause, at least in part, an identification of one or more visual elements; and

cause, at least in part, a construction of at least one query based, at least in part, on the content information and the one or more visual elements.

12. An apparatus of claim 11, wherein the apparatus is further caused to:

determine at least one screen location of the at least one input with respect to the at least one user interface,

wherein the identification of the one or more visual elements is further based, at least in part, on the at least one screen location.

13. An apparatus of claim 12, wherein the identification of the one or more visual elements is based, at least in part, on an overlapping of the at least one screen location with at least one other screen location associated with the one or more visual elements.

14. An apparatus of claim 11, wherein the apparatus is further caused to:

determine contextual information associated with at least one device presenting the at least one user interface, at least one user of the at least one device, the content information, the image data, the one or more visual elements, or a combination thereof,

wherein the construction of the at least one query is further based, at least in part, on the contextual information.

15. An apparatus of claim 11, wherein the apparatus is further caused to:

determine one or more services, one or more points of interest, or a combination thereof associated with the one or more visual elements, the image data, the content information, or a combination thereof,

wherein the construction of the at least one query is further based, at least in part, on the contextual information.

16. An apparatus of claim 11, wherein the apparatus is further caused to:

process and/or facilitate a processing of the at least one query to determine one or more query results; and

cause, at least in part, a presentation of the one or more query results as at least one overlay element on the one or more visual elements, one or more other visual elements present in the at least one user interface, or a combination thereof.

17. An apparatus of claim 11, wherein the apparatus is further caused to:

cause, at least in part, an initiating of one or more requests for one or more services based, at least in part, on the content information, the one or more visual elements, or a combination thereof.

18. A computer-readable storage medium carrying one or more sequences of one or more instructions which, when executed by one or more processors, cause an apparatus to at least perform the following steps:

receive at least one input specifying content information, wherein the input is received via at least one user interface presenting image data;

process and/or facilitate a processing of the image data to cause, at least in part, an identification of one or more visual elements; and

cause, at least in part, a construction of at least one query based, at least in part, on the content information and the one or more visual elements.

19. A computer-readable storage medium of claim 18, wherein the apparatus is further caused to:

determine at least one screen location of the at least one input with respect to the at least one user interface,

wherein the identification of the one or more visual elements is further based, at least in part, on the at least one screen location.

20. A computer-readable storage medium of claim 18, wherein the apparatus is further caused to:

determine contextual information associated with at least one device presenting the at least one user interface, at least one user of the at least one device, the content information, the image data, the one or more visual elements, or a combination thereof,

wherein the construction of the at least one query is further based, at least in part, on the contextual information.

21-48. -48. (canceled)