USER AUGMENTED REALITY FOR CAMERA-ENABLED MOBILE DEVICES

• US 20090102859A1
• Filed: 10/18/2007
• Published: 04/23/2009
• Est. Priority Date: 10/18/2007
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Yahoo! Inc. (Apollo Global Management, Inc.)
• Inventors: Kalaboukis, Chris T., Athsani, Athellina R. Ahmad
• Granted Patent: US 8,180,396 B2
• Time in Patent Office: Days
• US Class Current: 345/619
• CPC Class Codes: G06F 16/434 using image data, e.g. images, photos, pictures taken by a user
• CPC Class Codes: G06F 16/487 using geographical or spatial information, e.g. location
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: H04M 1/72403 with means for local support of applications that increase the functionality
• CPC Class Codes: H04M 1/72427 for supporting games or graphical animations
• CPC Class Codes: H04M 1/72454 according to context-related or environment-related conditions
• CPC Class Codes: H04M 2250/52 including functional features of a camera constructional features concerning the integration of cameras in portable phones H04M1/0264

Abstract

Disclosed are apparatus and methods for providing a user augmented reality (UAR) service for a camera-enabled mobile device, so that a user of such mobile device can use the mobile device to obtain meta data regarding one or more images/video that are captured with such device. As the user points the mobile device'"'"'s camera at one or more objects in one or more scenes, such objects are automatically analyzed by the UAR to identify the one or more objects and then provide meta data regarding the identified objects in the display of the mobile device. The meta data is interactive and allows the user to obtain additional information or specific types of information, such as information that will aid the user in making a decision regarding the identified objects or selectable action options that can be used to initiate actions with respect to the identified objects. The user can utilize the UAR to continuously pass the camera over additional objects and scenes so that the meta data presented in the display of the mobile device is continuously updated.

Claims

1. A method of providing information regarding one or more scenes captured with a camera of a mobile device, comprising:

when a camera of the mobile device is pointed at a scene having one or more object(s), (i) displaying an image/video of the scene in a display of the mobile device, and (ii) overlaying over the image/video a plurality of options for selecting one of a plurality of user augmented reality modes that include an encyclopedia mode, a decision support mode, and an action mode;

when the encyclopedia mode is selected, obtaining contextual information regarding an identity of the one or more objects and presenting the obtained contextual information in the display;

when the decision support mode is selected, obtaining decision information related to a set of actions that can be taken with respect to an identity of the one or more object(s) and presenting the decision information in the display; and

when the action mode is selected, obtaining a set of references to a plurality of actions that can be performed with respect to an identity of the one or more object(s) and presenting the set of references in the display so that they are selectable by a user to initiate the referenced actions.

2. A method as recited in claim 1, wherein obtaining the contextual information, decision information, and set of references to actions is accomplished by sending the image/video to one or more servers and receiving a second image/video from the one or more servers whereby the second image/video includes the contextual information, decision information, or the set of references, respectively, and wherein presenting the contextual information, decision information, and set of references comprises presenting the received second image/video.

3. A method as recited in claim 2, wherein the second image/video further includes the captured scene so that the contextual information, decision information, or the set of references is overlaid on the captured scene.

4. A method as recited in claim 2, further comprising sending a location of the mobile device to the one or more servers and wherein the contextual information, decision information, and set of references are further based on the location of the mobile device.

5. A method as recited in claim 2, wherein the contextual information, decision information, and set of references are further based on a social network of a user associated with the mobile device.

6. A method as recited in claim 1, wherein the identity of the one or more objects includes an indication as to a direction in which the camera is pointed with respect to a physical location of such camera.

7. A method as recited in claim 1, wherein

the contextual information includes one or more of the following items:

historical information regarding a place in the scene, historical information regarding a building in the scene, time period information regarding other businesses that were located at a place or in a building of the scene, weather information for a place in the scene, travel direction information for a place or building in the scene, one or more reviews, rating, or commentary for at least one object of the scene, a dress requirement of a restaurant or event of the scene, a menu from a restaurant in the scene, a price list for a business in the scene, hours of operation for a place in the scene, information regarding a person in the scene, a link to a personal web site of a person in the scene, contact information for a person in the scene, biographical information for a person in the scene, membership information for a person in the scene, or information regarding a direction of the scene with respect to a street or a transportation route in the scene;

the decision information includes one or more of the following items;

one or more reviews, rating, or commentary by a professional critic or friend in the user'"'"'s social network for at least one object of the scene, a dress requirement of a restaurant of the scene, a menu from a restaurant in the scene, a price list for a business in the scene, hours of operation for a place in the scene, a count of people or friends of the user who have visited a place in the scene, or information regarding a specific type of place in the scene; and

the actions include one or more of the following;

communicating with a particular person via such other person'"'"'s communication device, making reservations to visit a place in the scene, sending a video stream of one or more scene images/video captured with the user'"'"'s mobile device such that the sent video stream may also include the contextual information or the decision information for such one or more scene images/video, or updating a personal information manager (PIM) of the user or another user.

8. A method as recited in claim 1, further comprising when a reference is selected by the user, initiating the referenced action.

9. A method as recited in claim 9, further comprising presenting the contextual and/or decision information or obtaining and presenting additional contextual and/or decision information, after the reference is selected by the user.

10. A mobile device for providing information regarding one or more scenes, comprising:

a camera for capturing one or more scenes;

a display for displaying an image or video;

at least one a processor; and

at least one memory, the at least one processor and/or memory being configured for;

when the camera is pointed at a scene having one or more object(s), (i) displaying an image/video of the scene in the display, and (ii) overlaying over the image/video a plurality of options for selecting one of a plurality of user augmented reality modes that include an encyclopedia mode, a decision support mode, and an action mode;

when the encyclopedia mode is selected, obtaining contextual information regarding an identity of the one or more objects and presenting the obtained contextual information in the display;

when the decision support mode is selected, obtaining decision information related to a set of actions that can be taken with respect to an identity of the one or more object(s) and presenting the decision information in the display; and

when the action mode is selected, obtaining a set of references to a plurality of actions that can be performed with respect to an identity of the one or more object(s) and presenting the set of references in the display so that they are selectable by a user to initiate the referenced actions.

11. A mobile device as recited in claim 10, wherein obtaining the contextual information, decision information, and set of references to actions is accomplished by sending the image/video to one or more servers and receiving a second image/video from the one or more servers whereby the second image/video includes the contextual information, decision information, or the set of references, respectively, and wherein presenting the contextual information, decision information, and set of references comprises presenting the received second image/video.

12. A mobile device as recited in claim 11, wherein the second image/video further includes the captured scene so that the contextual information, decision information, or the set of references is overlaid on the captured scene.

13. A mobile device as recited in claim 11, further comprising sending a location of the mobile device to the one or more servers and wherein the contextual information, decision information, and set of references are further based on the location of the mobile device.

14. A mobile device as recited in claim 11, wherein the contextual information, decision information, and set of references are further based on a social network of a user associated with the mobile device.

15. A mobile device as recited in claim 10, wherein the identity of the one or more objects includes an indication as to a direction in which the camera is pointed with respect to a physical location of such camera.

16. At least one computer readable storage medium having computer program instructions stored thereon that are arranged to perform the following operations:

when a camera of the mobile device is pointed at a scene having one or more object(s), (i) displaying an image/video of the scene in a display of the mobile device, and (ii) overlaying over the image/video a plurality of options for selecting one of a plurality of user augmented reality modes that include an encyclopedia mode, a decision support mode, and an action mode;

when the encyclopedia mode is selected, obtaining contextual information regarding an identity of the one or more objects and presenting the obtained contextual information in the display;

when the decision support mode is selected, obtaining decision information related to a set of actions that can be taken with respect to an identity of the one or more object(s) and presenting the decision information in the display; and

when the action mode is selected, obtaining a set of references to a plurality of actions that can be performed with respect to an identity of the one or more object(s) and presenting the set of references in the display so that they are selectable by a user to initiate the referenced actions.

17. At least one computer readable storage medium as recited in claim 16, wherein obtaining the contextual information, decision information, and set of references to actions is accomplished by sending the image/video to one or more servers and receiving a second image/video from the one or more servers whereby the second image/video includes the contextual information, decision information, or the set of references, respectively, and wherein presenting the contextual information, decision information, and set of references comprises presenting the received second image/video, and wherein the second image/video further includes the captured scene so that the contextual information, decision information, or the set of references is overlaid on the captured scene.

18. At least one computer readable storage medium as recited in claim 16, wherein obtaining the contextual information, decision information, and set of references to actions is accomplished by sending the image/video to one or more servers and receiving a second image/video from the one or more servers whereby the second image/video includes the contextual information, decision information, or the set of references, respectively, and wherein presenting the contextual information, decision information, and set of references comprises presenting the received second image/video and wherein the computer program instructions are further arranged to send a location of the mobile device to the one or more servers and wherein the contextual information, decision information, and set of references are further based on the location of the mobile device.

19. At least one computer readable storage medium as recited in claim 16, wherein the computer program instructions are further arranged to initiate the referenced action when a reference is selected by the user.

20. At least one computer readable storage medium as recited in claim 19, wherein the computer program instructions are further arranged to present the contextual and/or decision information or obtain and present additional contextual and/or decision information, after the reference is selected by the user.

21. A method of providing information to a mobile device, comprising:

when one or more imaged scenes/video are received from a camera of a mobile device, obtaining an identification of the one or more objects of the one or more scenes;

when an encyclopedia mode is selected for the one or more imaged scenes/video, obtaining contextual information regarding the identified one or more objects and sending the obtained contextual information to the mobile device;

when a decision support mode is selected for the one or more imaged scenes/video, obtaining decision information related to a set of actions that can be taken with respect to the identified one or more object(s) and sending the decision information to the mobile device; and

when an action mode is selected for the one or more scenes, obtaining a set of references to a plurality of actions that can be performed with respect to the identified one or more object(s) and sending the set of references to the mobile device, wherein the references are selectable by a user to initiate the referenced actions.

22. A method as recited in claim 21, wherein the contextual information and the decision information are obtained by performing a search for the identified objects and wherein the actions correspond to a type of identified object.

23. A method as recited in claim 21, wherein the decision support mode is automatically selected when a person in an image/video scene is identified as a criminal, wherein the decision information indicates the presence of a criminal at a location of the user, wherein the action mode is automatically selected when a person in an image/video scene is identified as a criminal, wherein the actions include alerting the police regarding the identified criminal.

24. A system for providing text translation for a mobile device, comprising:

at least one a processor; and

at least one memory, the at least one processor and/or memory being configured for;

when one or more imaged scenes/video are received from a camera of a mobile device, obtaining an identification of the one or more objects of the one or more scenes;

when an encyclopedia mode is selected for the one or more imaged scenes/video, obtaining contextual information regarding the identified one or more objects and sending the obtained contextual information to the mobile device;

when a decision support mode is selected for the one or more imaged scenes/video, obtaining decision information related to a set of actions that can be taken with respect to the identified one or more object(s) and sending the decision information to the mobile device; and

when an action mode is selected for the one or more scenes, obtaining a set of references to a plurality of actions that can be performed with respect to the identified one or more object(s) and sending the set of references to the mobile device, wherein the references are selectable by a user to initiate the referenced actions.

25. A system as recited in claim 24, wherein the contextual information and the decision information are obtained by performing a search for the identified objects and wherein the actions correspond to a type of identified object.