USER AUGMENTED REALITY FOR CAMERA-ENABLED MOBILE DEVICES

• US 20120244907A1
• Filed: 05/11/2012
• Published: 09/27/2012
• Est. Priority Date: 10/18/2007
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Yahoo! Inc. (Apollo Global Management, Inc.)
• Inventors: Athsani, Athellina R. Ahmad, Kalaboukis, Chris T.
• Granted Patent: US 8,275,414 B1
• Time in Patent Office: Days
• US Class Current: 455/556.1
• CPC Class Codes: G06F 16/434 using image data, e.g. images, photos, pictures taken by a user
• CPC Class Codes: G06F 16/487 using geographical or spatial information, e.g. location
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: H04M 1/72403 with means for local support of applications that increase the functionality
• CPC Class Codes: H04M 1/72427 for supporting games or graphical animations
• CPC Class Codes: H04M 1/72454 according to context-related or environment-related conditions
• CPC Class Codes: H04M 2250/52 including functional features of a camera constructional features concerning the integration of cameras in portable phones H04M1/0264

Abstract

Apparatus and methods are described for providing a user augmented reality (UAR) service for a camera-enabled mobile device, so that a user of such mobile device can use the mobile device to obtain meta data regarding one or more images/video that are captured with such device. The meta data is interactive and allows the user to obtain additional information or specific types of information, such as information that will aid the user in making a decision regarding the identified objects or selectable action options that can be used to initiate actions with respect to the identified objects.

Claims

1. A method of providing information regarding one or more scenes captured with a camera of a mobile device, comprising:

in response to determining that a camera of the mobile device is pointed at a scene including one or more objects, (i) displaying a visual representation of the scene in a display of the mobile device, and (ii) presenting in combination with the visual representation a plurality of options for selecting one of a plurality of user augmented reality modes that include an encyclopedia mode, a decision support mode, and an action mode;

in response to selection of the decision support mode, obtaining decision information related to a set of actions that can be taken with respect to an identity of the one or more objects and presenting the decision information in the display; and

in response to selection of the action mode after selection of the decision support mode, obtaining a set of references to the set of actions related to the presented decision information and that can be performed with respect to the identity of the one or more objects, and presenting the set of references in the display so that the references are selectable by a user to initiate the referenced actions, wherein the referenced actions include sending information regarding the one or more objects to another user using a contact list on the mobile device, or making a reservation at a place associated with the one or more objects including creating an event in a calendar on the mobile device.

2. The method as recited in claim 1, further comprising:

if the encyclopedia mode is selected while the mobile device is in decision support mode or action mode, switching to the encyclopedia mode by obtaining contextual information regarding an identity of the one or more objects and presenting the obtained contextual information in the display, wherein obtaining the contextual information, decision information, and set of references to actions is accomplished by sending the visual representation to one or more servers and receiving a second visual representation from the one or more servers whereby the second visual representation includes the contextual information, decision information, or the set of references, respectively, and wherein presenting the contextual information, decision information, and set of references comprises presenting the received second visual representation.

3. The method as recited in claim 2, wherein the contextual information, decision information, or the set of references is overlaid on the captured scene.

4. The method as recited in claim 2, further comprising:

determining an identity of each of the one or more objects by processing the visual representation of the scene using an object recognition technique; and

sending the determined identity for each of the one or more objects and a location of the mobile device to the one or more servers and wherein the contextual information, decision information, and set of references are based on the determined identity for each of the one or more objects and the location of the mobile device.

5. The method as recited in claim 4, wherein the contextual information, decision information, and set of references are further based on a social network of the user associated with the mobile device.

6. The method as recited in claim 4, wherein the contextual information, decision information, and set of references are further based on an indication as to a direction in which the camera is pointed with respect to a physical location of such camera.

7. The method as recited in claim 2, wherein

the contextual information includes one or more of the following items:

historical information regarding a place in the scene, historical information regarding a building in the scene, time period information regarding other businesses that were located at a place or in a building of the scene, weather information for a place in the scene, travel direction information for a place or building in the scene, one or more reviews, rating, or commentary for at least one object of the scene, a dress requirement of a restaurant or event of the scene, a menu from a restaurant in the scene, a price list for a business in the scene, hours of operation for a place in the scene, information regarding a person in the scene, a link to a personal web site of a person in the scene, contact information for a person in the scene, biographical information for a person in the scene, membership information for a person in the scene, or information regarding a direction of the scene with respect to a street or a transportation route in the scene;

the decision information includes one or more of the following items;

one or more reviews, rating, or commentary by a professional critic or friend in the user'"'"'s social network for at least one object of the scene, a dress requirement of a restaurant of the scene, a menu from a restaurant in the scene, a price list for a business in the scene, hours of operation for a place in the scene, or a count of people or friends of the user who have visited a place in the scene; and

the actions further include one or more of the following;

communicating with a particular person via such other person'"'"'s communication device using the contact list on the mobile device or sending a video stream of one or more scene images/video captured with the user'"'"'s mobile device such that the sent video stream also includes the contextual information or the decision information for such one or more scene images/video.

8. The method as recited in claim 1, further comprising when a reference is selected by the user, initiating the referenced action.

9. The method as recited in claim 9, further comprising obtaining and presenting additional contextual and/or decision information after the reference is selected by the user.

10. A mobile device for providing information regarding one or more scenes, comprising:

a camera for capturing one or more scenes;

a display;

at least one a processor; and

at least one memory, the at least one processor and/or memory being configured for;

in response to determining that the camera is pointed at a scene including one or more objects, (i) displaying a visual representation of the scene in the display, and (ii) presenting in combination with the visual representation a plurality of options for selecting one of a plurality of user augmented reality modes that include an encyclopedia mode, a decision support mode, and an action mode;

in response to selection of the decision support mode, obtaining decision information related to a set of actions that can be taken with respect to an identity of the one or more objects and presenting the decision information in the display; and

in response to selection of the action mode after selection of the decision support mode, obtaining a set of references to the set of actions related to the presented decision information and that can be performed with respect to the identity of the one or more objects, and presenting the set of references in the display so that the references are selectable by a user to initiate the referenced actions, wherein the referenced actions include sending information regarding the one or more objects to another user using a contact list on the mobile device, or making a reservation at a place associated with the one or more objects including creating an event in a calendar on the mobile device.

11. The mobile device as recited in claim 10, the at least one processor and/or memory being configured for if the encyclopedia mode is selected while the mobile device is in decision support mode or action mode, switching to the encyclopedia mode by obtaining contextual information regarding an identity of the one or more objects and presenting the obtained contextual information in the display, wherein obtaining the contextual information, decision information, and set of references to actions is accomplished by sending the visual representation to one or more servers and receiving a second visual representation from the one or more servers whereby the second visual representation includes the contextual information, decision information, or the set of references, respectively, and wherein presenting the contextual information, decision information, and set of references comprises presenting the received second visual representation.

12. The mobile device as recited in claim 11, wherein the second visual representation is overlaid on the captured scene.

13. The mobile device as recited in claim 11, the at least one processor and/or memory being configured for:

determining an identity of each of the one or more objects by processing the visual representation of the scene using an object recognition technique; and

sending the determined identity for each of the one or more objects and a location of the mobile device to the one or more servers and wherein the contextual information, decision information, and set of references are based on the determined identity for each of the one or more objects and the location of the mobile device.

14. The mobile device as recited in claim 13, wherein the contextual information, decision information, and set of references are further based on a social network of the user associated with the mobile device.

15. The mobile device as recited in claim 13, wherein the contextual information, decision information, and set of references are further based on an indication as to a direction in which the camera is pointed with respect to a physical location of such camera.

16. At least one computer readable storage medium that is non-transitory and having computer program instructions stored thereon that are arranged such that when executed by one or more computing devices they cause the one or more computing devices to perform the following operations:

in response to determining that a camera of the mobile device is pointed at a scene including one or more objects, (i) displaying a visual representation of the scene in a display of the mobile device, and (ii) presenting in combination with the visual representation a plurality of options for selecting one of a plurality of user augmented reality modes that include an encyclopedia mode, a decision support mode, and an action mode;

in response to selection of the decision support mode, obtaining decision information related to a set of actions that can be taken with respect to an identity of the one or more objects and presenting the decision information in the display; and

in response to selection of the action mode after selection of the decision support mode, obtaining a set of references to the set of actions related to the presented decision information and that can be performed with respect to the identity of the one or more objects, and presenting the set of references in the display so that the references are selectable by a user to initiate the referenced actions, wherein the referenced actions include sending information regarding the one or more objects to another user using a contact list on the mobile device, or making a reservation at a place associated with the one or more objects including creating an event in a calendar on the mobile device.

17. The at least one computer readable storage medium as recited in claim 16, wherein the computer program instructions are further configured for if the encyclopedia mode is selected while the mobile device is in decision support mode or action mode, switching to the encyclopedia mode by obtaining contextual information regarding an identity of the one or more objects and presenting the obtained contextual information in the display, wherein obtaining the contextual information, decision information, and set of references to actions is accomplished by sending the visual representation to one or more servers and receiving a second visual representation from the one or more servers whereby the second visual representation includes the contextual information, decision information, or the set of references, respectively, and wherein presenting the contextual information, decision information, and set of references comprises presenting the received second visual representation, and wherein the second visual representation is overlaid on the captured scene.

18. The at least one computer readable storage medium as recited in claim 17, wherein the contextual information includes one or more of the following items:

historical information regarding a place in the scene, historical information regarding a building in the scene, time period information regarding other businesses that were located at a place or in a building of the scene, weather information for a place in the scene, travel direction information for a place or building in the scene, one or more reviews, rating, or commentary for at least one object of the scene, a dress requirement of a restaurant or event of the scene, a menu from a restaurant in the scene, a price list for a business in the scene, hours of operation for a place in the scene, information regarding a person in the scene, a link to a personal web site of a person in the scene, contact information for a person in the scene, biographical information for a person in the scene, membership information for a person in the scene, or information regarding a direction of the scene with respect to a street or a transportation route in the scene;

the decision information includes one or more of the following items;

one or more reviews, rating, or commentary by a professional critic or friend in the user'"'"'s social network for at least one object of the scene, a dress requirement of a restaurant of the scene, a menu from a restaurant in the scene, a price list for a business in the scene, hours of operation for a place in the scene, or a count of people or friends of the user who have visited a place in the scene; and

the actions further include one or more of the following;

communicating with a particular person via such other person'"'"'s communication device using the contact list on the mobile device or sending a video stream of one or more scene images/video captured with the user'"'"'s mobile device such that the sent video stream also includes the contextual information or the decision information for such one or more scene images/video.

19. The at least one computer readable storage medium as recited in claim 16, wherein the computer program instructions are further arranged to initiate the referenced action when a reference is selected by the user.

20. The at least one computer readable storage medium as recited in claim 19, wherein the computer program instructions are further arranged to present additional contextual and/or decision information, after the reference is selected by the user.

21. The at least one computer readable storage medium as recited in claim 16, wherein the computer program instructions are further configured for:

determining an identity of each of the one or more objects by processing the visual representation of the scene using an object recognition technique; and

sending the determined identity for each of the one or more objects and a location of the mobile device to the one or more servers and wherein the contextual information, decision information, and set of references are based on the determined identity for each of the one or more objects and the location of the mobile device.