Scalable visual search system simplifying access to network and device functionality

• US 8,849,821 B2
• Filed: 11/04/2006
• Issued: 09/30/2014
• Est. Priority Date: 11/04/2005
• Status: Active Grant

• Current Assignee: Nokia Technologies Oy (Nokia Corporation)
• Original Assignee: Nokia Corporation
• Inventors: Schloter, Philipp
• Primary Examiner(s): TRUONG, CAM Y T
• Application Number:
  US11/592,460
• Publication Number:
  US 20070106721A1
• Time in Patent Office: 2,887 Days
• Field of Search: 707/3, 707/737, 707/736, 382/305, 718/105
• US Class Current: 707/736
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: H04L 65/1101 Session protocols
• CPC Class Codes: H04L 65/612 for unicast
• CPC Class Codes: H04W 4/02 Services making use of location information
• CPC Class Codes: H04W 4/16 Communication-related supplementary services, e.g. call-transfer or call-hold

Abstract

A method, apparatus and computer-readable storage medium are provided for providing a list of candidates that may provide information requested by a user. In this regard, an indication of information desired by a user may be received. The indication of the desired information may include media content associated with one or more objects in a real world and/or meta-information providing one or more characteristics associated with the media content, a mobile device and/or the user. A list of candidates for the desired information may then be presented to the user, such as upon a mobile device of the user. In response to an indication of a selection of the user from among the list of candidates or otherwise, an association may be created between the selection of the user and the indication of information desired by the user.

Claims

1. A method comprising:

receiving in real time at a mobile device an indication of information desired by a user once the mobile device is pointed to one or more objects in a real world by the user, wherein the indication of the desired information comprises live visual media content presenting the one or more objects therein, meta-information providing one or more properties of the media content, and one or more characteristics associated with at least one of the mobile device and the user;

causing, at least in part, providing of a list of candidates for the desired information for presentation of the list on the mobile device based, at least in part, on at least one of the live visual media content and the meta-information;

determining a number of times that a candidate in the list of candidates has been selected by the user, and storing the number of selected times in a log per user; and

adjusting, based on the determined number of times, a priority of the selected candidate in the list of candidates for information desired by the user,

wherein the list of candidates is concurrently presented with the live visual media on one screen to the user for selection in real time without requiring the user to make any clicks from the time the user starts pointing the mobile device to the objects.

2. The method of claim 1, further comprising:

searching a database for data matching the indication of the desired information as the list of candidates; and

constraining a scope of a search based on at least one of the media content and the meta-information.

3. The method of claim 2, further comprising:

identifying the one or more objects based upon one or more recognition parameters,

wherein the search in the database for the list is further based upon one or more identified objects,

each candidate in the list is a link that maps the one or more objects, the one or more characteristics, or a combination thereof to a desired information item, and

the desired information item is a web site, a phone number, a message template, an application, an application state, a web page entry, a phone functionality, a keyword, a computer functionality, a document, a television program, a radio program, a music recording, or a combination thereof.

4. The method of claim 3, further comprising:

facilitating load balancing of the identifying among the mobile device and one or more servers based on the media content, the meta-information, one or more network characteristics, or a combination thereof.

5. The method of claim 2, further comprising:

creating a plurality of links automatically based on the live visual media content, if no matching data is found in the database;

associating each of the plurality of links with the live visual media content, the meta-information, or a combination thereof; and

storing the plurality of links in the database.

6. The method of claim 5, wherein each of the plurality of links has statistics data and one or more properties, the one or more properties comprising at least one of a title, a timestamp, a creator identifier, or access information.

7. The method of claim 2, further comprising:

creating an intermediate representation of the media content, the intermediate representation being used to search the database for data matching the indication of the desired information,

wherein the scope of the search is constrained by one or more other objects in the real world that are geographically near-by the one or more objects.

8. The method of claim 2, further comprising:

associating one or more links executed by the user in association with the live visual media content, the meta-information, or a combination thereof; and

storing the one or more links in the database.

9. The method of claim 1, wherein:

the list of candidates are overlaid on top of the live visual media content within a viewfinder of the mobile device; and

the list of candidates is presented to the user without requiring the user to make any clicks to capture an image of the objects, from the time the user starts pointing the mobile device to the objects.

10. The method of claim 9, further comprising:

in response to a subsequent search query of the user, searching within the stored association and the at least one of the media content and the meta-information to assemble a list of candidates as search results.

11. The method of claim 9, wherein:

at least one candidate from the list has a title that includes a preview of the desired information; and

the method further comprises causing display of the title including the preview of the desired information when presenting the list of candidates to the user,

wherein the displaying provides user access to the preview of the desired information when the user points the mobile device to one or more other objects.

12. The method of claim 1, further comprising:

determining that the user has not selected a candidate from the list;

presenting a directory of information items to the user;

allowing the user to select an information item from the directory;

creating a link mapping the selected information item to the received indication of the desired information; and

storing the link in a database.

13. The method of claim 3, further comprising:

periodically modifying links by re-linking previously linked objects, characteristics, or a combination thereof to new or existing information items via a system management application controlled by the user or an administrator.

14. The method of claim 1, wherein the meta-information comprises one or more personal characteristics of the user that include age, native language, occupation, or a combination thereof, and

wherein the method further comprises selecting the candidates at least partially based upon the personal characteristics of the user.

15. The method of claim 1, wherein the meta-information comprises one or more online behavior characteristics of the user, one or more social network profiles of the user, or a combination thereof, and wherein the method further comprises selecting the candidates at least partially based upon the online behavior characteristics of the user, the social network profiles of the user, or a combination thereof.

16. The method of claim 1, further comprising selecting the candidates by searching a link database to identify one or more candidates associated with the indication of the desired information and generating one or more candidates based upon one or more predefined rules.

17. The method of claim 1, wherein providing the list of candidates comprises concurrently presenting both the list of candidates and the media content.

18. The method of claim 1, wherein providing the list of candidates comprises categorizing the candidates into categories including a global category open to all users and a personal category available only to the respective user, and wherein creating an association comprises identifying the association to be global and open to all users or to be personal to the respective user.

19. The method of claim 1, further comprising:

causing, at least in part, reception of an indication of a selection of the user in response to the providing of the list of candidates;

creating an association between the selection of the user and the indication of information desired by the user; and

causing, at least in part, storage of the association and at least one of the media content and the meta-information,

wherein the media content is produced by the mobile device and comprises one or more digital photographs, one or more videos, one or more audio recordings, or a combination thereof, and

the meta-information comprises one or more characteristics of the media content, one or more environmental characteristics associated with the mobile device, one or more geographic characteristics associated with the mobile device, one or more personal characteristics associated with the user, one or more online behavior characteristics of the user, or a combination thereof.

20. The method of claim 19, further comprising:

caching the association and the at least one of the media content and the meta-information at the mobile device; and

synchronizing the association and the at least one of the media content and the meta-information stored at a server with the association and the at least one of the media content and the meta-information cached at the mobile device.

21. The method of claim 1, further comprising:

tagging at least sensor data to the media content as the meta-information, the sensor data including one or more data of media object proximity, one or more body parameters of one or more of the objects in a real world, one or more accelerations of one or more of the objects, or a combination thereof; and

causing, at least in part, providing of the list of candidates for the desired information for presentation on the mobile device of the user further based, at least in part, on the media content and the sensor data.

22. The method of claim 1, further comprising:

receiving a visual media content that is captured by the user device and includes therein one or more recipients of the one or more objects in the real world; and

causing, at least in part, billing for delivering the one or more objects to the one or more recipients.

23. A non-transitory computer-readable storage medium carrying one or more sequences of one or more instructions which, when executed by one or more processors, cause the apparatus to at least perform the following steps:

receiving in real time at a mobile device an indication of information desired by a user once the mobile device is pointed to one or more objects in a real world by the user, wherein the indication of the desired information comprises live visual media content presenting the one or more objects therein, meta-information providing one or more properties of the media content, and one or more characteristics associated with at least one of the mobile device and the user;

causing, at least in part, providing of a list of candidates for the desired information for presentation of the list on the mobile device based, at least in part, on at least one of the live visual media content and the meta-information;

determining a number of times that a candidate in the list of candidates has been selected, and storing the number of selected times in a log per user; and

adjusting, based on the determined number of times, a priority of the selected candidate in the list of candidates for information desired by the user,

wherein the list of candidates is concurrently presented with the live visual media on one screen to the user for selection in real time without requiring the user to make any clicks from the time the user starts pointing the mobile device to the objects.

24. The non-transitory computer-readable storage medium of claim 23, wherein the meta-information comprises one or more personal characteristics of the user that include age, native language, occupation, or a combination thereof, and wherein the apparatus is further cause to perform selecting the candidates at least partially based upon the personal characteristics of the user.

25. The non-transitory computer-readable storage medium of claim 23, wherein the meta-information comprises one or more online behavior characteristics of the user, and wherein the apparatus is further cause to perform selecting the candidates at least partially based upon the online behavior characteristics of the user.

26. The non-transitory computer-readable storage medium of claim 23, wherein the apparatus is further cause to perform selecting the candidates by searching a link database to identify one or more candidates associated with the indication of the desired information and generate one or more candidates based upon one or more predefined rules.

27. The non-transitory computer-readable storage medium of claim 23, wherein the apparatus is further cause to perform providing the list of candidates comprises concurrently presenting both the list of candidates and the media content.

28. The non-transitory computer-readable storage medium of claim 23, wherein the apparatus provides the list of candidates by categorizing the candidates into categories including a global category open to all users and a personal category available only to the respective user, and wherein the apparatus creates an association by identifying the association to be global and open to all users or to be personal to the respective user.

29. The non-transitory computer-readable storage medium of claim 23, wherein the apparatus is further cause to perform:

receiving an indication of a selection of the user in response to the providing of the list of candidates;

creating an association between the selection of the user and the indication of information desired by the user; and

storing the association and at least one of the media content and the meta-information;

wherein the media content is produced by the mobile device and comprises one or more digital photographs, one or more videos, one or more audio recordings, or a combination thereof, and

the meta-information comprises one or more characteristics of the media content, one or more environmental characteristics associated with the mobile device, one or more geographic characteristics associated with the mobile device, one or more personal characteristics associated with the user, one or more online behavior characteristics of the user, or a combination thereof.

30. An apparatus comprising:

at least one processor; and

at least one memory including computer program code,

the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to perform at least the following,

receive in real time at a mobile device an indication of information desired by a user once the mobile device is pointed to one or more objects in a real world by the user, wherein the indication of the desired information comprises live visual media content presenting the one or more objects therein, meta-information providing one or more properties of the media content, and one or more characteristics associated with at least one of the mobile device and the user;

cause, at least in part, providing of a list of candidates for the desired information for presentation of the list on the mobile device based, at least in part, on at least one of the live visual media content and the meta-information;

determine a number of times that a candidate in the list of candidates has been selected, and storing the number of selected times in a log per user; and

adjust, based on the determined number of times, a priority of the selected candidate in the list of candidates for information desired by the user,

wherein the list of candidates is concurrently presented with the live visual media on one screen to the user for selection in real time without requiring the user to make any clicks from the time the user starts pointing the mobile device to the objects.

31. The apparatus of claim 30, wherein the apparatus is the mobile device or a server coupled to the mobile device via a network.

32. The apparatus of claim 30, wherein the meta-information comprises one or more personal characteristics of the user that include age, native language, occupation, or a combination thereof, and wherein the apparatus is further caused to select the candidates at least partially based upon the personal characteristics of the user.

33. The apparatus of claim 30, wherein the meta-information comprises one or more online behavior characteristics of the user, and wherein the apparatus is further caused to select the candidates at least partially based upon the online behavior characteristics of the user.

34. The apparatus of claim 30, wherein the apparatus is further caused to select the candidates by searching a link database to identify one or more candidates associated with the indication of the desired information and generate one or more candidates based upon one or more predefined rules.

35. The apparatus of claim 30, wherein the apparatus is further caused to provide the list of candidates by concurrently presenting both the list of candidates and the media content.

36. The apparatus of claim 30, wherein the apparatus is further caused to provide the list of candidates by categorizing the candidates into categories including a global category open to all users and a personal category available only to the respective user, and to create an association by identifying the association to be global and open to all users or to be personal to the respective user.

37. The apparatus of claim 30, wherein the apparatus is further caused to:

receive an indication of a selection of the user in response to the providing of the list of candidates;

create an association between the selection of the user and the indication of information desired by the user; and

store the association and at least one of the media content and the meta-information,

wherein the media content is produced by the mobile device and comprises one or more digital photographs, one or more videos, one or more audio recordings, or a combination thereof, and

the meta-information comprises one or more characteristics of the media content, one or more environmental characteristics associated with the mobile device, one or more geographic characteristics associated with the mobile device, one or more personal characteristics associated with the user, one or more online behavior characteristics of the user, or a combination thereof.

38. The apparatus of claim 37, wherein the apparatus is further caused to:

cache the association and the at least one of the media content and the meta-information at the mobile device; and

synchronize the association and the at least one of the media content and the meta-information stored at the apparatus with the association and the at least one of the media content and the meta-information cached at the mobile device, the apparatus being a server.