Method and apparatus for selectively providing information on objects in a captured image

• US 9,292,764 B2
• Filed: 09/17/2013
• Issued: 03/22/2016
• Est. Priority Date: 09/17/2013
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Yun, Sungrack, Hwang, Kyu Woong, Cho, Jun-Cheol, Kim, Taesu, Jin, Minho, Cho, Yongwoo, Kim, Kang
• Primary Examiner(s): Desire, Gregory M
• Application Number:
  US14/029,431
• Publication Number:
  US 20150078667A1
• Time in Patent Office: 917 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/29 Geographical information databases
• CPC Class Codes: G06F 16/434 using image data, e.g. images, photos, pictures taken by a user
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/9537 Spatial or temporal dependent retrieval, e.g. spatiotemporal queries
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: H04W 4/02 Services making use of location information
• CPC Class Codes: H04W 4/029 Location-based management or tracking services

Abstract

A method for providing object information for a scene in a wearable computer is disclosed. In this method, an image of the scene is captured. Further, the method includes determining a current location of the wearable computer and a view direction of an image sensor of the wearable computer and extracting at least one feature from the image indicative of at least one object. Based on the current location, the view direction, and the at least one feature, information on the at least one object is determined. Then, the determined information is output.

Claims

1. A method of providing object information for a scene captured by a wearable computer, the method comprising:

capturing an image of the scene;

determining a location of the wearable computer and a view direction of an image sensor of the wearable computer;

extracting at least one feature from the image indicative of at least one object;

determining a plurality of information items corresponding to the at least one object based on the location, the view direction, and the at least one feature;

selecting at least one information item of the plurality of information items based on priority data; and

outputting the selected at least one information item.

2. The method of claim 1, wherein the plurality of information items corresponding to the at least one object include information related to an occluded object in the captured image.

3. The method of claim 1, wherein determining the plurality of information items corresponding to the at least one object comprises determining information related to an indoor scene of the at least one object.

4. The method of claim 1, further comprising retrieving the plurality of information items corresponding to the at least one object from the wearable computer.

5. The method of claim 1, further comprising retrieving the plurality of information items corresponding to the at least one object from an external device.

6. The method of claim 5, wherein the external device includes at least one of a mobile device and an external server.

7. The method of claim 1, wherein a plurality of predetermined features indicative of a plurality of objects are mapped to at least one of a plurality of locations, a plurality of view directions, and information related to the plurality of objects in an object database.

8. The method of claim 7, wherein determining the plurality of information items corresponding to the at least one object comprises:

selecting a plurality of candidate features from the plurality of predetermined features based on the location and the view direction;

identifying at least one of the plurality of candidate features based on the extracted at least one feature; and

retrieving the plurality of information items based on the identified at least one candidate feature.

9. The method of claim 8, further comprising determining a difference between the location and each of the plurality of locations associated with the plurality of candidate features, wherein selecting the plurality of candidate features is performed in response to determining that the difference is within a threshold.

10. The method of claim 8, further comprising determining a difference between the view direction and each of the plurality of view directions associated with the plurality of candidate features, wherein selecting the plurality of candidate features is performed in response to determining that the difference is within a threshold.

11. The method of claim 1, further comprising prioritizing the plurality of information items based on the priority data, wherein the priority data is determined based on at least one keyword received at the wearable computer.

12. The method of claim 1, wherein the priority data is determined based on at least one search query.

13. The method of claim 1, wherein the priority data is determined based on destination information.

14. The method of claim 1, wherein the at least one object comprises a plurality of objects, and wherein extracting the at least one feature comprises:

segmenting the image into a plurality of portions, wherein each portion of the plurality of portions corresponds to a particular object of the plurality of objects; and

extracting the at least one feature from at least one of the portions.

15. A wearable computer for providing object information for a scene, the wearable computer comprising:

an image sensor configured to capture an image of the scene;

a location determining unit configured to determine a location of the wearable computer;

a direction determining unit configured to determine a view direction of the image sensor of the wearable computer;

a feature extracting unit configured to extract at least one feature from the image indicative of at least one object;

an object information generator configured to;

determine a plurality of information en items corresponding to the at least one object based on the location, the view direction, and the at least one feature; and

select at least one information item of the plurality of information items based on priority data; and

an output unit configured to output the selected at least one information item.

16. The wearable computer of claim 15, wherein the plurality of information items corresponding to the at least one object includes information related to an occluded object in the captured image.

17. The wearable computer of claim 15, wherein the object information generator is further configured to determine information related to an indoor scene of the at least one object.

18. The wearable computer of claim 15, wherein the object information generator is further configured to retrieve the plurality of information items corresponding to the at least one object from the wearable computer.

19. The wearable computer of claim 15, wherein the object information generator is further configured to retrieve the plurality of information items corresponding to the at least one object from an external device.

20. The wearable computer of claim 19, wherein the external device includes at least one of a mobile device and an external server.

21. The wearable computer of claim 15, wherein a plurality of predetermined features indicative of a plurality of objects are mapped to at least one of a plurality of locations, a plurality of view directions, and information related to the plurality of objects in an object database.

22. The wearable computer of claim 21, wherein the object information generator is configured to select a plurality of candidate features from the plurality of predetermined features based on the location and the view direction, to identify at least one of the candidate features based on the extracted at least one feature, and to retrieve the plurality of information items based on the identified at least one candidate feature.

23. The wearable computer of claim 22, wherein the object information generator is configured to select the plurality of candidate features in response to determining that a difference between the location and each of the plurality of locations associated with the plurality of candidate features is within a threshold.

24. The wearable computer of claim 22, wherein the object information generator is configured to select the plurality of candidate features in response to determining that a difference between the view direction and each of the plurality of view directions associated with the plurality of candidate features is within a threshold.

25. The wearable computer of claim 15, wherein the priority data is determined based on at least one keyword received in the wearable computer.

26. The wearable computer of claim 15, wherein the priority data is determined based on at least one search query.

27. The wearable computer of claim 15, wherein the priority data is determined based on destination information.

28. The wearable computer of claim 15, wherein the feature extracting unit is further configured to segment the captured image into a plurality of portions, and to extract the at least one feature from at least one of the portions, wherein the at least one object comprises a plurality of objects, and wherein each portion of the plurality of portions corresponds to a particular object of the plurality of objects.

29. The wearable computer of claim 15, wherein the object information generator is configured to determine the priority data by accessing a context information database in a storage unit.

30. The wearable computer of claim 29, wherein the priority data corresponds to preferences of a user, and wherein the priority data is determined based on historical data, past user inputs, or a combination thereof.

31. A wearable computer for providing object information for a scene, comprising:

means for capturing an image of the scene;

means for determining a location of the wearable computer;

means for determining a view direction of an image sensor of the wearable computer;

means for extracting at least one feature from the image indicative of at least one object;

means for determining a plurality of information items corresponding to the at least one object based on the location, the view direction, and the at least one feature;

means for selecting at least one information item of the plurality of information items based on priority data; and

means for outputting the selected at least one information item.

32. The wearable computer of claim 31, wherein the plurality of information items corresponding to the at least one object includes information related to an occluded object in the captured image.

33. The wearable computer of claim 31, wherein the means for determining the plurality of information items corresponding to the at least one object comprises means for determining information related to an indoor scene of the at least one object.

34. The wearable computer of claim 31, wherein the means for determining the plurality of information items corresponding to the at least one object comprises means for retrieving the plurality of information items corresponding to the at least one object from the wearable computer.

35. The wearable computer of claim 31, wherein the means for determining the plurality of information items corresponding to the at least one object comprises means for retrieving the plurality of information items corresponding to the at least one object from an external device.

36. The wearable computer of claim 35, wherein the external device includes at least one of a mobile device and an external server.

37. The wearable computer of claim 31, wherein a plurality of predetermined features indicative of a plurality of objects are mapped to at least one of a plurality of locations, a plurality of view directions, and information related to the plurality of objects in an object database.

38. The wearable computer of claim 37, wherein the means for determining the plurality of information en items corresponding to the at least one object comprises:

means for selecting a plurality of candidate features from the plurality of predetermined features based on the location and the view direction;

means for identifying at least one of the candidate features based on the extracted at least one feature; and

means for retrieving the plurality of information items based on the identified at least one candidate feature.

39. The wearable computer of claim 38, wherein the plurality of candidate features are selected if a difference between the location and each of the plurality of locations associated with the plurality of candidate features is within a threshold.

40. The wearable computer of claim 38, wherein the plurality of candidate features are selected if a difference between the view direction and each of the plurality of view directions associated with the plurality of candidate features is within a threshold.

41. A non-transitory computer-readable storage medium comprising instructions for providing object information for a scene in a wearable computer, the instructions causing a processor of the wearable computer to perform operations of:

capturing an image of the scene;

determining a location of the wearable computer and a view direction of an image sensor of the wearable computer;

extracting at least one feature from the image indicative of at least one object;

determining a plurality of information items corresponding to the at least one object based on the location, the view direction, and the at least one feature;

selecting at least one information item of the plurality of information items based on priority data; and

outputting the selected at least one information item.

42. The medium of claim 41, wherein the plurality of information items corresponding to the at least one object includes information related to an occluded object in the captured image.

43. The medium of claim 41, wherein determining the plurality of information items corresponding to the at least one object comprises determining information related to an indoor scene of the at least one object.

44. The medium of claim 41, further comprising retrieving the plurality of information items from the wearable computer.

45. The medium of claim 41, further comprising retrieving the plurality of information items from an external device.

46. The medium of claim 45, wherein the external device includes at least one of a mobile device and an external server.

47. The medium of claim 41, wherein a plurality of predetermined features indicative of a plurality of objects are mapped to at least one of a plurality of locations, a plurality of view directions, and information related to the plurality of objects in an object database.

48. The medium of claim 47, wherein determining the plurality of information items corresponding to the at least one object comprises:

selecting a plurality of candidate features from the plurality of predetermined features based on the location and the view direction;

identifying at least one of the candidate features based on the extracted at least one feature; and

retrieving the plurality of information items based on the identified at least one candidate feature.

49. The medium of claim 48, wherein the plurality of candidate features are selected if a difference between the location and each of the plurality of locations associated with the plurality of candidate features is within a threshold.

50. The medium of claim 48, wherein the plurality of candidate features are selected if a difference between the view direction and each of the plurality of view directions associated with the plurality of candidate features is within a threshold.