Providing data associated with relationships between individuals and images

• US 9,773,285 B2
• Filed: 01/01/2012
• Issued: 09/26/2017
• Est. Priority Date: 03/08/2011
• Status: Active Grant

• Current Assignee: Bank of America Corp.
• Original Assignee: Bank of America Corp.
• Inventors: Calman, Matthew A., Ross, Erik Stephen, Hamilton, Alfred
• Primary Examiner(s): Bahl, Sangeeta
• Application Number:
  US13/342,050
• Publication Number:
  US 20120229657A1
• Time in Patent Office: 2,095 Days
• Field of Search: 705 11, 705 732, 705 267, 705 2662
• CPC Class Codes: G06Q 10/00 Administration; Management
• CPC Class Codes: G06Q 30/00 Commerce
• CPC Class Codes: G06Q 50/01 Social networking

Abstract

In general terms, embodiments of the present invention relate to methods and apparatuses for providing information regarding relationships between individuals, who may be associated with a user of a mobile device, and objects (e.g., products, locations, businesses, etc.) identified in an image. In some embodiments, this image may be captured by a mobile device. For example, in some embodiments, a method is provided that includes: (a) receiving first information associated with an image, where the image was captured by a mobile device operated by a user; (b) receiving second information regarding one or more individuals; (c) determining, based at least partially on the first and second information, one or more relationships between information regarding individuals and information associated with the image; and (d) presenting, via the mobile device of the user, information associated with the one or more relationships.

Claims

1. A method for providing relationship information in real-time, the method comprising:

providing an augmented reality (AR) application to a user for installation on a mobile device of the user;

establishing a communication link with the mobile device of the user, wherein establishing a communication link further comprises establishing a data channel with a storage medium associated with the mobile device;

capturing, continuously, a real-time video stream of a surrounding environment using a video capture device associated with a mobile device of the user, wherein the real-time video stream comprises an image of one or more goods or service providers;

receiving the image, wherein receiving further comprises receiving first information associated with the image, wherein the first information comprises an identifiable feature associated with one or more objects in the image and a location of the identifiable feature in the retrieved image, wherein the one or more objects comprises at least a product, a business, and a location, wherein the identifiable feature associated with the product comprises at least a size, a shape, a color of the product'"'"'s packaging, a product logo, a bar code information, and a product name, wherein the identifiable feature associated with the business comprises a business logo, a decor of the business and one or more trademarks associated with the business, wherein the identifiable feature associated with the location comprises a street sign, an identifiable landmark, a street number and a signal associated with a geographic position device associated with the mobile device;

receiving second information regarding one or more individuals, wherein receiving further comprises retrieving the second information from at least one or more social media accounts associated with the user, a contact list comprising contact information for the one or more individuals stored on the mobile device of the user, and a user input;

determining, based at least partially on the first and second information, a plurality of relationships between the one or more individuals and the one or more objects in the image;

initiating a presentation of a first user interface on the mobile device via the established data channel, wherein initiating activates the AR application to cause the mobile device to display a list of the plurality of relationships between the one or more individuals and the image on the first user interface, wherein the user interface comprises a list of the plurality of relationships between the one or more individuals and the image, wherein the list is prioritized based on a predetermined strength associated with each of the plurality of relationships whereby a relationship with a high strength is ranked higher in the list of the plurality of relationships than a relationship with a low strength;

receiving, via the first user interface on the mobile device and the established data channel associated with the mobile device, a user selection of at least one of the plurality of relationships between the one or more individuals and the image;

initiating, via the established data channel, a presentation of a second user interface for display on the mobile device, wherein the second user interface comprises a relationship information associated with the user selected relationship from the plurality of relationships, wherein presenting the relationship information comprises superimposing the relationship information over a real-time video stream of the one or more individuals that is captured by the mobile device, wherein the real-time video stream comprises at least one of the one or more objects;

determining that the mobile device is associated with at least one trigger event, wherein the trigger event comprises;

(i) the mobile device being exposed to predetermined low levels of illumination for a first predetermined extended period of time, (ii) the mobile device being stationary for a second predetermined extended period of time or (iii) a power source of the mobile device being associated with predetermined low power levels;

disengaging the continuous capture of the real-time video stream of the surrounding environment, based on the at least one trigger event; and

re-engaging, automatically, the continuous capture of the real-time video stream of the surrounding environment, based on determining that the mobile device is not associated with the at least one trigger event.

2. The method of claim 1, wherein receiving first image further comprises:

identifying one or more objects associated with the image, the image is part of a real-time video stream;

receiving metadata associated with the one or more objects, the received metadata comprises at least location information associated with the one or more objects;

comparing the received metadata associated with the one or more objects with one or more stored metadata in a metadata database, the one or more stored metadata associated with the one or more individuals; and

determining a relationship between the image and the one or more individuals based at least partially on comparing the received metadata with the one or more stored metadata.

3. The method of claim 1, wherein the image is a still image captured by a digital camera.

4. The method of claim 1, wherein the first information associated with the image is the image.

5. The method of claim 1, wherein the mobile device is a mobile phone.

6. The method of claim 1, wherein the object is associated with a product.

7. The method of claim 1, wherein the object is associated with a business.

8. The method of claim 1, wherein the object is associated with a location.

9. The method of claim 1, further comprising:

identifying one or more individuals associated with the user.

10. The method of claim 1, wherein the receiving the second information comprises:

identifying one or more objects associated with the image;

receiving metadata associated with the one or more objects, wherein the metadata comprises information associated with the one or more objects;

comparing the received metadata associated with the one or more objects with social network information associated with the one or more individuals; and

determining a relationship between the image and the one or more individuals based at least partially on comparing the received metadata with the social network information.

11. The method of claim 1, wherein the receiving the second information comprises:

receiving the second information from a contact list associated with the user.

12. The method of claim 1, wherein the receiving the second information comprises:

receiving the second information from a database that comprises financial transaction history of one or more individuals.

13. The method of claim 12, wherein the transaction history comprises purchase history and return history associated with products and services.

14. The method of claim 1, wherein the receiving the second information comprises:

receiving the second information from one or more message sharing networks.

15. The method of claim 1, wherein the receiving the second information comprises:

allowing the user to input the second information, wherein the second information comprises a name of an individual.

16. The method of claim 15, further comprising:

searching, on a network, for information associated with the individual.

17. The method of claim 1, wherein the determining the relationship comprises:

comparing the first information with the second information; and

determining a match between the first information and the second information.

18. The method of claim 17, wherein the determining a match comprises:

determining that an individual associated with the second information made a transaction associated with an object depicted in the first information.

19. The method of claim 17, wherein the determining a match comprises:

determining that an individual associated with the second information shared a message associated with an object depicted in the first information.

20. The method of claim 17, wherein the determining a match comprises:

determining that an individual associated with the second information was located at an object depicted in the first information.

21. The method of claim 1, wherein presenting the relationship information comprises:

displaying a relationship indicator on a display of the mobile device.

22. The method of claim 21, wherein the relationship indicator comprises a border that surrounds a depiction of an object.

23. The method of claim 21, wherein the relationship indicator is selectable by the user.

24. The method of claim 23, wherein the relationship indicator, upon being selected, provides information about an object associated with the image.

25. The method of claim 23, wherein the relationship indicator, upon being selected, provides information about the individual.

26. An apparatus for providing relationship information in real-time, the apparatus comprising:

a memory;

a processor; and

a module stored in the memory, executable by the processor, and configured to;

provide an augmented reality (AR) application to a user for installation on a mobile device of the user;

establish a communication link with the mobile device of the user, wherein establishing a communication link further comprises establishing a data channel with a storage medium associated with the mobile device;

capture continuously, a real-time video stream of a surrounding environment using a video capture device associated with a mobile device of the user, wherein the real-time video stream comprises an image of one or more goods or service providers;

receive the image, wherein receiving further comprises receiving first information associated with the image, wherein the first information comprises an identifiable feature associated with one or more objects in the image and a location of the identifiable feature in the retrieved image, wherein the one or more objects comprises at least a product, a business, and a location, wherein the identifiable feature associated with the product comprises at least a size, a shape, a color of the product'"'"'s packaging, a product logo, a bar code information, and a product name, wherein the identifiable feature associated with the business comprises a business logo, a decor of the business and one or more trademarks associated with the business, wherein the identifiable feature associated with the location comprises a street sign, an identifiable landmark, a street number and a signal associated with a geographic position device associated with the mobile device;

receive second information regarding one or more individuals, wherein receiving further comprises retrieving the second information from at least one or more social media accounts associated with the user, a contact list comprising contact information for the one or more individuals stored on the mobile device of the user, and a user input;

determine, based at least partially on the first and second information, a plurality of relationships between the one or more individuals and the one or more objects in the image;

initiate a presentation of a first user interface on the mobile device via the established data channel, wherein initiating activates the AR application to cause the mobile device to display a list of the plurality of relationships between the one or more individuals and the image on the first user interface, wherein the user interface comprises a list of the plurality of relationships between the one or more individuals and the image, wherein the list is prioritized based on a predetermined strength associated with each of the plurality of relationships whereby a relationship with a high strength is ranked higher in the list of the plurality of relationships than a relationship with a low strength;

receive, via the first user interface on the mobile device and the established data channel associated with the mobile device, a user selection of at least one of the plurality of relationships between the one or more individuals and the image;

initiate, via the established data channel, a presentation of a second user interface for display on the mobile device, wherein the second user interface comprises a relationship information associated with the user selected relationship from the plurality of relationships, wherein presenting the relationship information comprises superimposing the relationship information over a real-time video stream of the one or more individuals that is captured by the mobile device, wherein the real-time video stream comprises at least one of the one or more objects;

determine that the mobile device is associated with at least one trigger event, wherein the trigger event comprises;

(i) the mobile device being exposed to predetermined low levels of illumination for a first predetermined extended period of time, (ii) the mobile device being stationary for a second predetermined extended period of time or (iii) a power source of the mobile device being associated with predetermined low power levels;

disengage the continuous capture of the real-time video stream of the surrounding environment, based on the at least one trigger event; and

re-engage, automatically, the continuous capture of the real-time video stream of the surrounding environment, based on determining that the mobile device is not associated with the at least one trigger event.

27. The apparatus of claim 26, wherein the module is further configured to cause the processor to:

identify one or more objects associated with the image, wherein the image is part of a real-time video stream;

receive metadata associated with the one or more objects, the received metadata comprises at least location information associated with the one or more objects;

compare the received metadata associated with the one or more objects with one or more stored metadata in a metadata database, the one or more stored metadata associated with the one or more individuals; and

determine a relationship between the image and the one or more individuals based at least partially on comparing the received metadata with the one or more stored metadata.

28. The apparatus of claim 26, wherein the image is a still image captured by a digital camera.

29. The apparatus of claim 26, wherein the first information associated with the image is the image.

30. The apparatus of claim 26, wherein the mobile device is a mobile phone.

31. The apparatus of claim 26, wherein the object is associated with a product.

32. The apparatus of claim 26, wherein the object is associated with a business.

33. The apparatus of claim 26, wherein the object is associated with a location.

34. The apparatus of claim 26, wherein the module is further configured to:

identify one or more individuals associated with the user.

35. The apparatus of claim 26, wherein the module configured to receive the second information is further configured to:

identify one or more objects associated with the image;

receive metadata associated with the one or more objects, wherein the metadata comprises information associated with the one or more objects;

compare the received metadata associated with the one or more objects with social network information associated with the one or more individuals; and

determine a relationship between the image and the one or more individuals based at least partially on comparing the received metadata with the social network information.

36. The apparatus of claim 26, wherein the module configured to receive the second information is further configured to:

receive the second information from a contact list associated with the user.

37. The apparatus of claim 26, wherein the module configured to receive the second information is further configured to:

receive the second information from a database that comprises financial transaction history of one or more individuals.

38. The apparatus of claim 37, wherein the transaction history comprises purchase history and return history associated with products and services.

39. The apparatus of claim 26, wherein the module configured to receive the second information is further configured to:

receive the second information from one or more message sharing networks.

40. The apparatus of claim 26, wherein the module configured to receive the second information is further configured to:

allow the user to input the second information, wherein the second information comprises a name of an individual.

41. The apparatus of claim 40, wherein the module is further configured to:

search, on a network, for information associated with the individual.

42. The apparatus of claim 26, wherein the module configured to determine the relationship is further configured to:

compare the first information with the second information; and

determine a match between the first information and the second information.

43. The apparatus of claim 42, wherein the module configured to determine a match is further configured to:

determine that an individual associated with the second information made a transaction associated with an object depicted in the first information.

44. The apparatus of claim 42, wherein the module configured to determine a match is further configured to:

determine that an individual associated with the second information shared a message associated with an object depicted in the first information.

45. The apparatus of claim 42, wherein the module configured to determine a match is further configured to:

determine that an individual associated with the second information was located at an object depicted in the first information.

46. The apparatus of claim 26, wherein the module configured to present the relationship information is further configured to:

display a relationship indicator on a display of the mobile device.

47. The apparatus of claim 46, wherein the relationship indicator comprises a border that surrounds a depiction of an object.

48. The apparatus of claim 46, wherein the relationship indicator is selectable by the user.

49. The apparatus of claim 48, wherein the relationship indicator, upon being selected, provides information about an object associated with the image.

50. The apparatus of claim 48, wherein the relationship indicator, upon being selected, provides information about the individual.

51. An computer program product for providing relationship information in real-time, the computer program product comprising:

a non-transitory computer-readable medium comprising a set of codes for causing a computer to;

provide an augmented reality (AR) application to a user for installation on a mobile device of the user;

establish a communication link with the mobile device of the user, wherein establishing a communication link further comprises establishing a data channel with a storage medium associated with the mobile device;

capture continuously, a real-time video stream of a surrounding environment using a video capture device associated with a mobile device of the user, wherein the real-time video stream comprises an image of one or more goods or service providers;

receive the image, wherein receiving further comprises receiving first information associated with the image, wherein the first information comprises an identifiable feature associated with one or more objects in the image and a location of the identifiable feature in the retrieved image, wherein the one or more objects comprises at least a product, a business, and a location, wherein the identifiable feature associated with the product comprises at least a size, a shape, a color of the product'"'"'s packaging, a product logo, a bar code information, and a product name, wherein the identifiable feature associated with the business comprises a business logo, a decor of the business and one or more trademarks associated with the business, wherein the identifiable feature associated with the location comprises a street sign, an identifiable landmark, a street number and a signal associated with a geographic position device associated with the mobile device;

receive second information regarding one or more individuals, wherein receiving further comprises retrieving the second information from at least one or more social media accounts associated with the user, a contact list comprising contact information for the one or more individuals stored on the mobile device of the user, and a user input;

determine, based at least partially on the first and second information, a plurality of relationships between the one or more individuals and the one or more objects in the image;

initiate a presentation of a first user interface on the mobile device via the established data channel, wherein initiating activates the AR application to cause the mobile device to display a list of the plurality of relationships between the one or more individuals and the image on the first user interface, wherein the user interface comprises a list of the plurality of relationships between the one or more individuals and the image, wherein the list is prioritized based on a predetermined strength associated with each of the plurality of relationships whereby a relationship with a high strength is ranked higher in the list of the plurality of relationships than a relationship with a low strength;

receive, via the first user interface on the mobile device and the established data channel associated with the mobile device, a user selection of at least one of the plurality of relationships between the one or more individuals and the image;

initiate, via the established data channel, a presentation of a second user interface for display on the mobile device, wherein the second user interface comprises a relationship information associated with the user selected relationship from the plurality of relationships, wherein presenting the relationship information comprises superimposing the relationship information over a real-time video stream of the one or more individuals that is captured by the mobile device, wherein the real-time video stream comprises at least one of the one or more objects;

determine that the mobile device is associated with at least one trigger event, wherein the trigger event comprises;

(i) the mobile device being exposed to predetermined low levels of illumination for a first predetermined extended period of time, (ii) the mobile device being stationary for a second predetermined extended period of time or (iii) a power source of the mobile device being associated with predetermined low power levels;

disengage the continuous capture of the real-time video stream of the surrounding environment, based on the at least one trigger event; and

re-engage, automatically, the continuous capture of the real-time video stream of the surrounding environment, based on determining that the mobile device is not associated with the at least one trigger event.

52. The computer program product of claim 51 further comprises:

identify one or more objects associated with the image, wherein the image is part of a real-time video stream;

receive metadata associated with the one or more objects, the received metadata comprises at least location information associated with the one or more objects;

compare the received metadata associated with the one or more objects with one or more stored metadata in a metadata database, the one or more stored metadata associated with the one or more individuals; and

determine a relationship between the image and the one or more individuals based at least partially on comparing the received metadata with the one or more stored metadata.

53. The computer program product of claim 51, wherein the image is a still image captured by a digital camera.

54. The computer program product of claim 51, wherein the first information associated with the image is the image.

55. The computer program product of claim 51, wherein the mobile device is a mobile phone.

56. The computer program product of claim 51, wherein the object is associated with a product.

57. The computer program product of claim 51, wherein the object is associated with a business.

58. The computer program product of claim 51, wherein the object is associated with a location.

59. The apparatus of claim 51, wherein the set of codes further causes a computer to:

identify one or more individuals associated with the user.

60. The computer program product of claim 51, wherein the set of codes that cause a computer to receive the second information further causes a computer to:

identifying one or more objects associated with the image;

receiving metadata associated with the one or more objects, wherein the metadata comprises information associated with the one or more objects;

comparing the received metadata associated with the one or more objects with social network information associated with the one or more individuals; and

determining a relationship between the image and the one or more individuals based at least partially on comparing the received metadata with the social network information.

61. The computer program product of claim 51, wherein the set of codes that cause a computer to receive the second information further causes a computer to:

receive the second information from a contact list associated with the user.

62. The computer program product of claim 51, wherein the set of codes that cause a computer to receive the second information further causes a computer to:

receive the second information from a database that comprises financial transaction history of one or more individuals.

63. The computer program product of claim 62, wherein the transaction history comprises purchase history and return history associated with products and services.

64. The computer program product of claim 51, wherein the set of codes that cause a computer to receive the second information further causes a computer to:

receive the second information from one or more message sharing networks.

65. The computer program product of claim 51, wherein the set of codes that cause a computer to receive the second information further causes a computer to:

allow the user to input the second information, wherein the second information comprises a name of an individual.

66. The computer program product of claim 65, wherein the set of codes further causes a computer to:

search, on a network, for information associated with the individual.

67. The computer program product of claim 51, wherein the set of codes that cause a computer to determine the relationship further cause a computer to:

compare the first information with the second information; and

determine a match between the first information and the second information.

68. The computer program product of claim 67, wherein the set of codes that cause a computer to determine a match further cause a computer to:

determine that an individual associated with the second information made a transaction associated with an object depicted in the first information.

69. The computer program product of claim 67, wherein the set of codes that cause a computer to determine a match further cause a computer to:

determine that an individual associated with the second information shared a message associated with an object depicted in the first information.

70. The computer program product of claim 67, wherein the set of codes that cause a computer to determine a match further cause a computer to:

determine that an individual associated with the second information was located at an object depicted in the first information.

71. The computer program product of claim 51, wherein the set of codes that cause a computer to present the relationship information further cause a computer to:

display a relationship indicator on a display of the mobile device.

72. The computer program product of claim 71, wherein the relationship indicator comprises a border that surrounds a depiction of an object.

73. The computer program product of claim 71, wherein the relationship indicator is selectable by the user.

74. The computer program product of claim 73, wherein the relationship indicator, upon being selected, provides information about an object associated with the image.

75. The computer program product of claim 73, wherein the relationship indicator, upon being selected, provides information about the individual.