Retrieving product information from embedded sensors via mobile device video analysis

• US 9,224,166 B2
• Filed: 01/01/2012
• Issued: 12/29/2015
• Est. Priority Date: 03/08/2011
• Status: Active Grant

• Current Assignee: Bank of America Corp.
• Original Assignee: Bank of America Corp.
• Inventors: Calman, Matthew A., Ross, Erik Stephen, Hamilton, Alfred
• Primary Examiner(s): Desai, Resha
• Application Number:
  US13/342,053
• Publication Number:
  US 20120233025A1
• Time in Patent Office: 1,458 Days
• Field of Search: 705 261- 272
• US Class Current: 1/1
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06Q 10/00 Administration; Management
• CPC Class Codes: G06Q 10/06315 Needs-based resource requirements planning or analysis
• CPC Class Codes: G06Q 20/12 specially adapted for electronic shopping systems
• CPC Class Codes: G06Q 20/322 Aspects of commerce using mobile devices [M-devices]
• CPC Class Codes: G06Q 20/3278 RFID or NFC payments by means of M-devices
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06Q 30/0623 Item investigation
• CPC Class Codes: G06Q 30/0635 Processing of requisition or of purchase orders
• CPC Class Codes: G06Q 30/0641 Shopping interfaces
• CPC Class Codes: G06Q 30/0643 Graphical representation of items or shoppers
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 2201/10 Recognition assisted with metadata
• CPC Class Codes: H04W 4/70 Services for machine-to-machine communication [M2M] or machine type communication [MTC]

Abstract

A system for accessing product information from embedded sensors via mobile device video analysis is disclosed. The system includes a processing device configured for analyzing a video stream of the surrounding environment captured on a mobile device. At least one product that has one or more embedded sensors associated therewith is identified within the surrounding environment. The system retrieves information associated with the one or more products from the one or more embedded sensors. Information retrieved, or information derived from the information retrieved, from the embedded sensors is presented to the user via a display of the video stream on the mobile device.

Claims

1. A method for accessing product information from embedded sensors comprising:

analyzing, via a processing device, a real-time video stream representing the surrounding environment captured on a mobile device;

identifying one or more products within the video stream that has one or more embedded sensors associated therewith;

retrieving information associated with the one or more products from the one or more embedded sensors;

presenting, via a display of the video stream on the mobile device, the information retrieved, or information derived from the information retrieved, from the one or more embedded sensors, wherein presenting the retrieved or derived information comprises superimposing the retrieved or derived information over the video stream of the one or more products; and

alerting, via the display, the user to critical information associated with the one or more products, wherein the critical information comprises information regarding a product default or an impending product default, wherein the product default is a mechanical default requiring maintenance.

2. The method of claim 1, wherein the mobile device is a mobile phone.

3. The method of claim 1, wherein the mobile device comprises a camera capable of image and/or video capture.

4. The method of claim 1, wherein the one or more embedded sensors emit a radio frequency signal.

5. The method of claim 1, wherein the information presented is displayed as a virtual object overlaying the video stream of the surrounding environment.

6. The method of claim 5, wherein the virtual object is interactive.

7. The method of claim 1, wherein the mobile device is operated in the “

always on”

mode to continuously analyze the surrounding environment.

8. The method of claim 1, wherein the product default is the depletion or expiration of a product requiring replacement.

9. The method of claim 1, wherein the step of alerting the user comprises producing an audible sound, vibration, generating a Short Message Service or text message, generating an email, or combinations thereof.

10. The method of claim 1, wherein the one or more products is a food product.

11. The method of claim 1, wherein the one or more products is a household item.

12. The method of claim 11, wherein the household item is a household appliance.

13. The method of claim 1, further comprising presenting the user with an option to purchase the one or more products via the mobile device.

14. The method of claim 1, further comprising transmitting payment information from the mobile device to a seller of the one or more products in order to purchase the one or more products.

15. The method of claim 1, further comprising transmitting payment information via a wireless connection between the mobile device and a contactless payment terminal to purchase a product or service.

16. The method of claim 1, further comprising transmitting financial data and/or other user-specific data to a financial institution.

17. A system for accessing product information from embedded sensors comprising:

a processing device configured for;

analyzing a real-time video stream representing the surrounding environment captured on a mobile device;

identifying one or more products within the video stream that has one or more embedded sensors associated therewith;

retrieving information associated with the one or more products from the one or more embedded sensors;

presenting, via a display of the video stream on the mobile device, the information retrieved, or information derived from the information retrieved, from the one or more embedded sensors, wherein presenting the retrieved or derived information comprises superimposing the retrieved or derived information over the video stream of the one or more products; and

alerting, via the display, the user to critical information associated with the one or more products, wherein the critical information comprises information regarding a product default or an impending product default, wherein the product default is a mechanical default requiring maintenance.

18. The system of claim 17, wherein the mobile device is a mobile phone.

19. The system of claim 17, wherein the mobile device comprises a camera capable of image and/or video capture.

20. The system of claim 17, wherein the one or more embedded sensors emit a radio frequency signal.

21. The system of claim 17, wherein the information presented is displayed as a virtual object overlaying the video stream of the surrounding environment.

22. The system of claim 21, wherein the virtual object is interactive.

23. The system of claim 17, wherein the mobile device is operated in the “

always on”

mode to continuously analyze the surrounding environment.

24. The system of claim 17, wherein the product default is the depletion or expiration of a product requiring replacement.

25. The system of claim 17, wherein alerting the user comprises producing an audible sound, vibration, generating a Short Message Service or text message, generating an email, or combinations thereof.

26. The system of claim 17, wherein the one or more products is a food product.

27. The system of claim 17, wherein the one or more products is a household item.

28. The system of claim 27, wherein the household item is a household appliance.

29. The system of claim 17, wherein the processing device is further configured to present the user with an option to purchase the one or more products via the mobile device.

30. The system of claim 17, wherein the processing device is further configured to transmit payment information from the mobile device to a seller of the one or more products in order to purchase the one or more products.

31. The system of claim 17, wherein the processing device is further configured to transmit payment information via a wireless connection between the mobile device and a contactless payment terminal to purchase a product or service.

32. The system of claim 17, wherein the processing device is further configured to transmit financial data and/or other user-specific data to a financial institution.

33. A computer program product for accessing product information from embedded sensors comprising a non-transitory computer readable medium comprising one or more computer-readable code portions configured for:

analyzing a real-time video stream representing the surrounding environment captured on a mobile device;

identifying one or more products within the video stream that has one or more embedded sensors associated therewith;

retrieving information associated with the one or more products from the one or more embedded sensors;

presenting, via a display of the video stream on the mobile device, the information retrieved, or information derived from the information retrieved, from the one or more embedded sensors, wherein presenting the retrieved or derived information comprises superimposing the retrieved or derived information over the video stream of the one or more products; and

alerting, via the display, the user to critical information associated with the one or more products, wherein the critical information comprises information regarding a product default or an impending product default, wherein the product default is a mechanical default requiring maintenance.

34. The computer program product of claim 33, wherein the mobile device is a mobile phone.

35. The computer program product of claim 33, wherein the mobile device comprises a camera capable of image and/or video capture.

36. The computer program product of claim 33, wherein the one or more embedded sensors emit a radio frequency signal.

37. The computer program product of claim 33, wherein the information presented is displayed as a virtual object overlaying the video stream of the surrounding environment.

38. The computer program product of claim 37, wherein the virtual object is interactive.

39. The computer program product of claim 33, wherein the mobile device is operated in the “

always on”

mode to continuously analyze the surrounding environment.

40. The computer program product of claim 33, wherein the product default is the depletion or expiration of a product requiring replacement.

41. The computer program product of claim 33, wherein alerting the user comprises producing an audible sound, vibration, generating a Short Message Service or text message, generating an email, or combinations thereof.

42. The computer program product of claim 33, wherein the one or more products is a food product.

43. The computer program product of claim 33, wherein the one or more products is a household item.

44. The computer program product of claim 43, wherein the household item is a household appliance.

45. The computer program product of claim 33, wherein the one or more computer-readable code portions is further configured to present the user with an option to purchase the one or more products via the mobile device.

46. The computer program product of claim 33, wherein the one or more computer-readable code portions is further configured to transmit payment information from the mobile device to a seller of the one or more products in order to purchase the one or more products.

47. The computer program product of claim 33, wherein the one or more computer-readable code portions is further configured to transmit payment information via a wireless connection between the mobile device and a contactless payment terminal to purchase a product or service.

48. The computer program product of claim 33, wherein the one or more computer-readable code portions is further configured to transmit financial data and/or other user-specific data to a financial institution.