Retrieving product information from embedded sensors via mobile device video analysis

• US 10,268,891 B2
• Filed: 02/19/2016
• Issued: 04/23/2019
• Est. Priority Date: 03/08/2011
• Status: Active Grant

• Current Assignee: Bank of America Corp.
• Original Assignee: Bank of America Corp.
• Inventors: Calman, Matthew A., Ross, Erik Stephen, Hamilton, Alfred
• Primary Examiner(s): Desai, Resha
• Application Number:
  US15/048,011
• Publication Number:
  US 20160171306A1
• Time in Patent Office: 1,159 Days
• Field of Search: 705 261- 272, 705305
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06Q 10/00 Administration; Management
• CPC Class Codes: G06Q 10/06315 Needs-based resource requirements planning or analysis
• CPC Class Codes: G06Q 20/12 specially adapted for electronic shopping systems
• CPC Class Codes: G06Q 20/322 Aspects of commerce using mobile devices [M-devices]
• CPC Class Codes: G06Q 20/3278 RFID or NFC payments by means of M-devices
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06Q 30/0623 Item investigation
• CPC Class Codes: G06Q 30/0635 Processing of requisition or of purchase orders
• CPC Class Codes: G06Q 30/0641 Shopping interfaces
• CPC Class Codes: G06Q 30/0643 Graphical representation of items or shoppers
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 2201/10 Recognition assisted with metadata
• CPC Class Codes: H04W 4/70 Services for machine-to-machine communication [M2M] or machine type communication [MTC]

Abstract

A system for accessing product information from embedded sensors via mobile device video analysis is disclosed. The system includes a processing device configured for analyzing a video stream of the surrounding environment captured on a mobile device. At least one product that has one or more embedded sensors associated therewith is identified within the surrounding environment. The system retrieves information associated with the one or more products from the one or more embedded sensors. Information retrieved, or information derived from the information retrieved, from the embedded sensors is presented to the user via a display of the video stream on the mobile device.

Claims

1. A computer program product for accessing product information from embedded sensors, the computer program product comprising a non-transitory computer-readable medium comprising code causing a computer to:

analyze a real-time video stream representing the surrounding environment captured on a mobile device;

identify one or more products within the video stream that has one or more embedded sensors associated therewith;

retrieve information associated with the one or more products from the one or more embedded sensors;

determine from the retrieved information critical information associated with the one or more products, wherein the critical information comprises information regarding a product default or an impending product default, wherein the product default is a mechanical default requiring maintenance;

present, via a display of the video stream on the mobile device, the information retrieved, or information derived from the information retrieved, from the one or more embedded sensors, wherein presenting the retrieved or derived information comprises superimposing the retrieved or derived information over the video stream of the one or more products; and

alert, via the mobile device, the user to the critical information associated with the one or more products.

2. The computer program product of claim 1, wherein the mobile device is a mobile phone.

3. The computer program product of claim 1, wherein the mobile device comprises a camera capable of image and/or video capture.

4. The computer program product of claim 1, wherein the one or more embedded sensors emit a radio frequency signal.

5. The computer program product of claim 1, wherein the information presented is displayed as a virtual object overlaying the video stream of the surrounding environment.

6. The computer program product of claim 5, wherein the virtual object is interactive.

7. The computer program product of claim 1, wherein the mobile device is operated in the “

always on”

mode to continuously analyze the surrounding environment.

8. The computer program product of claim 1, wherein the product default is the depletion or expiration of a product requiring replacement.

9. The computer program product of claim 1, wherein alerting the user comprises producing an audible sound, vibration, generating a Short Message Service or text message, generating an email, or combinations thereof.

10. The computer program product of claim 1, wherein the non-transitory computer-readable medium comprises code causing a computer to present the user with an option to purchase the one or more products via the mobile device.

11. The computer program product of claim 10, wherein the non-transitory computer-readable medium comprises code causing a computer to transmit payment information from the mobile device to a seller of the one or more products in order to purchase the one or more products.

12. The computer program product of claim 10, wherein the non-transitory computer-readable medium comprises code causing a computer to transmit payment information via a wireless connection between the mobile device and a contactless payment terminal to purchase a product or service.

13. The computer program product of claim 1, wherein the non-transitory computer-readable medium comprises code causing a computer to transmit financial data and/or other user-specific data to a financial institution.