Object information derived from object images

• US 9,087,240 B2
• Filed: 07/18/2014
• Issued: 07/21/2015
• Est. Priority Date: 11/06/2000
• Status: Expired due to Fees

• Current Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Original Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Inventors: Boncyk, Wayne C., Cohen, Ronald H.
• Primary Examiner(s): Sherali, Ishrat I
• Application Number:
  US14/335,559
• Publication Number:
  US 20140330677A1
• Time in Patent Office: 368 Days
• Field of Search: 382/100, 382/103, 382/165, 382/181, 382/305, 705 261- 272
• US Class Current: 1/1
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/24554 Unary operations; Data partitioning operations
• CPC Class Codes: G06F 16/50 of still image data
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 16/7335 Graphical querying, e.g. query-by-region, query-by-sketch, query-by-trajectory, GUIs for designating a person/face/object as a query predicate end-user interface involving hot spots associated with the video H04N21/4725; end-user interface for selecting a Region of Interest H04N21/4728
• CPC Class Codes: G06F 16/904 Browsing; Visualisation therefor for navigating the web G06F16/954; browsing optimisation for the web G06F16/957
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9537 Spatial or temporal dependent retrieval, e.g. spatiotemporal queries
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06F 18/25 Fusion techniques
• CPC Class Codes: G06F 2218/08 Feature extraction
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06Q 20/10 specially adapted for electronic funds transfer [EFT] systems; specially adapted for home banking systems
• CPC Class Codes: G06Q 30/0261 based on user location
• CPC Class Codes: G06Q 30/0267 Wireless devices
• CPC Class Codes: G06Q 30/0269 based on user profile or attribute
• CPC Class Codes: G06Q 30/0601 : Electronic shopping [e-shopping]
• CPC Class Codes: G06Q 30/0613 : Third-party assisted
• CPC Class Codes: G06Q 30/0623 : Item investigation
• CPC Class Codes: G06Q 30/0625 : Directed, with specific intent or strategy
• CPC Class Codes: G06Q 30/0635 : Processing of requisition or of purchase orders
• CPC Class Codes: G06T 7/00 : Image analysis
• CPC Class Codes: G06V 10/10 : Image acquisition document image scanning and transmission H04N1/00; control of digital cameras H04N23/60
• CPC Class Codes: G06V 10/255 : Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 10/768 : using context analysis, e.g. recognition aided by known co-occurring patterns
• CPC Class Codes: G06V 20/00 : Scenes; Scene-specific elements control of digital cameras H04N23/60
• CPC Class Codes: G06V 20/10 : Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/20 : in augmented reality scenes
• CPC Class Codes: G06V 20/46 : Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 20/80 : Recognising image objects characterised by unique random patterns
• CPC Class Codes: G06V 2201/07 : Target detection
• CPC Class Codes: G06V 40/16 : Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/18 : Eye characteristics, e.g. of the iris
• CPC Class Codes: H04N 23/80 : Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 5/44 : Receiver circuitry for the reception of television signals according to analogue transmission standards H04N5/14 takes precedence

Abstract

Search terms are derived automatically from images captured by a camera equipped cell phone, PDA, or other image capturing device, submitted to a search engine to obtain information of interest, and at least a portion of the resulting information is transmitted back locally to, or nearby, the device that captured the image.

Claims

1. A method implemented by a computing device, wherein the computing device determines whether a product has associated product information, the method comprising:

receiving information associated with an image captured by a mobile device, wherein the image comprises a digital representation of a product;

recognizing, based at least partially on the information associated with the image, the product as a target object that can be purchased via an account, wherein the product is associated with an information address; and

presenting, via the mobile device, based on the information address, graphical content information representing options associated with the product.

2. The method of claim 1, wherein receiving information associated with the image comprises receiving information associated with an image that is part of a real-time video stream.

3. The method of claim 1, wherein the receiving information associated with the image comprises receiving information associated with a still image.

4. The method of claim 1, wherein the account comprises at least one of an account linked to a terminal, an account linked to an individual user, an account linked to a bank account, and an account linked to a credit card.

5. The method of claim 1, wherein receiving information associated with the image comprises receiving an image of the product.

6. The method of claim 1, wherein recognizing, based at least partially on the information associated with the image the product as the target object that can be purchased via the account comprises:

identifying the product based at least partially on the information associated with the image; and

recognizing the product as a target object that can be purchased via the account by comparing information associated with the product to information associated with a known set of products.

7. The method of claim 1, wherein recognizing, based at least partially on the information associated with the image, the product as the target object that can be purchased via an account comprises:

comparing the information associated with the image with information about products that have associated product information.

8. The method of claim 1, wherein presenting, via the mobile device, based on the information address, graphical content information representing options associated with the product comprises displaying, on a display of the mobile device, graphical content information representing an option to purchase the product.

9. The method of claim 8, wherein displaying, on the display of the mobile device, graphical content information representing the option to purchase the product comprises displaying at least one of an identity, a position, an orientation, and a status of the product.

10. The method of claim 8, further comprising:

displaying, on the display of the mobile device, information about the product based upon the information address.

11. The method of claim 8, further comprising:

displaying, on the display of the mobile device, information pertinent to a meaning of at least a portion of the product.

12. The method of claim 8, further comprising:

displaying, on the display of the mobile device, search results triggered by a keyword derived from at least a portion of the product.

13. The method of claim 8, further comprising:

displaying, on the display of the mobile device, content information in the form of at least one of text, audio, video, and graphics that is pertinent to the product.

14. The method of claim 8, further comprising:

displaying, on the display of the mobile device, at least one price of the product.

15. The method of claim 8, further comprising:

displaying, on the display of the mobile device, a web page that is pertinent to the product.

16. The method of claim 8, further comprising:

displaying, on the display of the mobile device, a web page of a company selling the product.

17. The method of claim 1, further comprising:

determining that the mobile device is capturing a real-time video stream that shows the product during a specified time before presenting, via the mobile device, the content information representing options associated with the product.

18. The method of claim 1, wherein presenting, via the mobile device, based on the information address, graphical content information representing options associated with the product further comprises presenting the graphical content information based on at least one of a position and an orientation of the product relative to a second object represented in the image.

19. The method of claim 1, wherein presenting, via the mobile device, based on the information address, graphical content information representing options associated with the product comprises presenting a transaction user interface associated with the product.

20. The method of claim 1, wherein the presenting, via the mobile device, based on the information address, graphical content information representing options associated with the product comprises presenting, on the mobile device, an option to purchase the product via the account.

21. An apparatus for determining whether a product has associated target information, comprising:

a communication device;

a processing device communicably coupled to communication device, wherein the processing device is configured to;

receive information associated with an image of the product, wherein the image is captured by a mobile communication device;

recognize, based at least partially on the information associated with the image, the product as a target object that can be purchased via an account, wherein the product is associated with an information address; and

present, via the mobile device, based on the information address, graphical content information representing options associated with the product.

22. The apparatus of claim 21, wherein the image of the product is part of a real-time video stream.

23. The apparatus of claim 21, wherein the image of the product is a still image.

24. The apparatus of claim 21, wherein the account comprises at least one of an account linked to a terminal, to an individual user, a bank account, and a credit card.

25. The apparatus of claim 21, wherein the information associated with the image comprises an image of the product.

26. The apparatus of claim 21, wherein the processor is further configured to:

identify the product based at least partially on the information associated with the image; and

recognize the product as the target object that can be purchased via the account by comparing information about the product to information about a known set of products.

27. The apparatus of claim 21, wherein the processor is further configured to:

compare the information associated with the image with information about products that have associated product information.

28. The apparatus of claim 21, wherein the graphical content information representing options associated with the product comprises graphical content information representing an option to purchase the product.

29. The apparatus of claim 21, wherein presenting the graphical content information comprises displaying at least one of an identity, a position, an orientation, and a status of the product.

30. The apparatus of claim 28, wherein the processor is further configured to display, on the display of the mobile device, information about the product based upon the information address.

31. The apparatus of claim 28, wherein the processor is further configured to display, on the display of the mobile device, information pertinent to a meaning of at least a portion of the product.

32. The apparatus of claim 28, wherein the processor is further configured to display, on the display of the mobile device, search results triggered by a keyword derived from at least a portion of the image of the product.

33. The apparatus of claim 28, wherein the processor is further configured to display, on the display of the mobile device, content information in the form of at least one of text, audio, video, and graphics that is pertinent to the product.

34. The apparatus of claim 28, wherein the processor is further configured to display, on the display of the mobile device, at least one price of the product.

35. The apparatus of claim 28, wherein the processor is further configured to display, on the display of the mobile device, a web page that is pertinent to the product.

36. The apparatus of claim 28, wherein the processor is further configured to display, on the display of the mobile device, a web page of a company selling the product.

37. The apparatus of claim 21, wherein the processor is further configured to:

determine that the mobile device is capturing a real-time video stream that shows the product during a specified time before presenting, via the mobile device, the graphical content information representing options associated with the product.

38. The apparatus of claim 21, wherein the graphical content information representing options associated with the product is presented based at least in part on at least one of a position and an orientation of the product relative to a second object in the image.

39. The apparatus of claim 21, wherein the graphical content information representing options associated with the product comprises a transaction user interface associated with the product.

40. The apparatus of claim 21, wherein the graphical content information representing options associated with the product comprises an option to purchase the product via the account.

41. A computer program product for determining whether a product is associated with product information, the computer program product comprising a memory, wherein the memory comprises computer executable software stored therein, the computer-executable software comprising:

a first executable portion configured for receiving information associated with an image of the product, wherein the image was captured by a mobile device;

a second executable portion configured for recognizing, based at least partially on the information associated with the image, the product as a target object that can be purchased via an account, wherein the product is associated with an information address; and

a third executable portion configured for presenting, via the mobile device, based on the information address, graphical content information representing options associated with the product.

42. The computer program product of claim 41, wherein the image of the product is part of a real-time video stream.

43. The computer program product of claim 41, wherein the image of the product is a still image.

44. The computer program product of claim 41, wherein the account comprises at least one of the account linked to a terminal, to an individual user, a bank account, and a credit card.

45. The computer program product of claim 41, wherein the information associated with the image comprises an image of the product.

46. The computer program product of claim 41, wherein the second executable portion is further configured for:

identifying the product based at least partially on the information associated with the image; and

recognizing the product as the target object that can be purchased via the account by comparing information about the product to information about a known set of products.

47. The computer program product of claim 41, wherein the second executable portion is further configured for:

comparing the information associated with the image with information about products that have associated product information.

48. The computer program product of claim 41, wherein the third executable portion is further configured for displaying, on a display of the mobile device, graphical content information representing an option to purchase the product.

49. The computer program product of claim 48, wherein the graphical content information comprises at least one of an identity, a position, an orientation, and a status of the product.

50. The computer program product of claim 48, wherein the computer-executable program software further comprises a fourth executable portion configured for displaying, on the display of the mobile device, information about the product based upon the information address.

51. The computer program product of claim 48, wherein the computer-executable program software further comprises a fourth executable portion configured for displaying, on the display of the mobile device, information pertinent to a meaning of at least a portion of the product.

52. The computer program product of claim 48, wherein the computer-executable program software further comprises a fourth executable portion configured for displaying, on the display of the mobile device, search results triggered by a keyword derived from at least a portion of the product.

53. The computer program product of claim 48, wherein the computer-executable program software further comprises a fourth executable portion configured for displaying, on the display of the mobile device, content information in the form of at least one of text, audio, video, and graphics that is pertinent to the product.

54. The computer program product of claim 48, wherein the computer-executable program software further comprises a fourth executable portion configured for displaying, on the display of the mobile device, at least one price of the product.

55. The computer program product of claim 48, wherein the computer-executable program software further comprises a fourth executable portion configured for displaying, on the display of the mobile device, a web page that is pertinent to the product.

56. The computer program product of claim 48, wherein the computer-executable program software further comprises a fourth executable portion configured for displaying, on the display of the mobile device, a web page of a company selling the product.

57. The computer program product of claim 41, wherein the computer-executable program software further comprises a fourth executable portion configured for determining that the mobile device is capturing a real-time video stream that shows the product during a specified time before presenting, via the mobile device, the graphical content information representing options associated with the product.

58. The computer program product of claim 41, wherein the third executable portion is further configured for presenting the graphical content information representing options associated with the product based at least in part on at least one of a position and an orientation of the product relative to a second object in the image.

59. The computer program product of claim 41, wherein the graphical content information representing options associated with the product comprises a transaction user interface associated with the product.

60. The computer program product of claim 41, wherein the graphical content information representing options associated with the product comprises an option to purchase the product via the account.