Object information derived from object images

• US 8,503,787 B2
• Filed: 08/10/2011
• Issued: 08/06/2013
• Est. Priority Date: 11/06/2000
• Status: Expired due to Term

• Current Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Original Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Inventors: Boncyk, Wayne C., Cohen, Ronald H.
• Primary Examiner(s): Sherali, Ishrat I
• Application Number:
  US13/207,112
• Publication Number:
  US 20110295829A1
• Time in Patent Office: 727 Days
• Field of Search: 382/181, 382/162, 382/165, 382/100, 382/305, 382/224, 382115-118, 382/103, 348/239, 3482112-2116, 348/207.1, 348/460, 348/552, 705 261- 272, 705/23, 713/186, 713/168, 455/414.2, 455/412.1, 455/411, 455/456.5, 455/456.6, 709201-203, 709217-219, 709/250, 715/751
• US Class Current: 382/181
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/24554 Unary operations; Data partitioning operations
• CPC Class Codes: G06F 16/50 of still image data
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 16/7335 Graphical querying, e.g. query-by-region, query-by-sketch, query-by-trajectory, GUIs for designating a person/face/object as a query predicate end-user interface involving hot spots associated with the video H04N21/4725; end-user interface for selecting a Region of Interest H04N21/4728
• CPC Class Codes: G06F 16/904 Browsing; Visualisation therefor for navigating the web G06F16/954; browsing optimisation for the web G06F16/957
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9537 Spatial or temporal dependent retrieval, e.g. spatiotemporal queries
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06F 18/25 Fusion techniques
• CPC Class Codes: G06F 2218/08 Feature extraction
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06Q 20/10 specially adapted for electronic funds transfer [EFT] systems; specially adapted for home banking systems
• CPC Class Codes: G06Q 30/0261 based on user location
• CPC Class Codes: G06Q 30/0267 Wireless devices
• CPC Class Codes: G06Q 30/0269 based on user profile or attribute
• CPC Class Codes: G06Q 30/0601 : Electronic shopping [e-shopping]
• CPC Class Codes: G06Q 30/0613 : Third-party assisted
• CPC Class Codes: G06Q 30/0623 : Item investigation
• CPC Class Codes: G06Q 30/0625 : Directed, with specific intent or strategy
• CPC Class Codes: G06Q 30/0635 : Processing of requisition or of purchase orders
• CPC Class Codes: G06T 7/00 : Image analysis
• CPC Class Codes: G06V 10/10 : Image acquisition document image scanning and transmission H04N1/00; control of digital cameras H04N23/60
• CPC Class Codes: G06V 10/255 : Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 10/768 : using context analysis, e.g. recognition aided by known co-occurring patterns
• CPC Class Codes: G06V 20/00 : Scenes; Scene-specific elements control of digital cameras H04N23/60
• CPC Class Codes: G06V 20/10 : Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/20 : in augmented reality scenes
• CPC Class Codes: G06V 20/46 : Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 20/80 : Recognising image objects characterised by unique random patterns
• CPC Class Codes: G06V 2201/07 : Target detection
• CPC Class Codes: G06V 40/16 : Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/18 : Eye characteristics, e.g. of the iris
• CPC Class Codes: H04N 23/80 : Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 5/44 : Receiver circuitry for the reception of television signals according to analogue transmission standards H04N5/14 takes precedence

Abstract

Search terms are derived automatically from images captured by a camera equipped cell phone, PDA, or other image capturing device, submitted to a search engine to obtain information of interest, and at least a portion of the resulting information is transmitted back locally to, or nearby, the device that captured the image.

Claims

1. An object-based query system comprising:

a mobile device capable of acquiring a digital representation of at least a portion of an object and including location information;

an object identification platform coupled with the mobile device and capable of;

obtaining the digital representation;

deriving a query related to the object from the digital representation including the location information, where the query is derived according an indexing system of a search engine;

causing a result set considered pertinent to the object to be sent from the search engine to the mobile device based at least in part on the query; and

wherein the result set enables the mobile device to perform an interaction related to the object with a computing device.

2. The system of claim 1, wherein the query comprises attributes associated with a target object recognized by the object identification platform as the object.

3. The system of claim 1, wherein the digital representation comprises image data representative of a scene.

4. The system of claim 1, wherein the digital representation comprises non-visual information.

5. The system of claim 4, wherein the non-visual information comprises at least one of the following:

human speech and a natural sound.

6. The system of claim 1, wherein the location information comprises a fixed location.

7. The system of claim 1, wherein the location information comprises at least one of the following:

Global Positioning System information, radio-frequency ranging information, and cellular network ranging information.

8. The system of claim 1, wherein the location information comprises a location of the object.

9. The system of claim 8, wherein the object location comprises a position with respect to a user.

10. The system of claim 1, wherein the object identification platform is further capable of deriving the query tailored according to a time.

11. The system of claim 10, wherein the time comprises at least one of the following:

a limited time, and a specific time.

12. The system of claim 1, wherein the object comprises a purchasable product.

13. The system of claim 12, wherein the interaction comprises an interactive promotion associated with the purchasable product.

14. The system of claim 1, wherein the interaction comprises a transaction associated with the object and with an account over a network based on the result set.

15. The system of claim 14, wherein the account comprises at least one of the following types of accounts:

an on-line account, an account linked with the mobile device, an account linked to a user of the mobile device, a bank account, and a credit card account.

16. The system of claim 14, wherein the transaction comprises a purchase related to the object.

17. The system of claim 1, wherein the mobile device comprises one of the following devices configured with a sensor:

a cell phone, a personal data assistant, and a portable computer.

18. The system of claim 1, wherein the computing device comprises one of the following devices:

a server, a vending machine, and a television.

19. The system of claim 1, wherein the result set comprises at least one of the following:

text data, audio data, graphic data, and video data.

20. The system of claim 1, wherein the object comprises a specific target object identified by the object identification platform as a target object out of objects having similar appearance.

21. The system of claim 1, wherein the object comprises one of the following:

a sign, a billboard, a poster, a building, a store, a vehicle, a consumer product, a magazine, a television, a kiosk, a machine part, a newspaper, and a vending machine.

22. The system of claim 1, wherein the query is derived based on the digital representation obtained from the mobile device and a second digital representation obtained from a second mobile device.

23. An object-based query system comprising:

a mobile device having at least one sensor capable of acquiring a digital representation of at least a portion of an object and including time information;

an object identification platform coupled with the mobile device and capable of;

obtaining the digital representation;

deriving a query related to the object from the digital representation where the query is tailored according to the time information and is derived according to an indexing system of a search engine;

causing a result set considered pertinent to the object to be sent from the search engine to the mobile device based at least in part on the query; and

wherein the result set enables the mobile device to perform an interaction related to the object with a computing device.

24. The system of claim 23, wherein the query comprises attributes associated with a target object recognized by the identification platform as the object.

25. The system of claim 23, wherein the interaction is based on the time information relative to at least one of the following:

a limited time, and a specific time.

26. The system of claim 23, wherein the object identification platform is further capable of deriving the query based on location information within the digital representation.

27. The system of claim 23, wherein the interaction comprises a transaction associated with the object and with an account over a network based on the result set.

28. The system of claim 27, wherein the account comprises at least one of the following types of accounts:

an on-line account, an account linked with the mobile device, an account linked to a user of the mobile device, a bank account, and a credit card account.

29. The system of claim 27, wherein the transaction comprises a purchase related to the object.

30. The system of claim 23, wherein the mobile device comprises one of the following devices configured with a sensor:

a cell phone, a person data assistant, and a portable computer.

31. The system of claim 23, wherein the computing device comprises one of the following devices:

a server, a vending machine, and a television.

32. The system of claim 23, wherein the result set comprises at least one of the following:

text data, audio data, graphic data, and video data.

33. The system of claim 23, wherein the object comprises a purchasable product.

34. The system of claim 33, wherein the interaction comprises an interactive promotion associated with the purchasable product.

35. The system of claim 23, wherein the digital representation comprises image data related to the object.

36. The system of claim 23, wherein the digital representation comprises sound data related to the object.