Object information derived from object images

• US 8,798,322 B2
• Filed: 08/20/2013
• Issued: 08/05/2014
• Est. Priority Date: 11/06/2000
• Status: Expired due to Fees

• Current Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Original Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Inventors: Boncyk, Wayne C., Cohen, Ronald H.
• Primary Examiner(s): Sherali, Ishrat I
• Application Number:
  US13/971,758
• Publication Number:
  US 20130336533A1
• Time in Patent Office: 350 Days
• Field of Search: 382/305, 382/181, 382/103, 704 1- 10
• US Class Current: 382/103
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/24554 Unary operations; Data partitioning operations
• CPC Class Codes: G06F 16/50 of still image data
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 16/7335 Graphical querying, e.g. query-by-region, query-by-sketch, query-by-trajectory, GUIs for designating a person/face/object as a query predicate end-user interface involving hot spots associated with the video H04N21/4725; end-user interface for selecting a Region of Interest H04N21/4728
• CPC Class Codes: G06F 16/904 Browsing; Visualisation therefor for navigating the web G06F16/954; browsing optimisation for the web G06F16/957
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9537 Spatial or temporal dependent retrieval, e.g. spatiotemporal queries
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06F 18/25 Fusion techniques
• CPC Class Codes: G06F 2218/08 Feature extraction
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06Q 20/10 specially adapted for electronic funds transfer [EFT] systems; specially adapted for home banking systems
• CPC Class Codes: G06Q 30/0261 based on user location
• CPC Class Codes: G06Q 30/0267 Wireless devices
• CPC Class Codes: G06Q 30/0269 based on user profile or attribute
• CPC Class Codes: G06Q 30/0601 : Electronic shopping [e-shopping]
• CPC Class Codes: G06Q 30/0613 : Third-party assisted
• CPC Class Codes: G06Q 30/0623 : Item investigation
• CPC Class Codes: G06Q 30/0625 : Directed, with specific intent or strategy
• CPC Class Codes: G06Q 30/0635 : Processing of requisition or of purchase orders
• CPC Class Codes: G06T 7/00 : Image analysis
• CPC Class Codes: G06V 10/10 : Image acquisition document image scanning and transmission H04N1/00; control of digital cameras H04N23/60
• CPC Class Codes: G06V 10/255 : Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 10/768 : using context analysis, e.g. recognition aided by known co-occurring patterns
• CPC Class Codes: G06V 20/00 : Scenes; Scene-specific elements control of digital cameras H04N23/60
• CPC Class Codes: G06V 20/10 : Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/20 : in augmented reality scenes
• CPC Class Codes: G06V 20/46 : Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 20/80 : Recognising image objects characterised by unique random patterns
• CPC Class Codes: G06V 2201/07 : Target detection
• CPC Class Codes: G06V 40/16 : Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/18 : Eye characteristics, e.g. of the iris
• CPC Class Codes: H04N 23/80 : Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 5/44 : Receiver circuitry for the reception of television signals according to analogue transmission standards H04N5/14 takes precedence

Abstract

Search terms are derived automatically from images captured by a camera equipped cell phone, PDA, or other image capturing device, submitted to a search engine to obtain information of interest, and at least a portion of the resulting information is transmitted back locally to, or nearby, the device that captured the image.

Claims

1. A method for enabling a mobile device to use non-visual information, the method comprising:

capturing non-visual information related to a media object via the mobile device;

recognizing, via the mobile device, the media object as a target object based at least in part on the non-visual information;

deriving a search term associated with the target object by analyzing the non-visual information;

submitting the search term to a search engine that uses the search term and produces a result set;

receiving at least a portion of the result set comprising an information address via the mobile device; and

enabling the mobile device to use the information address to interact with a content server.

2. The method of claim 1, wherein the content server is associated with television programming, and wherein the step of enabling the mobile device to use the information address to interact with the content server includes enabling the mobile device to use the information address to interact with the content server in real time.

3. The method of claim 1, wherein the mobile device comprises a cell phone.

4. The method of claim 1, wherein the non-visual information comprises a sound.

5. The method of claim 1, wherein the information address comprises an address of a webpage.

6. The method of claim 1, wherein the search term is derived from at least one attribute of the media object.

7. The method of claim 1, wherein the step of submitting the search term to a search engine that uses the search term to produce a result set comprises comparing the search term to representations of a plurality of known objects stored in a database.

8. The method of claim 1, wherein the media object comprises presented media.

9. The method of claim 8, wherein the media object comprises screen content.

10. The method of claim 9, wherein the screen content comprises on-screen content.

11. The method of claim 9, wherein the screen content comprises television screen content.

12. The method of claim 9, wherein the screen content comprises a tag associated with the media object.

13. The method of claim 1, wherein the media object lacks a tag.

18. The system of claim 12, wherein the portion of the result set comprises a webpage.

14. A system for using non-visual information for interactions, the system comprising:

an identification server communicatively coupled with a mobile device, a search engine and a database storing representations of a plurality of known objects;

wherein the mobile device is configured to capture non-visual information related to a media object;

wherein at least one of the identification server and the mobile device is configured to;

recognize the media object as a target object based at least in part on the non-visual information; and

derive a search term associated with the target object by analyzing the non-visual information;

wherein the search term is submitted to the search engine that compares the search term to representations of the plurality of known objects stored in the database to produce a result set; and

wherein the mobile device is configured to receive at least a portion of the result set and interact with a content server via at least one information address associated with the result set.

15. The system of claim 14, wherein the content server is associated with television programming, and wherein the mobile device is configured to use the portion of the result set to interact with the content server in real time.

16. The system of claim 14, wherein the mobile device comprises a cell phone.

17. The system of claim 14, wherein the non-visual information comprises a sound.

19. The system of claim 14, wherein the search term is derived from at least one attribute of the object.

20. The system of claim 14, wherein the media object comprises presented media.

21. The system of claim 20, wherein the media object comprises screen content.

22. The system of claim 21, wherein the screen content comprises on-screen content.

23. The system of claim 21, wherein the screen content comprises television screen content.

24. The system of claim 21, wherein the screen content comprises a tag associated with the media object.

25. The system of claim 14, wherein the media object lacks a tag.

26. A system for using non-visual information for interactions, the system comprising:

an identification server communicatively coupled with a mobile device, a search engine and a database storing representations of a plurality of known objects;

wherein the mobile device is configured to capture non-visual information related to a media object;

wherein at least one of the identification server and the mobile device is configured to;

recognize the media object as a target object based at least in part on the non-visual information; and

derive a search term associated with the target object by analyzing the non-visual information and at least one of position information or location information;

wherein the search term derived from the non-visual information and at least one of position information or location information is submitted to the search engine that compares the search term to representations of the plurality of known objects stored in the database to produce a result set; and

wherein the mobile device is configured to receive at least a portion of the result set and interact with a content server via at least one information address associated with the result set.

27. A system for using non-visual information for interactions, the system comprising:

an identification server communicatively coupled with a mobile device, a search engine and a database storing representations of a plurality of known objects;

wherein the mobile device is configured to capture non-visual information related to a media object;

wherein at least one of the identification server and the mobile device is configured to;

recognize the media object as a target object based at least in part on the non-visual information; and

derive a search term associated with the target object by analyzing the non-visual information and position information;

wherein the search term derived from the non-visual information and position information is submitted to the search engine that compares the search term to representations of the plurality of known objects stored in the database to produce a result set; and

wherein the mobile device is configured to receive at least a portion of the result set and interact with a content server via at least one information address associated with the result set, wherein the result set is related to a time.

28. A system for using non-visual information for interactions, the system comprising:

an identification server communicatively coupled with a mobile device, a search engine and a database storing representations of a plurality of known objects;

wherein the mobile device is configured to capture non-visual information related to a media object;

wherein at least one of the identification server and the mobile device is configured to;

recognize the media object as a target object based at least in part on the non-visual information; and

derive a search term associated with the target object by analyzing the non-visual information;

wherein the search term is submitted to the search engine that compares the search term to representations of the plurality of known objects stored in the database to produce a result set; and

wherein the mobile device is configured to receive at least a portion of the result set and interact with a content server via an information address associated with the result set, wherein the portion of the result set is related to a time.

29. A non-transitory computer readable medium storing software instructions executable by a mobile device processor of a mobile device to cause the mobile device processor to execute the steps of:

capturing non-visual information related to a media object;

recognizing the media object as a target object;

obtaining a search term associated with the target object derived by at least one of an identification server and the mobile device based on non-visual information related to a media object captured by the mobile device;

comparing the search term to representations of a plurality of known objects stored in a database and producing a result set; and

transmitting at least a portion of the result set to the mobile device and configuring the mobile device to interact with a content server via an information address associated with the result set.

30. The non-transitory computer readable medium of claim 29, wherein the content server is associated with television programming, and wherein the step of configuring the mobile device to interact with the content server comprises configuring the mobile device to use the portion of the result set to interact with the content server in real time.

31. The non-transitory computer readable medium of claim 29, wherein the mobile device comprises a cell phone.

32. The non-transitory computer readable medium of claim 29, wherein the non-visual information comprises a sound.

33. The non-transitory computer readable medium of claim 29, wherein the portion of the result set comprises a webpage.

34. The non-transitory computer readable medium of claim 29, wherein the search term is derived from at least one attribute of the media object.

35. The non-transitory computer readable medium of claim 29, wherein the media object comprises presented media.

36. The non-transitory computer readable medium of claim 35, wherein the media object comprises screen content.

37. The non-transitory computer readable medium of claim 36, wherein the screen content comprises on-screen content.

38. The non-transitory computer readable medium of claim 36, wherein the screen content comprises television screen content.

39. The non-transitory computer readable medium of claim 35, wherein the screen content comprises a tag associated with the media object.

40. The system of claim 29, wherein the media object lacks a tag.