Object information derived from object images

• US 9,360,945 B2
• Filed: 12/14/2014
• Issued: 06/07/2016
• Est. Priority Date: 11/06/2000
• Status: Expired due to Term

• Current Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Original Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Inventors: Boncyk, Wayne C., Cohen, Ronald H.
• Primary Examiner(s): Sherali, Ishrat I
• Application Number:
  US14/569,766
• Publication Number:
  US 20150302087A1
• Time in Patent Office: 541 Days
• Field of Search: 382/100, 382/224, 382/181, 382/305, 705 261- 272, 707758-759, 707769-770
• US Class Current: 1/1
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/24554 Unary operations; Data partitioning operations
• CPC Class Codes: G06F 16/50 of still image data
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 16/7335 Graphical querying, e.g. query-by-region, query-by-sketch, query-by-trajectory, GUIs for designating a person/face/object as a query predicate end-user interface involving hot spots associated with the video H04N21/4725; end-user interface for selecting a Region of Interest H04N21/4728
• CPC Class Codes: G06F 16/904 Browsing; Visualisation therefor for navigating the web G06F16/954; browsing optimisation for the web G06F16/957
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9537 Spatial or temporal dependent retrieval, e.g. spatiotemporal queries
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06F 18/25 Fusion techniques
• CPC Class Codes: G06F 2218/08 Feature extraction
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06Q 20/10 specially adapted for electronic funds transfer [EFT] systems; specially adapted for home banking systems
• CPC Class Codes: G06Q 30/0261 based on user location
• CPC Class Codes: G06Q 30/0267 Wireless devices
• CPC Class Codes: G06Q 30/0269 based on user profile or attribute
• CPC Class Codes: G06Q 30/0601 : Electronic shopping [e-shopping]
• CPC Class Codes: G06Q 30/0613 : Third-party assisted
• CPC Class Codes: G06Q 30/0623 : Item investigation
• CPC Class Codes: G06Q 30/0625 : Directed, with specific intent or strategy
• CPC Class Codes: G06Q 30/0635 : Processing of requisition or of purchase orders
• CPC Class Codes: G06T 7/00 : Image analysis
• CPC Class Codes: G06V 10/10 : Image acquisition document image scanning and transmission H04N1/00; control of digital cameras H04N23/60
• CPC Class Codes: G06V 10/255 : Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 10/768 : using context analysis, e.g. recognition aided by known co-occurring patterns
• CPC Class Codes: G06V 20/00 : Scenes; Scene-specific elements control of digital cameras H04N23/60
• CPC Class Codes: G06V 20/10 : Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/20 : in augmented reality scenes
• CPC Class Codes: G06V 20/46 : Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 20/80 : Recognising image objects characterised by unique random patterns
• CPC Class Codes: G06V 2201/07 : Target detection
• CPC Class Codes: G06V 40/16 : Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/18 : Eye characteristics, e.g. of the iris
• CPC Class Codes: H04N 23/80 : Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 5/44 : Receiver circuitry for the reception of television signals according to analogue transmission standards H04N5/14 takes precedence

Abstract

Search terms are derived automatically from images captured by a camera equipped cell phone, PDA, or other image capturing device, submitted to a search engine to obtain information of interest, and at least a portion of the resulting information is transmitted back locally to, or nearby, the device that captured the image.

Claims

1. A method of enabling an interaction with on-screen video content, comprising:

obtaining, via a mobile device, image data associated with on-screen video content;

deriving a search term from the image data;

recognizing at least a portion of the on-screen video content as including a representation of a target object via a query of a data set indexing a collection of data associated with known target objects using the search term;

obtaining, via an address, content information corresponding to the target object; and

enabling, via a second mobile device, an interaction associated with the content information based on at least one of the address and the content information.

2. The method of claim 1, wherein the second mobile device comprises the first mobile device.

3. The method of claim 1, wherein the on-screen video content includes a representation of an object comprising a tag.

4. The method of claim 3, wherein the tag comprises at least one of a code, an image, a pattern, a logo and a sign.

5. The method of claim 4, wherein the tag is time varying.

6. The method of claim 1, wherein the on-screen video content lacks a representation of a tag.

7. The method of claim 1, wherein the target object comprises television programming.

8. The method of claim 1, wherein enabling the interaction associated with the content information comprises enabling a changing of a channel of a television.

9. The method of claim 1, wherein enabling the interaction associated with the content information comprises enabling a purchase associated with the target object.

10. The method of claim 1, wherein enabling the interaction associated with the content information comprises enabling a streaming of a sample of a video related to the on-screen video content.

11. The method of claim 1, wherein enabling the interaction associated with the content information comprises enabling the second mobile device to display showtimes related to the on-screen video content.

12. The method of claim 11, wherein enabling the interaction associated with the content information comprises enabling a purchase of a ticket related to the showtimes.

13. The method of claim 1, wherein enabling the interaction associated with the content information comprises enabling the second mobile device to make a telephone call related to the on-screen video content.

14. The method of claim 1, wherein enabling the interaction associated with the content information comprises enabling a real time interaction with the on-screen video content.

15. The method of claim 1, wherein enabling the interaction associated with the content information comprises enabling a purchase associated with the target object at a reduced price.

16. The method of claim 1, wherein enabling the interaction associated with the content information comprises enabling a vote registration related to the on-screen content.

17. The method of claim 1, wherein enabling the interaction associated with the content information comprises enabling an indication of an action.

18. The method of claim 1, wherein enabling the interaction associated with the content information comprises enabling obtaining a television listing corresponding to the target object.

19. The method of claim 1, wherein enabling the interaction associated with the content information comprises enabling obtaining a telephone number relating to the target object.

20. The method of claim 1, wherein enabling the interaction associated with the content information comprises enabling obtaining audio content relating to the target object.

21. The method of claim 1, wherein enabling the interaction associated with the content information comprises enabling obtaining text content relating to the target object.

22. The method of claim 1, wherein enabling the interaction associated with the content information comprises enabling obtaining video content relating to the target object.

23. The method of claim 1, wherein enabling the interaction associated with the content information comprises enabling obtaining graphics content relating to the target object.

24. The method of claim 1, wherein the on-screen video content comprises television programming.

25. The method of claim 1, wherein the on-screen video content comprises an advertisement.

26. The method of claim 1, wherein the on-screen video content represents a promotion.

27. The method of claim 1, wherein the on-screen video content is associated with a game.

28. The method of claim 1, wherein the on-screen video content comprises symbolic content.

29. The method of claim 1, further comprising obtaining at least one of a position, a location and an orientation related to the on-screen video content.

30. The method of claim 1, further comprising obtaining a time of capture of the on-screen video content.

31. The method of claim 1, further comprising obtaining non-visual information, and wherein deriving the search term further comprises deriving the search term from at least a portion of the non-visual information.

32. The method of claim 31, wherein the non-visual information comprises audio.

33. The method of claim 1, wherein the search term comprises a portion of the image data.

34. An on-screen content interaction system comprising:

a device interface configured to receive a digital representation of on-screen video content from a mobile device; and

a processing platform communicatively coupled with the device interface and configured to;

obtain the digital representation via the device interface;

derive a search term from the digital representation;

recognize at least a portion of the on-screen video content as the target object via a query of a data set indexing a collection of data associated with known target objects using the search term;

obtain, via an address, content information corresponding to the target object; and

enable, via a second device, an interaction associated with the content information based on at least one of the address and the content information.

35. The system of claim 34, wherein the on-screen video content comprises television programming.

36. The system of claim 34, wherein the target object comprises television programming.

37. The system of claim 34, wherein the interaction comprises a changing of a channel of a television.

38. The system of claim 34, wherein the interaction comprises a streaming of a sample of a video related to the target object.

39. The system of claim 34, wherein the interaction comprises a purchase related to the target object.

40. The system of claim 34, wherein the interaction comprises an indication of an action.

41. The system of claim 34, wherein the content information comprises a television listing corresponding to the target object.

42. The system of claim 34, wherein the content information comprises text content relating to the target object.

43. A portable imaging device, comprising:

a sensor configured to acquire a digital representation of on-screen video content; and

a computing platform coupled with the sensor and including an identification server configured to;

obtain the digital representation via the sensor;

derive a search term from the digital representation;

recognize at least a portion of the on-screen video content as including a target object via a query of a data set indexing a collection of data associated with known target objects using the search term;

obtain, via an address, content information corresponding to the target object; and

enable, via a second device, an interaction associated with the target object based on at least one of the address and the content information.

44. The device of claim 43, wherein the second device comprises the portable imaging device.

45. The device of claim 43, wherein the on-screen video content includes a representation of an object comprising a tag.

46. The device of claim 45, wherein the tag comprises at least one of a code, an image, a pattern, a logo and a sign.

47. The device of claim 46, wherein the tag is time varying.

48. The device of claim 43, wherein the on-screen video content lacks a representation of a tag.

49. The device of claim 43, wherein the target object comprises television programming.

50. The device of claim 43, wherein the interaction associated with the target object comprises a purchase associated with the target object.

51. The device of claim 43, wherein the interaction associated with the target object comprises a streaming of a sample of a video related to the on-screen video content.

52. The device of claim 43, wherein the interaction associated with the target object comprises displaying showtimes related to the on-screen video content.

53. The device of claim 43, wherein the interaction associated with the target object comprises a real time interaction with the on-screen video content.

54. The device of claim 43, wherein the interaction associated with the target object comprises an indication of an action.

55. The device of claim 43, wherein the on-screen video content comprises an advertisement.

56. The device of claim 43, wherein the on-screen video content is associated with a game.

57. The device of claim 43, wherein the on-screen video content represents a promotion.

58. The device of claim 43, wherein the identification server is further configured to obtain non-visual information, and derive the search term from at least a portion of the non-visual information.

59. The device of claim 58, wherein the non-visual information comprises audio.

60. The device of claim 43, wherein the identification server is further configured to obtain at least one of a position, a location and an orientation related to the on-screen video content, and derive the search term from the at least one of the position, the location and the orientation.

61. The device of claim 43, wherein the identification server is further configured to obtain a time of capture of the on-screen video content, and derive the search term from the time of capture.