Object information derived from object images

• US 10,509,820 B2
• Filed: 09/26/2018
• Issued: 12/17/2019
• Est. Priority Date: 11/06/2000
• Status: Expired due to Term

• Current Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Original Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Inventors: Boncyk, Wayne C., Cohen, Ronald H.
• Primary Examiner(s): Sherali, Ishrat I
• Application Number:
  US16/143,257
• Publication Number:
  US 20190026314A1
• Time in Patent Office: 447 Days
• Field of Search: 382100, 382305, 705 261- 271
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/24554 Unary operations; Data partitioning operations
• CPC Class Codes: G06F 16/50 of still image data
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 16/7335 Graphical querying, e.g. query-by-region, query-by-sketch, query-by-trajectory, GUIs for designating a person/face/object as a query predicate end-user interface involving hot spots associated with the video H04N21/4725; end-user interface for selecting a Region of Interest H04N21/4728
• CPC Class Codes: G06F 16/904 Browsing; Visualisation therefor for navigating the web G06F16/954; browsing optimisation for the web G06F16/957
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9537 Spatial or temporal dependent retrieval, e.g. spatiotemporal queries
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06F 18/25 Fusion techniques
• CPC Class Codes: G06F 2218/08 Feature extraction
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06Q 20/10 specially adapted for electronic funds transfer [EFT] systems; specially adapted for home banking systems
• CPC Class Codes: G06Q 30/0261 based on user location
• CPC Class Codes: G06Q 30/0267 Wireless devices
• CPC Class Codes: G06Q 30/0269 based on user profile or attribute
• CPC Class Codes: G06Q 30/0601 : Electronic shopping [e-shopping]
• CPC Class Codes: G06Q 30/0613 : Third-party assisted
• CPC Class Codes: G06Q 30/0623 : Item investigation
• CPC Class Codes: G06Q 30/0625 : Directed, with specific intent or strategy
• CPC Class Codes: G06Q 30/0635 : Processing of requisition or of purchase orders
• CPC Class Codes: G06T 7/00 : Image analysis
• CPC Class Codes: G06V 10/10 : Image acquisition document image scanning and transmission H04N1/00; control of digital cameras H04N23/60
• CPC Class Codes: G06V 10/255 : Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 10/768 : using context analysis, e.g. recognition aided by known co-occurring patterns
• CPC Class Codes: G06V 20/00 : Scenes; Scene-specific elements control of digital cameras H04N23/60
• CPC Class Codes: G06V 20/10 : Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/20 : in augmented reality scenes
• CPC Class Codes: G06V 20/46 : Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 20/80 : Recognising image objects characterised by unique random patterns
• CPC Class Codes: G06V 2201/07 : Target detection
• CPC Class Codes: G06V 40/16 : Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/18 : Eye characteristics, e.g. of the iris
• CPC Class Codes: H04N 23/80 : Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 5/44 : Receiver circuitry for the reception of television signals according to analogue transmission standards H04N5/14 takes precedence

Abstract

A real-world object is recognized from image data based on derived image characteristics, which are used to derive search information to conduct a search. The search returns object information which is used to execute a software process by a mobile device.

Claims

1. A method for visual search, comprising:

receiving, by an identification server and from an image sensor of a mobile device, image data containing a digital representation of a real-world object;

deriving, by the identification server, image characteristics corresponding to the real-world object from the digital representation of the real-world object in the image data;

deriving, by the identification server, search information associated with the real-world object based on the image characteristics;

initiating, by at least one of the identification server or the mobile device, a search based on the search information;

obtaining, by the at least one of the identification server or the mobile device, object information associated with the real-world object as a result of the search; and

executing, by the mobile device, a software process to carry out a purchase transaction as a function of the object information.

2. The method of claim 1, wherein executing the software process further comprises displaying, by the mobile device, information about the real-world object.

3. The method of claim 2, wherein the information about the real-world object comprises textual information about the real-world object.

4. The method of claim 3, wherein the information about the real-world object further comprises additional image data associated with the real-world object.

5. The method of claim 1, wherein the real-world object comprises an automobile, the method further comprising:

receiving, by the identification server and from the mobile device, location information of the mobile device;

wherein the search is also based on the location information; and

wherein executing the software process further comprises;

displaying, by the mobile device, information about the automobile based on the object information, wherein the object information comprises information about the automobile; and

providing, by the mobile device, a location of a dealer selling the automobile based on the object information and the location of the mobile device.

6. The method of claim 1, further comprising:

wherein the real-world object comprises an item of printed media advertising a movie;

wherein the search information comprises search terms associated with at least one of a movie title of the movie, an actor in the movie, or a characteristic of the movie; and

wherein executing the software process further comprises presenting, by the mobile device, at least one of;

a sample of media from the movie, show times associated with the movie, or an option to purchase tickets to the movie.

7. The method of claim 1, further comprising:

wherein the real-world object comprises at least one of a visual advertisement for a restaurant or a restaurant sign for the restaurant;

wherein the object information comprises information about the restaurant; and

wherein executing the software process further comprises presenting, by the mobile device, at least one of a location of the restaurant, a menu of the restaurant, or a telephone number for the restaurant.

8. The method of claim 1, further comprising:

wherein the search information comprises at least one keyword associated with the real-world object;

wherein the object information comprises information corresponding to at least one video associated with the at least one keyword; and

wherein executing the software process further comprises streaming, by the mobile device, the at least one video associated with the at least one keyword.

9. The method of claim 1, further comprising:

wherein the real-world object includes visual content associated with an audio recording;

wherein the object information includes information associated with the audio recording; and

wherein executing the software process further comprises playing, by the mobile device, audio associated with the audio recording based on the object information.

10. The method of claim 1, further comprising wherein the real-world object comprises a product and wherein the object information comprises a URL where the product is available for purchase.

11. The method of claim 10, wherein executing the software process further comprises providing, by the mobile device, a link to access a website via the URL.

12. The method of claim 10, wherein executing the software process further comprises providing, by the mobile device, an interface to carry out the purchase transaction for the product via the URL.

13. The method of claim 1, further comprising:

wherein the real-world object comprises at least one a product offered by a company, an item of visual media associated with a company, and a logo associated with a company;

wherein the object information includes at least one of a telephone number for the company or a URL for a website associated with the company; and

wherein executing the software process further comprises presenting, by the mobile device, the at least one of the telephone number or the URL.

27. The method of claim 1, wherein the real-world object comprises a vending machine and executing the software process to carry out the purchase transaction further comprises purchasing a product from the vending machine.

14. At least one non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to:

receive, from an image sensor of a mobile device, image data containing a digital representation of a real-world object;

derive image characteristics corresponding to the real-world object from the digital representation of the real-world object in the image data;

derive search information associated with the real-world object based on the image characteristics;

initiate a search based on the search information;

obtain object information associated with the real-world object as a result of the search; and

cause the mobile device to execute a software process to carry out a purchase transaction as a function of the object information.

15. The at least one non-transitory computer-readable medium of claim 14, wherein causing the mobile device to execute the software process further comprises causing the mobile device to display information about the real-world object.

16. The at least one non-transitory computer-readable medium of claim 15, wherein the information about the real-world object comprises textual information about the real-world object.

17. The at least one non-transitory computer-readable medium of claim 16, wherein the information about the real-world object further comprises additional image data associated with the real-world object.

18. The at least one non-transitory computer-readable medium of claim 14, wherein the real-world object comprises an automobile and further storing instructions that, when executed by the at least one processor, cause the at least one processor to:

receive location information of the mobile device;

wherein the search is also based on the location information; and

wherein causing the mobile device to execute the software process further comprises;

causing the mobile device to display information about the automobile based on the object information, wherein the object information comprises information about the automobile; and

causing the mobile device to provide a location of a dealer selling the automobile based on the object information and the location of the mobile device.

19. The at least one non-transitory computer-readable medium of claim 14, further comprising:

wherein the real-world object comprises an item of printed media advertising a movie;

wherein the search information comprises search terms associated with at least one of a movie title of the movie, an actor in the movie, or a characteristic of the movie; and

wherein causing the mobile device to execute the software process further comprises causing the mobile device to present at least one of;

a sample of media from the movie, show times associated with the movie, or an option to purchase tickets to the movie.

20. The at least one non-transitory computer-readable medium of claim 14, further comprising:

wherein the real-world object comprises at least one of a visual advertisement for a restaurant or a restaurant sign for the restaurant;

wherein the object information comprises information about the restaurant; and

wherein causing the mobile device to execute the software process further comprises causing the mobile device to present at least one of a location of the restaurant, a menu of the restaurant, or a telephone number for the restaurant.

21. The at least one non-transitory computer-readable medium of claim 14, further comprising:

wherein the search information comprises at least one keyword associated with the real-world object;

wherein the object information comprises information corresponding to at least one video associated with the at least one keyword; and

wherein causing the mobile device to execute the software process further comprises causing the mobile device to stream the at least one video associated with the at least one keyword.

22. The at least one non-transitory computer-readable medium of claim 14, further comprising:

wherein the real-world object includes visual content associated with an audio recording;

wherein the object information includes information associated with the audio recording; and

wherein causing the mobile device to execute the software process further comprises causing the mobile device to play audio associated with the audio recording based on the object information.

23. The at least one non-transitory computer-readable medium of claim 14, further comprising wherein the real-world object comprises a product and wherein the object information comprises a URL where the product is available for purchase.

24. The at least one non-transitory computer-readable medium of claim 14, wherein causing the mobile device to execute the software process further comprises causing the mobile device to provide a link to access a website via the URL.

25. The at least one non-transitory computer-readable medium of claim 14, wherein causing the mobile device to execute the software process further comprises causing the mobile device to provide an interface to carry out the purchase transaction for the product via the URL.

26. The at least one non-transitory computer-readable medium of claim 14, further comprising:

wherein the real-world object comprises at least one a product offered by a company, an item of visual media associated with a company, and a logo associated with a company;

wherein the object information includes at least one of a telephone number for the company or a URL for a website associated with the company; and

wherein causing the mobile device to execute the software process further comprises causing the mobile device to present the at least one of the telephone number or the URL.

28. The at least one non-transitory computer-readable medium of claim 14, wherein the real-world object comprises a vending machine and causing the mobile device to execute the software process to carry out the purchase transaction further comprises enabling a purchase of a product from the vending machine via the mobile device.