Object information derived from object images

• US 9,785,651 B2
• Filed: 06/01/2016
• Issued: 10/10/2017
• Est. Priority Date: 11/06/2000
• Status: Expired due to Term

• Current Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Original Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Inventors: Boncyk, Wayne C., Cohen, Ronald H.
• Primary Examiner(s): Sherali, Ishrat I
• Application Number:
  US15/169,948
• Publication Number:
  US 20160275115A1
• Time in Patent Office: 496 Days
• Field of Search: 382100, 382305, 705 261- 271
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/24554 Unary operations; Data partitioning operations
• CPC Class Codes: G06F 16/50 of still image data
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 16/7335 Graphical querying, e.g. query-by-region, query-by-sketch, query-by-trajectory, GUIs for designating a person/face/object as a query predicate end-user interface involving hot spots associated with the video H04N21/4725; end-user interface for selecting a Region of Interest H04N21/4728
• CPC Class Codes: G06F 16/904 Browsing; Visualisation therefor for navigating the web G06F16/954; browsing optimisation for the web G06F16/957
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9537 Spatial or temporal dependent retrieval, e.g. spatiotemporal queries
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06F 18/25 Fusion techniques
• CPC Class Codes: G06F 2218/08 Feature extraction
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06Q 20/10 specially adapted for electronic funds transfer [EFT] systems; specially adapted for home banking systems
• CPC Class Codes: G06Q 30/0261 based on user location
• CPC Class Codes: G06Q 30/0267 Wireless devices
• CPC Class Codes: G06Q 30/0269 based on user profile or attribute
• CPC Class Codes: G06Q 30/0601 : Electronic shopping [e-shopping]
• CPC Class Codes: G06Q 30/0613 : Third-party assisted
• CPC Class Codes: G06Q 30/0623 : Item investigation
• CPC Class Codes: G06Q 30/0625 : Directed, with specific intent or strategy
• CPC Class Codes: G06Q 30/0635 : Processing of requisition or of purchase orders
• CPC Class Codes: G06T 7/00 : Image analysis
• CPC Class Codes: G06V 10/10 : Image acquisition document image scanning and transmission H04N1/00; control of digital cameras H04N23/60
• CPC Class Codes: G06V 10/255 : Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 10/768 : using context analysis, e.g. recognition aided by known co-occurring patterns
• CPC Class Codes: G06V 20/00 : Scenes; Scene-specific elements control of digital cameras H04N23/60
• CPC Class Codes: G06V 20/10 : Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/20 : in augmented reality scenes
• CPC Class Codes: G06V 20/46 : Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 20/80 : Recognising image objects characterised by unique random patterns
• CPC Class Codes: G06V 2201/07 : Target detection
• CPC Class Codes: G06V 40/16 : Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/18 : Eye characteristics, e.g. of the iris
• CPC Class Codes: H04N 23/80 : Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 5/44 : Receiver circuitry for the reception of television signals according to analogue transmission standards H04N5/14 takes precedence

Abstract

Search terms are derived automatically from images captured by a camera equipped cell phone, PDA, or other image capturing device submitted to a search engine to obtain information of interest, and at least a portion of the resulting information is transmitted back locally to, or nearby, the device that captured the image.

Claims

1. A computer implemented visual search engine comprising:

a searchable database indexing a collection of data associated with a plurality of known objects, the collection indexed according to a plurality of image-derived keys; and

a network-accessible search server coupled with the searchable database and, upon a processor in the server executing software instructions stored in a non-transitory computer readable memory, configured to;

receive, from a mobile device over a network, image-related data from an image of a query object;

derive a search query associated with the query object and including a first set of image-derived keys derived from the image-related data;

submit the search query to the searchable database;

receive a results set, including an information address associated with at least one known object from the plurality of known objects, from the searchable database; and

transmit at least a portion of the results set back to the mobile device over the network.

2. The engine of claim 1, wherein the image-related data includes at least one of the following:

a single image, multiple images, motion imagery, and video data.

3. The engine of claim 1, wherein the image-related data includes at least some image-derivable keys.

4. The engine of claim 1, wherein the image-related data includes salient parameters.

5. The engine of claim 1, wherein the image-related data includes search terms.

6. The engine of claim 1, wherein the search query includes attributes of the query object.

7. The engine of claim 6, wherein the attributes include at least one of the following:

a name, a type, a size, a shape, a dimension of relevance, a color, a position, a location, a make, a model, and a time.

8. The engine of claim 1, wherein the search query and the first set of image-derivable keys are both based on at least one of the following:

search terms, keywords, salient parameters, images, sounds, hashes, and know object attributes.

9. The engine of claim 1, wherein the results set include at least one of the following:

a hyperlink, an image, audio data, video data, a translation, spoken information, a web page, streaming data, product information, transaction information, account information, the information address, a code, instructions, and graphics.

10. The engine of claim 1, wherein the plurality of known objects comprise a propriety collection of records.

11. The engine of claim 1, wherein the plurality of known objects include at least one of the following:

a box, a DVD, a poster, a logo, text, a document, a magazine, an advertisement, a car, a building, a store, a theatre, a machine, a part, a symbol, a newspaper, a product, a vending machine, a television, a person, a face, biometric information, a fingerprint, an iris, a billboard, a street sign, on-screen content, a sign language, a beverage product, printed media, a photograph, a restaurant, a piece of art, a ticket, a barcodes, a matrix code, a computer, a three dimensional object, a product promotion, and a logo.

12. The engine of claim 1, wherein the plurality of known objects include three dimensional objects that change over time.

13. The engine of claim 12, wherein the three dimensional objects that change over time include at least one of the following:

a gesture, a pattern, an automobile, an animate object, a person'"'"'s face.

14. The engine of claim 1, wherein the at least the portion of the results set is transmitted back to the mobile device in real-time.

15. The engine of claim 14, wherein the at least the portion of the result set enables interactivity with a machine.

16. The engine of claim 1, wherein the network-accessible search server is further configured to generate the at least the portion of the result set by tailoring the result set to accommodate the mobile device.

17. The engine of claim 1, wherein the information address includes at least one of the following:

a URL, a URI, a hyperlink, a network address, an IP address, a telephone number, a radio channel, a television channel, and a physical location.

18. The engine of claim 1, wherein the at least the portion of the results set comprises sorted results.

19. The engine of claim 18, wherein the sorted results are determined based on a metric distance.

20. The engine of claim 1, wherein the searchable database comprises multiple databases partitioned across multiple processors.