Object information derived from object images

• US 7,899,252 B2
• Filed: 09/28/2009
• Issued: 03/01/2011
• Est. Priority Date: 11/06/2000
• Status: Expired due to Fees

• Current Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Original Assignee: Evryx Technologies Inc (Ipplex Holdings Corporation)
• Inventors: Boncyk, Wayne C., Cohen, Ronald H.
• Primary Examiner(s): SHERALI, ISHRAT I
• Application Number:
  US12/568,130
• Publication Number:
  US 20100034468A1
• Time in Patent Office: 519 Days
• Field of Search: 382181-182, 382/100, 382305-306, 382/224, 382/170, 382/209, 382/218, 382/233, 707 1- 7, 707/104.1, 709201-203, 709217-219, 709/250, 705 26- 27, 705/23, 705/64, 348/239, 348/460, 348/552, 3482112-2116, 348/207.1, 340/989, 340/988, 455/550.1, 455/414.2, 455/456.3, 455/556.1, 455/412.1, 455/419, 455/556.2, 235375-386, 235/462.09, 235/462.46
• US Class Current: 382/181
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/24554 Unary operations; Data partitioning operations
• CPC Class Codes: G06F 16/50 of still image data
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 16/7335 Graphical querying, e.g. query-by-region, query-by-sketch, query-by-trajectory, GUIs for designating a person/face/object as a query predicate end-user interface involving hot spots associated with the video H04N21/4725; end-user interface for selecting a Region of Interest H04N21/4728
• CPC Class Codes: G06F 16/904 Browsing; Visualisation therefor for navigating the web G06F16/954; browsing optimisation for the web G06F16/957
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9537 Spatial or temporal dependent retrieval, e.g. spatiotemporal queries
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06F 18/25 Fusion techniques
• CPC Class Codes: G06F 2218/08 Feature extraction
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06Q 20/10 specially adapted for electronic funds transfer [EFT] systems; specially adapted for home banking systems
• CPC Class Codes: G06Q 30/0261 based on user location
• CPC Class Codes: G06Q 30/0267 Wireless devices
• CPC Class Codes: G06Q 30/0269 based on user profile or attribute
• CPC Class Codes: G06Q 30/0601 : Electronic shopping [e-shopping]
• CPC Class Codes: G06Q 30/0613 : Third-party assisted
• CPC Class Codes: G06Q 30/0623 : Item investigation
• CPC Class Codes: G06Q 30/0625 : Directed, with specific intent or strategy
• CPC Class Codes: G06Q 30/0635 : Processing of requisition or of purchase orders
• CPC Class Codes: G06T 7/00 : Image analysis
• CPC Class Codes: G06V 10/10 : Image acquisition document image scanning and transmission H04N1/00; control of digital cameras H04N23/60
• CPC Class Codes: G06V 10/255 : Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 10/768 : using context analysis, e.g. recognition aided by known co-occurring patterns
• CPC Class Codes: G06V 20/00 : Scenes; Scene-specific elements control of digital cameras H04N23/60
• CPC Class Codes: G06V 20/10 : Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/20 : in augmented reality scenes
• CPC Class Codes: G06V 20/46 : Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 20/80 : Recognising image objects characterised by unique random patterns
• CPC Class Codes: G06V 2201/07 : Target detection
• CPC Class Codes: G06V 40/16 : Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/18 : Eye characteristics, e.g. of the iris
• CPC Class Codes: H04N 23/80 : Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 5/44 : Receiver circuitry for the reception of television signals according to analogue transmission standards H04N5/14 takes precedence

Abstract

Search terms are derived automatically from images captured by a camera equipped cell phone, PDA, or other image capturing device, submitted to a search engine to obtain information of interest, and at least a portion of the resulting information is transmitted back locally to, or nearby, the device that captured the image.

Claims

1. An image processing and information retrieval system comprising:

a mobile device having a camera, the mobile device configured to capture an image, and configured to transmit data relating to the image to an image processing platform;

the image processing platform configured to receive the data relating to the image and to conduct image processing including;

operating on the data relating to the image to determine if the image contains one or more recognizable symbols; and

decoding the recognizable symbols to extract symbol information by analyzing the recognizable symbols according to type;

a distal server configured to access a database to identify pertinent information associated with the recognizable symbols based on the symbol information; and

the mobile device further configured to receive the pertinent information over a network.

2. The system of claim 1, wherein the pertinent information comprises data that can be utilized to conduct a commercial transaction.

3. The system of claim 2, wherein the mobile device is further programmed to allow a user to automatically make a purchase relating to an object in the image.

4. The system of claim 1, wherein the pertinent information comprises contact information relating to an object in the image.

5. The system of claim 3, wherein the contact information comprises a phone number.

6. The system of claim 1, wherein the recognizable symbols include a bar code.

7. The system of claim 6, wherein the bar code comprises a 2D bar code.

8. The system of claim 1, wherein the pertinent information comprises one of the following types of data:

audio, image, video, and text.

9. The system of claim 1, wherein the pertinent information comprises a network address.

10. The system of claim 1, wherein the data relating to the image comprises the image.

11. The system of claim 10, wherein the data relating to the image includes a compressed version of the image.

12. The system of claim 11, wherein the compressed version of the image comprises compression artifacts.

13. The system of claim 1, wherein image processing further includes converting at least a portion of the data relating to the image into grayscale.

14. The system of claim 1, wherein image processing further includes discriminating between the recognizable symbols and other types of objects in the image.

15. The system of claim 1, wherein the distal server is further configured to use the database to identify the pertinent information based on image characteristics derived by the image processing platform based on the data relating to the image.

16. The system of claim 1, wherein the mobile device comprises at least part of the image processing platform.

17. The system of claim 16, wherein the image processing platform is distributed between the mobile device and the distal server.

18. A method for retrieving information from image processing, the method comprising:

providing a mobile device having a camera the mobile device configured to capture an image and configured to transmit data relating to the image an image processing platform;

configuring the image processing platform to receive the data relating to the image and to conduct image processing, including;

operating on the data relating to the image to determine if the image contains one or more recognizable symbols; and

decoding the recognizable symbols to extract symbol information by analyzing the recognizable symbols according to type;

providing access to a distal server configured to use a database to identify pertinent information associated with the recognizable symbols based on the symbol information; and

allowing the mobile device to receive the pertinent information over a network.

19. The method of claim 18, further comprising using the pertinent information to conduct a commercial transaction.

20. The method of claim 19, further comprising allowing the mobile device to automatically make a purchase relating to an object in the image.

21. The method of claim 18, wherein the pertinent information comprises contact information relating to an object in the image.

22. The method of claim 21, wherein the contact information comprises a phone number.

23. The method of claim 18, wherein image processing further includes identifying the recognizable symbols as comprising a bar code.

24. The method of claim 23, wherein the bar code comprises a 2D bar code.

25. The method of claim 18, wherein the pertinent information comprises a network address.

26. The method of claim 18, wherein the data relating to the image comprises the image.

27. The method of claim 26, wherein image processing further comprises compressing the image to form the data relating to the image.

28. The method of claim 27, wherein a compressed version of the image comprises compression artifacts.

29. The method of claim 18, wherein image processing further includes converting at least a portion of the data relating to the image into a grayscale image.

30. The method of claim 18, wherein the image processing further includes discriminating between the recognizable symbols and other types of objects in the image.

31. The method of claim 18, further comprising the distal server using database to identify the pertinent information based on image characteristics of the image derived by the image processing platform based on the data relating to the image.

32. The method of claim 18, further comprising providing the image processing platform by distributing image processing of the image between the mobile device and distal server.

33. The method of claim 18, wherein the distal service comprises the image processing platform.

34. The method of claim 18, further comprising returning to the pertinent information to the mobile device as of the following types of data:

audio, image, video, and text.