Object information derived from object images

• US 9,182,828 B2
• Filed: 08/27/2014
• Issued: 11/10/2015
• Est. Priority Date: 11/06/2000
• Status: Expired due to Term

• Current Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Original Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Inventors: Boncyk, Wayne C., Cohen, Ronald H.
• Primary Examiner(s): Sherali, Ishrat I
• Application Number:
  US14/470,784
• Publication Number:
  US 20140368691A1
• Time in Patent Office: 440 Days
• Field of Search: 382/100, 382/305, 705 261- 272
• US Class Current: 1/1
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/24554 Unary operations; Data partitioning operations
• CPC Class Codes: G06F 16/50 of still image data
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 16/7335 Graphical querying, e.g. query-by-region, query-by-sketch, query-by-trajectory, GUIs for designating a person/face/object as a query predicate end-user interface involving hot spots associated with the video H04N21/4725; end-user interface for selecting a Region of Interest H04N21/4728
• CPC Class Codes: G06F 16/904 Browsing; Visualisation therefor for navigating the web G06F16/954; browsing optimisation for the web G06F16/957
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9537 Spatial or temporal dependent retrieval, e.g. spatiotemporal queries
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06F 18/25 Fusion techniques
• CPC Class Codes: G06F 2218/08 Feature extraction
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06Q 20/10 specially adapted for electronic funds transfer [EFT] systems; specially adapted for home banking systems
• CPC Class Codes: G06Q 30/0261 based on user location
• CPC Class Codes: G06Q 30/0267 Wireless devices
• CPC Class Codes: G06Q 30/0269 based on user profile or attribute
• CPC Class Codes: G06Q 30/0601 : Electronic shopping [e-shopping]
• CPC Class Codes: G06Q 30/0613 : Third-party assisted
• CPC Class Codes: G06Q 30/0623 : Item investigation
• CPC Class Codes: G06Q 30/0625 : Directed, with specific intent or strategy
• CPC Class Codes: G06Q 30/0635 : Processing of requisition or of purchase orders
• CPC Class Codes: G06T 7/00 : Image analysis
• CPC Class Codes: G06V 10/10 : Image acquisition document image scanning and transmission H04N1/00; control of digital cameras H04N23/60
• CPC Class Codes: G06V 10/255 : Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 10/768 : using context analysis, e.g. recognition aided by known co-occurring patterns
• CPC Class Codes: G06V 20/00 : Scenes; Scene-specific elements control of digital cameras H04N23/60
• CPC Class Codes: G06V 20/10 : Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/20 : in augmented reality scenes
• CPC Class Codes: G06V 20/46 : Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 20/80 : Recognising image objects characterised by unique random patterns
• CPC Class Codes: G06V 2201/07 : Target detection
• CPC Class Codes: G06V 40/16 : Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/18 : Eye characteristics, e.g. of the iris
• CPC Class Codes: H04N 23/80 : Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 5/44 : Receiver circuitry for the reception of television signals according to analogue transmission standards H04N5/14 takes precedence

Abstract

Search terms are derived automatically from images captured by a camera equipped cell phone, PDA, or other image capturing device, submitted to a search engine to obtain information of interest, and at least a portion of the resulting information is transmitted back locally to, or nearby, the device that captured the image.

Claims

1. A mobile device implemented translation method comprising:

capturing, by a mobile device, a video stream;

identifying, by the mobile device, within at least a portion of the video stream a recognizable key;

determining, by the mobile device, at least one of a position and orientation relative to an object in the video stream;

recognizing, by the mobile device, the recognizable key as a target object from the at least the portion of the video stream;

obtaining content information via an address pertinent to a translation associated with the target object; and

displaying, by the mobile device, the content information based on the at least one of the position and orientation.

2. The method of claim 1, wherein the recognizable key comprises at least one of the following:

a gesture, a symbol, a bar code, a matrix code, a literal, the object, a number, and an alphanumeric.

3. The method of claim 1, further comprising recognizing the recognizable key as the target object based on a portion of a single image related to the video stream.

4. The method of claim 1, further comprising obtaining the content information based on a search query derived from the recognizable key.

5. The method of claim 1, further comprising obtaining the content information based on a time when the video stream was captured.

6. The method of claim 1, further comprising obtaining the content information based on a location.

7. The method of claim 1, further comprising obtaining the content information based on a directory look-up.

8. The method of claim 1, wherein the content information comprises at least one of the following:

text, a graphic, video, a read sign, a read article, and an audio sample.

9. The method of claim 1, wherein the content information includes a menu of options.

10. The method of claim 1, wherein the step of recognizing the recognizable key as the target object comprises selecting a best match based on image characteristics.

11. The method of claim 1, wherein the mobile device comprises at least one of the following:

a vehicle, a cell phone, and a personal data assistant.

12. The method of claim 1, further comprising recognizing the recognizable key as a key in a dictionary.

13. The method of claim 1, wherein the step of displaying the content information includes presenting the content information on a display of the mobile device.

21. The mobile device implemented translation method of claim 1, wherein the translation is of the target object from an input language.

22. The mobile device implemented translation method of claim 1, wherein at least one of the target object and the recognizable key represents at least one of a symbol, a gesture, a text, a speech, and a sign language.

23. The mobile device implemented translation method of claim 1, wherein the content information comprises the translation.

14. A personal mobile device comprising:

a video capture sensor configured to capture a video stream;

a display;

a processor; and

a memory storing instruction that cause the processor to execute the steps of;

capturing, via the sensor, the video stream;

identifying within at least a portion of the video stream, a recognizable key;

determining at least one of a position and orientation relative to an object in the video stream;

recognizing the recognizable key as a target object based on the at least the portion of the video stream;

obtaining content information via an address pertinent to a translation, associated with the target object; and

displaying, via the display, the content information based on the at least one of the position and orientation.

15. The personal mobile device of claim 14, wherein the memory further comprises a database of known target objects.

16. The personal mobile device of claim 15, wherein the memory stores instructions that further configure the processor to execute the step of comparing image characteristics from the video stream to those of the known target objects.

17. The personal mobile device of claim 16, wherein the memory stores instructions that further configure the processor to execute the step of comparing image characteristics from the video stream to those of the known target objects in a parallel processing fashion.

18. The personal mobile device of claim 14, wherein the memory stores instructions that further configure the processor to execute the step of selecting the target object as a match for the recognizable key.

19. The personal mobile device of claim 18, wherein the memory stores instructions that further configure the processor to execute the step of selecting a best match for the target object.

24. The personal mobile device of claim 14, wherein the translation is of the target object from an input language.

25. The personal mobile device of claim 14, wherein at least one of the target object and the recognizable key represents at least one of a symbol, a gesture, a text, a speech, and a sign language.

26. The personal mobile device of claim 14, wherein the content information comprises the translation.

20. A non-transitory computer readable storage medium having stored instructions that, when executed by a processor, configure the processor to execute the steps of:

capturing, by a mobile device, a video stream;

identifying, by the mobile device, within at least a portion of the video stream a recognizable key;

determining, by the mobile device, at least one of a position and orientation relative to an object in the video stream;

recognizing, by the mobile device, the recognizable key as a target object based on that least the portion of the video stream;

obtaining content information via an address pertinent to a translation associated with the target object; and

displaying, by the mobile device, the content information based on the at least one of the position and orientation.

27. The non-transitory computer readable storage medium of claim 20, wherein the content information comprises the translation.

28. The non-transitory computer readable storage medium of claim 20, wherein at least one of the target object and the recognizable key represents at least one of a symbol, a gesture, a text, a speech, and a sign language.

29. The non-transitory computer readable storage medium of claim 20, wherein the content information comprises the translation.