Portable object identification and translation system

• US 20030164819A1
• Filed: 03/04/2002
• Published: 09/04/2003
• Est. Priority Date: 03/04/2002
• Status: Abandoned Application

• Current Assignee: Alex Waibel
• Original Assignee: Alex Waibel
• Inventors: Waibel, Alex
• Application Number:
  US10/090,559
• Publication Number:
  US 20030164819A1
• Time in Patent Office: Days
• US Class Current: 345/173
• CPC Class Codes: G06F 1/1626 with a single-body enclosure integrating a flat display, e.g. Personal Digital Assistants [PDAs]
• CPC Class Codes: G06F 1/1632 External expansion units, e.g. docking stations
• CPC Class Codes: G06F 1/1684 Constructional details or arrangements related to integrated I/O peripherals not covered by groups G06F1/1635 - G06F1/1675
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 1/1698 the I/O peripheral being a sending/receiving arrangement to establish a cordless communication link, e.g. radio or infrared link, integrated cellular phone details of antennas disposed inside a computer H01Q1/2266
• CPC Class Codes: G06F 16/3334 Selection or weighting of terms from queries, including natural language queries
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/5846 using extracted text
• CPC Class Codes: G06F 16/907 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 2200/1632 Pen holder integrated in the computer
• CPC Class Codes: G06V 30/142 using hand-held instruments; Constructional details of the instruments

Abstract

A portable information system is comprised of an input device for capturing an image having a user-selected object or text, and a background. A hand-held computer is responsive to the input device and is programmed to: distinguish the user-selected object/text from the background; compare the user-selected object to a database of objects/characters; and output a translation of, information about, or interpretation of, the user-selected object or text in response to the step of comparing. The invention is particularly useful as a portable aid for translating or remembering text messages foreign to the user that are found in visual scenes. A second important use is to provide mobile information and guidance to the mobile user in connection with surrounding objects (such as, identifying landmarks, people, and/or acting as a navigational aid). Methods of operating the present invention are also disclosed.

Claims

1. A portable information system, comprising:

an input device for capturing an image having a user-selected object and a background; and

a hand-held computer responsive to said input device and programmed to;

distinguish said user-selected object from said background;

compare said user-selected object to a database of objects; and

output information about said user-selected object in response to said step of comparing.

2. The portable information system of claim 1 wherein said input device includes one of a camera and a scanner.

3. The portable information system of claim 1 wherein said hand-held computer includes a personal digital assistant.

4. The portable information system of claim 1 wherein said hand-held computer comprises an output device for displaying said captured image and wherein said hand-held computer is programmed to operate in a continuous mode based on said user-selected object being positioned within an active area of said output device.

5. The portable information system of claim 1 wherein said hand-held computer comprises a touch sensitive output device for displaying said captured image, and wherein said hand-held computer is programmed to operate based on the user-selected object being one of touched or outlined.

6. A portable translation system, comprising:

an input device for capturing an image including text and a background; and

a hand-held computer responsive to said input device and programmed to;

distinguish text in said sign from said background;

recognize characters forming the text;

translate said text; and

output a translation of said text.

7. The portable translation system of claim 6 wherein said output includes one of acoustic and visual output.

8. The portable system of claim 7 wherein said acoustic output includes speech synthesis.

9. The portable system of claim 8 additionally comprising outputting said translation visually and outputting said recognized characters acoustically.

10. The portable translation system of claim 6 wherein said input device includes one of a camera and a scanner.

11. The portable translation system of claim 6 wherein said handheld computer includes a personal digital assistant.

12. The portable translation system of claim 6 wherein said hand-held computer comprises an output device for displaying said captured image and wherein said hand-held computer is programmed to continuously translate characters positioned within an active area of said output device.

13. The portable translation system of claim 6 wherein said hand-held computer comprises a touch sensitive output device for displaying said captured image, and wherein said hand-held computer is programmed to operate based on characters being one of touched or outlined.

14. A portable system, comprising:

an input device for capturing an image including text and a background; and

a hand-held computer responsive to said input device and programmed to;

distinguish text in said sign from said background;

recognize characters forming the text;

convert said characters into a different set of characters; and

output said different set of characters.

15. The portable system of claim 14 wherein said output includes one of acoustic and visual output.

16. The portable system of claim 15 wherein said acoustic output includes speech synthesis.

17. The portable system of claim 14 additionally comprising outputting said different set of characters visually and outputting said recognized characters acoustically.

18. The portable system of claim 14 wherein said input device includes one of a camera and a scanner.

19. The portable system of claim 14 wherein said handheld computer includes a personal digital assistant.

20. The portable system of claim 14 wherein said hand-held computer comprises an output device for displaying said captured image and wherein said hand-held computer is programmed to continuously convert characters positioned within an active area of said output device.

21. The portable system of claim 14 wherein said hand-held computer comprises a touch sensitive output device for displaying said captured image, and wherein said hand-held computer is programmed to operate based on the characters of the sign being one of touched or outlined.

22. A video camera for producing an image having at least one object and a background, the improvement comprising:

a computer having a processor and memory, said computer programmed to;

extract said at least one object from said background;

compare said at least one object to a database of objects; and

output information about said at least one object in response to said step of comparing.

23. The camera of claim 22 additionally comprising a screen for displaying said produced image, and wherein said computer is programmed to operate based on the object being positioned within some portion of said screen.

24. The camera of claim 22 wherein said information output about said at least one object is selected from the set comprising a translation, a conversion, historical information, biographical information, and geographical information.

25. A cell phone having a camera for producing an image having at least one object and a background, the improvement comprising:

a computer having a processor and memory, said computer programmed to;

extract said at least one object from said background;

compare said at least one object to a database of objects; and

output information about said at least one object in response to said step of comparing.

26. The cell phone of claim 25 additionally comprising an output screen for displaying said produced image and wherein said computer is programmed to operate in a continuous mode based on said at least one object being positioned within an active area of said output screen.

27. The cell phone of claim 25 additionally comprising a touch sensitive output screen for displaying said produced image, and wherein said computer is programmed to operate based on the object being one of touched or outlined.

28. The cell phone of claim 25 wherein said information output about said at least one object is selected from the set comprising a translation, a conversion, historical information, biographical information, and geographical information.

29. The cell phone of claim 25 wherein said computer is provided by a server, and wherein said cell phone is in communication with said server.

30. A combination, comprising:

eyewear;

an input device carried by said eyewear for capturing an image having an object and a background; and

a hand-held computer responsive to said input device and programmed to;

extract said at least one object from said background;

compare said at least one object to a database of objects; and

output information about said at least one object in response to said step of comparing.

31. The combination of claim 30 additionally comprising an output device for displaying said captured image and wherein said computer is programmed to operate in a continuous mode based on said at least one object being positioned within an active area of said output device.

32. The combination of claim 30 additionally comprising a touch sensitive output screen for displaying said produced image, and wherein said computer is programmed to operate based on the object being one of touched or outlined.

33. The combination of claim 30 wherein said information output about said at least one object is selected from the set comprising a translation, a conversion, historical information, biographical information, and geographical information.

34. A method for using a hand-held computer to provide information related to a user-selected object, comprising:

populating a database within a hand-held computer with a plurality of objects and information related thereto;

capturing an image having a user-selected object and a background;

distinguishing said user-selected object from said background;

comparing said user-selected object to said plurality of objects;

selecting an object matching said user-selected object from said plurality of objects; and

retrieving and outputting information in response to said selecting step.

35. The method of claim 34 additionally comprising determining said hand-held computer'"'"'s relative location and populating the data based on the computer'"'"'s relative location.

36. The method of claim 34 wherein said capturing an image includes storing said image in a memory device.

37. The method of claim 34 wherein said capturing an image includes storing a stream of images.

38. The method of claim 34 wherein said distinguishing said user-selected object from said background further comprises:

employing at least one of edge filtering, neural networks and bootstrapping, texture segmentation, and color quantization.

39. The method of claim 34 wherein said distinguishing said user-selected object from said background further comprises manually designating said user-selected object within said image.

40. A method for translating a sign having a plurality of characters in a first language to a second language, comprising:

capturing an image containing a background and a sign;

extracting a plurality of characters from said sign;

recognizing said plurality of characters; and

translating said plurality of characters from a first language to a second language.

41. The method of claim 40 wherein said capturing an image includes storing said image in a memory device.

42. The method of claim 40 wherein said capturing an image containing a background and a sign further comprises storing a stream of images in a memory device.

43. The method of claim 40 wherein said extracting said plurality of characters from said background further comprises manually designating said characters within said image.

44. The method of claim 40 wherein said extracting said plurality of characters further comprises:

employing at least one of edge filtering, neural networks and bootstrapping, texture segmentation, and color quantization.

45. The method of claim 40 wherein said translating said plurality of characters from said first language to said second language further comprises employing one of an example based system, rule-based system, statistical machine translation system, a phrase-book, and a lookup dictionary.