Apparatus and method for image processing of hand-written characters using coded structured light and time series frame capture

• US 6,970,600 B2
• Filed: 06/28/2001
• Issued: 11/29/2005
• Est. Priority Date: 06/29/2000
• Status: Expired due to Fees

• Current Assignee: Fuji Xerox Company Limited (Xerox Holdings Corp.)
• Original Assignee: Fuji Xerox Company Limited (Xerox Holdings Corp.)
• Inventors: Abe, Tsutomu
• Primary Examiner(s): Au, Amelia M.
• Assistant Examiner(s): TUCKER, WESLEY J
• Application Number:
  US09/892,884
• Publication Number:
  US 20020001029A1
• Time in Patent Office: 1,615 Days
• Field of Search: 382/154, 382183-189, 345/773, 345/178, 345848-852, 345/863, 345/865, 725/19, 348/14.16
• US Class Current: 382/187
• CPC Class Codes: G01B 11/2509 Color coding
• CPC Class Codes: G01B 11/2545 with one projection direction and several detection directions, e.g. stereo
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06V 20/62 Text, e.g. of license plates, overlay texts or captions on TV images
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/1434 Special illumination such as grating, reflections or deflections, e.g. for characters with relief
• CPC Class Codes: G06V 30/228 of three-dimensional handwriting, e.g. writing in the air

Abstract

The present invention provides an image processing apparatus that efficiently identifies input characters and the like by an intensity image analysis based on range information. The image processing apparatus performs distance measurement by performing three-dimensional measurement by pattern projection to characters written to a manuscript, whiteboard, and the like, gets an intensity image used as a so-called actual image and an image for distance measurement in parallel, and identifies input characters by an intensity image analysis based on range information. The image processing apparatus picks up characters, patterns, and the like written onto paper or the like with a pen, performs geometric transformation for picked-up images, and performs natural input character analysis and reading processing. Furthermore, by comparing images picked up in a time-series, noise elimination and manuscript position modifications become possible.

Claims

1. An image processing apparatus, comprising:

a three-dimensional image pickup part that includes a projecting part that projects a pattern along a direction of an optical axis, a first image pickup part that picks up an intensity image and a projection pattern image deflected from the direction of the optical axis of the projecting part by a half mirror, and a second image pickup part that picks up the projection pattern image from a direction different from the optical axis of the projecting part, the three-dimensional image pickup part creating first range information based on a pattern picked up by the second image pickup part;

a geometric transformation part that performs geometric transformation for the intensity image picked up by the first image pickup part, based on the first range information;

a storage part that stores, as initial frame data, an initial image of frame data in a time-series transformed by the geometric transformation part;

a frame data comparison part that makes comparison between successive frame data images in the time-series transformed by the geometric transformation part; and

an image processing part that retrieves only differential data between successive frame data in the time-series as storage data based on a result of the comparison of the frame data comparison part of the initial frame data and frame data subsequently transformed in the time-series.

2. The image processing apparatus according to claim 1, wherein, for an area where the amount of change of the pattern picked up by the first image pickup part with respect to the projection pattern is equal to or greater than a predetermined value, new code corresponding to the pattern picked up by the first image pickup part is assigned, and the first range information is created from the pattern picked up by the second image pickup part based on the new code.

3. The image processing apparatus according to claim 1, wherein

the image processing part eliminates noise data from the frame data image based on a result of the comparison between the frame data images in the frame data comparison part.

4. The image processing apparatus according to claim 1, wherein

the image processing part modifies a position of the frame data image based on a result of the comparison between the frame data images in the frame data comparison part.

5. The image processing apparatus according to claim 1, wherein:

the projecting part has a light source to emit light of an invisible region of a wavelength band; and

the first and second image pickup parts have a filter for transmitting light of the invisible region of the wavelength band and a filter for cutting off light of the invisible region of the wavelength band, and pick up the projection pattern image and intensity image in parallel.

6. The image processing apparatus according to claim 1, wherein, for an area where the amount of change of the pattern picked up by the first image pickup part with respect to the projection pattern by the projecting part is less than a predetermined value, second range information is created by deriving a correspondence between intensity information obtained by the first and second image pickup parts.

7. The image processing apparatus according to claim 1, wherein the second image pickup part includes plural image pickup parts that pick up the measurement target at different angles, and range information is created based on projection patterns respectively picked up by the plural image pickup parts of the second image pickup part.

8. An image processing method, comprising:

projecting a pattern by a projecting part along an optical axis direction;

picking up an intensity image and a projection pattern image by a first image pickup part deflected from the optical axis direction of the projecting part by a half mirror, and picking up the projection pattern image by a second image pickup part from a direction different from the optical axis direction of the projecting part;

creating first range information based on the pattern picked up by the second image pickup part;

performing geometric transformation for the intensity image produced by the first image pickup part based on the range information;

storing an initial geometric-transformed intensity image in a time-series transformed in the geometric transformation step;

making comparison between successive geometric-transformed intensity images in the time-series transformed in geometric transformation step; and

retrieving only differential data between successive geometric-transformed intensity images in the time series as storage data based on a result of the comparison of the comparison step of the initial geometric-transformed intensity image and geometric-transformed intensity images subsequently transformed in the time-series.

9. The image processing method according to claim 8, wherein the range information creation step includes the step of:

for an area where the amount of change of the pattern picked up by the first image pickup part with respect to the projection pattern is equal to or greater than a predetermined value, assigning new code corresponding to the pattern picked up by the first image pickup part, and creating the first range information from the pattern picked up by the second image pickup part based on the new code.

10. The image processing method according to claim 8, further comprising:

eliminating noise data from the frame data image based on a result of the comparison between the frame data images in the frame data comparison step.

11. The image processing method according to claim 8, further comprising:

modifying a position of the frame data image based on a result of the comparison between the frame data images in the frame data comparison step.

12. The image processing method according to claim 8, wherein:

a pattern light is formed by a light of an invisible-region of a wavelength band by using infrared or ultraviolet light as the light source; and

the pattern projection image and intensity image are picked up in parallel.

13. The image processing method according to claim 8, further comprising the step of:

for an area where the amount of change of the pattern picked up by the first image pickup part with respect to the projection pattern by the projecting part is less than a predetermined value, creating second range information by deriving a correspondence between intensity information obtained by the first and second image pickup parts.

14. The image processing method according to claim 8, wherein:

the second image pickup part includes plural image pickup parts that pick up the measurement target at different angles, and includes the step of creating range information based on projection patterns respectively picked up by the plural image pickup parts of the second image pickup part.

15. An image processing apparatus, comprising:

a projecting part that projects light from a baseline to an image holding medium to form an image thereon, the projecting part associated with a re-coding part that creates a code imposed in the image;

an image pickup part that picks up the image on the image holding medium projected by the projecting part, the image pickup part including a decoding part that detects the code created in the re-coding part;

an intensity image acquisition part that acquires an intensity image based on the image picked up by the image pickup part;

a range information acquisition part that acquires range information from the picked-up image by determining a distance between the image holding medium and the baseline based on the code;

a geometric transformation part that performs geometric transformation for the intensity image based on the range information acquired in the range information acquisition part;

an image extracting part that extracts difference between a geometric-transformed intensity image and an intensity image acquired in advance;

a storage part that stores, as the geometric-transformed intensity image, an initial geometric-transformed intensity image in a time-series transformed by the geometric transformation part;

the image extracting part making comparison between successive geometric-transformed intensity images in the time-series transformed by the geometric transformation part; and

an image processing part that retrieves only differential data between successive geometric-transformed intensity images in the time-series as storage data based on a result of the comparison of the image extracting part of the initial geometric-transformed intensity image and geometric-transformed intensity images subsequently transformed in the time-series, wherein the stored geometric-transformed intensity image is the initial geometric-transformed intensity image and the differential data between successive geometric-transformed intensity images in the time-series.

16. The image processing apparatus according to claim 15, wherein the image holding medium is one of a manuscript sheet, whiteboard, blackboard, screen, wall, and screen projection sheet.

17. The image processing apparatus according to claim 15, wherein the intensity image acquired in advance as a processing target in the image extracting part is a preceding frame image inputted precedent to the geometric transformation part.

18. The image processing apparatus according to claim 15, further comprising:

a storage part that stores image data, wherein the intensity image acquired in advance as a processing target in the image extracting part is the image data stored in advance in the storage part.

19. The image processing apparatus according to claim 15, further comprising:

a document database in which plural pieces of document format data are stored; and

a document identifying part that performs matching between the geometric-transformed intensity image and the pieces of document format data stored in the document database,

wherein the image extracting part extracts differences between the geometric-transformed intensity image and the pieces of document format data stored in the document database.

20. The image processing apparatus according to claim 15, further comprising a character transformation processing part that reads character data extracted by the image extracting part and identifies the character data with identification data.

21. The image processing apparatus according to claim 15, further comprising:

an authentication information database in which handwriting history data of registered users is stored; and

an authentication processing part that inputs the geometric-transformed intensity image and performs matching between the input image and handwriting history data stored in the authentication information database.

22. The image processing apparatus according to claim 21, wherein:

the authentication information database stores handwriting history data and signature shape data of registered users; and

the authentication processing part inputs the geometric-transformed intensity image and performs matching between the input image and the handwriting history data stored in the authentication information database, and between the input image and the signature shape data.

23. The image processing method according to claim 15, further comprising a display part that displays an image produced as a result of performing geometric transformation for the intensity image, based on the range information in the geometric transformation part.

24. The image processing apparatus claim 15, further comprising:

a storage part that stores range information acquired by the range information acquisition part,

wherein a distance between the image holding medium and the image pickup part is fixed, and the geometric transformation part performs geometric transformation for the intensity image based on the range information stored in the storage part.

25. An image processing method, comprising:

projecting light from a baseline to an image holding medium to form an image thereon;

creating a code to be imposed in the image;

picking up the image projected on the image holding medium and detecting the code imposed on the image;

acquiring an intensity image based on the image picked up in the image pickup step;

acquiring range information from the picked-up image by determining a distance between the image holding medium and the baseline based on the code;

performing geometric transformation for the intensity image based on the range information acquired in the range information acquisition step; and

extracting difference between the geometric-transformed intensity image and an intensity image acquired in advance;

storing, as the geometric-transformed intensity image, an initial geometric-transformed intensity image in a time-series transformed in the geometric transformation step;

making comparison between successive geometric-transformed intensity images in the time-series transformed in the geometric transformation step; and

retrieving only differential data between successive geometric-transformed intensity images in the time-series as storage data based on a result of the comparison step of the initial geometric-transformed intensity image and geometric-transformed intensity images subsequently transformed in the time-series, wherein the stored geometric-transformed intensity image is the initial geometric-transformed intensity image and the differential data between successive geometric-transformed intensity images in the time-series.

26. The image processing method according to claim 25, wherein the image holding medium is one of a manuscript sheet, whiteboard, blackboard, screen, wall and screen projection sheet.

27. The image processing method according to claim 25, wherein the intensity image acquired in advance as a processing target in the image extracting step is a preceding frame image inputted precedent to the geometric transformation step.

28. The image processing method according to claim 25, wherein the intensity image acquired in advance as a processing target in the image extracting step is image data stored in advance in a storage part.

29. The image processing method according to claim 25, further comprising:

storing plural pieces of document format data in a document database; and

performing matching between a geometric-transformed intensity image and the pieces of document format data stored in the document database,

wherein the image extracting step extracts difference between the geometric-transformed intensity image and the pieces of document format data stored in the document database.

30. The image processing method according to claim 25, further comprising:

reading character data extracted in the image extracting step and identifies the character data with identification data.

31. The image processing method according to claim 25, further comprising:

storing handwriting history data of registered users in a authentication information database; and

inputting the geometric-transformed intensity image and performing matching between the input image and the handwriting history data stored in the authentication information database.

32. The image processing apparatus according to claim 31, further comprising:

in addition to the handwriting history data, storing signature shape data of registered users in the authentication information database; and

inputting the geometric-transformed intensity image and performing matching between the input image and the handwriting history data stored in the authentication information database, and between the input image and the signature shape data.

33. The image processing method according to claim 25, further comprising:

displaying an image produced as a result of performing geometric transformation for the intensity image based on the range information.

34. The image processing method according to claim 25, further comprising:

storing range information acquired in the range information acquiring step, wherein a distance between an image holding medium and the image pickup part is fixed and the geometric transformation for the intensity image is performed based on the range information stored in the storage step.

35. A storage medium readable by a computer, the storage medium storing a program of instructions executable by the computer to perform method steps for performing image processing, the method comprising the steps of:

projecting light from a baseline to an image holding medium to form an image thereon;

creating a code to be imposed on the image;

picking up the image formed on the image holding medium;

detecting the code created in the re-coding part;

acquiring an intensity image based on the image picked up in the image pickup step;

acquiring range information from the picked-up image by determining a distance between the image holding medium and the baseline based on the code;

performing geometric transformation for the intensity image based on the range information acquired in the range information acquisition step; and

extracting difference between the geometric-transformed intensity image and an intensity image acquired in advance;

storing, as the geometric-transformed intensity image, an initial geometric-transformed intensity image in a time-series transformed in the geometric transformation step;

making comparison between successive geometric-transformed intensity images in the time-series transformed in the geometric transformation step; and

retrieving only differential data between successive geometric-transformed intensity images in the time-series as storage data based on a result of the comparison step of the initial geometric-transformed intensity image and geometric-transformed intensity images subsequently transformed in the time-series, wherein the stored geometric-transformed intensity image is the initial geometric-transformed intensity image and the differential data between successive geometric-transformed intensity images in the time-series.

36. An image processing apparatus, comprising:

a projecting part that projects light from a baseline, the projecting part associated with a re-coding part that a creates code imposed in the image;

an image pickup part that picks up the projected light, the image pickup part including a decoding part that detects the code created in the re-coding part;

an intensity image acquisition part that acquires an intensity image from the picked-up light;

a range information acquisition part that acquires range information from the picked-up light by determining a distance from the baseline based on the code;

a geometric transformation part that performs geometric transformation for the intensity image based on the range information; and

an image extracting part that extracts difference between the geometric-transformed intensity image and an intensity image acquired in advance;

a storage part that stores, as the geometric-transformed intensity image, an initial geometric-transformed intensity image in a time-series transformed by the geometric transformation part;

the image extracting part making comparison between successive geometric-transformed intensity images in the time-series transformed by the geometric transformation part; and

an image processing part that retrieves only differential data between successive geometric-transformed intensity images in the time series as storage data based on a result of the comparison of the image extracting part of the initial geometric-transformed intensity image and geometric-transformed intensity images subsequently transformed in the time-series, wherein the stored geometric-transformed intensity image is the initial geometric-transformed intensity image and the differential data between successive geometric-transformed intensity images in the time-series.

37. An image processing method, comprising:

projecting coded light from a baseline;

picking up and decoding the projected coded light to produce picked-up light;

acquiring an intensity image from the picked-up light;

acquiring range information from the picked-up light by determining a distance from the baseline;

performing geometric transformation for the intensity image based on the range information; and

extracting difference between the geometric-transformed intensity image and an intensity image acquired in advance;

storing, as the geometric-transformed intensity image, an initial geometric-transformed intensity image in a time-series transformed in the geometric transformation step;

making comparison between successive geometric-transformed intensity images in the time-series transformed in the geometric transformation step; and

retrieving only differential data between successive geometric-transformed intensity images in the time-series as storage data based on a result of the comparison step of the initial geometric-transformed intensity image and geometric-transformed intensity images subsequently transformed in the time-series, wherein the stored geometric-transformed intensity image is the initial geometric-transformed intensity image and the differential data between successive geometric-transformed intensity images in the time-series.

38. A storage medium readable by a computer, the storage medium storing a program of instructions executable by the computer to perform method steps for performing image processing, the method comprising the steps of:

projecting coded light from a baseline;

picking up and decoding the projected coded light to produce picked-up light;

acquiring an intensity image from the picked-up light;

acquiring range information from the picked-up light by determining a distance from the baseline;

performing geometric transformation for the intensity image based on the range information; and

extracting difference between the geometric-transformed intensity image and an intensity image acquired in advance;

storing, as the geometric-transformed intensity image, an initial geometric-transformed intensity image in a time-series transformed in the geometric transformation step;

making comparison between successive geometric-transformed intensity images in the time-series transformed in the geometric transformation step; and

retrieving only differential data between successive geometric-transformed intensity images in the time-series as storage data based on a result of the comparison step of the initial geometric-transformed intensity image and geometric-transformed intensity images subsequently transformed in the time-series, wherein the stored geometric-transformed intensity image is the initial geometric-transformed intensity image and the differential data between successive geometric-transformed intensity images in the time-series.