System and method for displaying an image indicating a positional relation between partially overlapping images

• US 6,798,924 B2
• Filed: 09/03/2002
• Issued: 09/28/2004
• Est. Priority Date: 09/10/1997
• Status: Expired due to Term

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Aoki, Shin, Saitoh, Takashi, Haike, Guan, Sakamoto, Takuji, Ejiri, Koichi
• Primary Examiner(s): Mehta, Bhavesh M.
• Assistant Examiner(s): Kassa, Yosef
• Application Number:
  US10/232,516
• Publication Number:
  US 20030002750A1
• Time in Patent Office: 756 Days
• Field of Search: 382/151, 382/216, 382/218, 382/219, 382/282, 382/284, 382/291, 382/293, 382/294, 382/295, 358/448, 358/450, 358/453, 348/36, 348/39, 348/49, 348/50, 396/326, 396/332
• US Class Current: 382/284
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]
• CPC Class Codes: G06T 2200/32 involving image mosaicing
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 7/223 using block-matching
• CPC Class Codes: G06T 7/30 Determination of transform parameters for the alignment of images, i.e. image registration
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: H04N 23/635 Region indicators; Field of view indicators
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 5/772 the recording apparatus and the television camera being placed in the same enclosure
• CPC Class Codes: H04N 5/907 using static stores, e.g. storage tubes or semiconductor memories H04N5/91 takes precedence

Abstract

A camera system includes a display monitor which displays an image of an object, taken by an optical unit, on a screen of the monitor. A reading unit reads a preceding image and a current image among a plurality of partially overlapping images, from a memory device, the preceding image and the current image containing a common element. A determining unit determines a positional relation between the preceding image and the current image based on a common pattern derived from the common element in the two adjacent images read by the reading unit. A displaying unit displays an image indicating a boundary of the preceding image on the screen of the monitor at a shifted position according to the positional relation determined by the determining unit, with the current image concurrently displayed on the screen of the monitor.

Claims

1. A camera comprising:

a display monitor configured to display on a screen of the display monitor at least one of a current image and a preceding image taken by an optical unit;

a storage unit configured to store the preceding image; and

a control unit configured to display a part of the preceding image on the screen when the current image is displayed on the screen, the displayed part of the preceding image indicating a relation between the current image and the preceding image at a shifted position,

wherein the current image is taken by referring to the displayed part of the preceding image on the screen.

2. A camera comprising:

a display monitor configured to display on a screen of the display monitor at least one of a current image and a preceding image taken by an optical unit;

a storage unit configured to store the preceding image; and

a control unit configured to display a part of the preceding image on the screen when the current image is displayed on the screen, the displayed part of the preceding image indicating a common boundary portion between the current image and the preceding image at a shifted position,

wherein the current image is taken at a position at which a portion of the current image and the displayed part of the preceding image are overlapped and displayed on the screen of the display monitor.

3. The camera of claim 2, further comprising:

a determining unit configured to determine a positional relation between the preceding image and the current image based on a common part in the preceding image and the current image.

4. The camera of claim 3, wherein the display monitor is configured to display a displacement vector, which corresponds to the positional relation between the preceding image and the current image, during movement of an optical axis of the optical unit.

5. The camera of claim 3, wherein the determining unit is configured to perform a matching between corresponding blocks taken from an overlapping portion of the preceding image and the current image, so that a maximum-similarity common part in the preceding image and the current image is identified.

6. The camera of claim 3, wherein the determining unit is configured to perform a matching between corresponding blocks taken from an overlapping portion of the preceding image and the current image by checking intensities of individual pixels of the corresponding blocks.

7. The camera of claim 3, wherein the determining unit is configured to determine both coordinates of a central pixel of a maximum-similarity common part in the preceding image and coordinates of a central pixel of the maximum similarity common part in the current image.

8. The camera of claim 3, wherein the determining unit is configured to determine a displacement vector, which indicates a positional relation between the preceding image and the current image, based on a difference between coordinates of a central pixel of a maximum-similarity common part in the preceding image and coordinates of a central pixel of the maximum-similarity common part in the current image, and wherein the displaying unit is configured to display an image of the displacement vector on the display monitor with the current image concurrently displayed on the display monitor.

9. The camera of claim 3, further comprising:

a sensor configured to output an optical axis direction signal indicating a direction of an optical axis of the optical unit; and

a secondary determining unit configured to determine a positional relation between the preceding image and the current image based on a difference between the optical axis direction signal output by the sensor with respect to the current image and the optical axis direction signal output by the sensor with respect to the preceding image.

10. A camera system comprising:

a display monitor configured to display on a screen of the display monitor at least one of a current image and a preceding image taken by an optical unit;

a determining unit configured to determine a positional relation between the preceding image and the current image based on a common part in the preceding image and the current image;

a control unit configured to display a part of the preceding image on the screen when the current image is displayed on the screen, the displayed part of the preceding image indicating a common boundary portion between the current image and the preceding image at a shifted position,

wherein the current image is taken at a position at which a portion of the current image and the displayed part of the preceding image are overlapped and displayed on the screen of the display monitor, and

an image storing unit configured to store images taken by the optical unit in an image memory, wherein the image storing unit stores auxiliary data, which contains information indicating the positional relation from the determining unit, in the image memory, in addition to the images stored therein.

11. The camera system of claim 10, wherein the display monitor is configured to display a displacement vector, corresponding to the positional relation between the preceding image and the current image, during movement of an optical axis of the optical unit.

12. The camera system of claim 10, wherein the determining unit is configured to perform a matching between corresponding blocks taken from an overlapping portion of the preceding image and the current image, so that a maximum-similarity common part in the preceding image and the current image is identified.

13. The camera system of claim 10, wherein the determining unit is configured to perform a matching between corresponding blocks taken from an overlapping portion of the preceding image and the current image by checking intensities of individual pixels of the corresponding blocks.

14. The camera system of claim 10, wherein the determining unit is configured to determine both coordinates of a central pixel of a maximum-similarity common part in the preceding image and coordinates of a central pixel of the maximum-similarity common part in the current image.

15. The camera system of claim 10, wherein the determining unit is configured to determine a displacement vector, which indicates a positional relation between the preceding image and the current image, based on a difference between coordinates of a central pixel of a maximum-similarity common part in the preceding image and coordinates of a central pixel of the maximum-similarity common part in the current image, and wherein the displaying unit is configured to display an image of the displacement vector on the display monitor with the current image concurrently displayed on the display monitor.

16. A camera system comprising:

a display monitor configured to display on a screen of the display monitor at least one of a current image and a preceding image taken by an optical unit;

a determining unit configured to determine a relation between the preceding image and the current image based on a common part in the preceding image and the current image;

a control unit configured to display a part of the preceding image on the screen when the current image is displayed on the screen, the displayed part of the preceding image indicating a common boundary portion between the current image and the preceding image at a shifted position,

wherein the current image is taken at a position at which a portion of the current image and the displayed part of the preceding image are overlapped and displayed on the screen of the display monitor;

a sensor configured to output an optical axis direction signal indicating a direction of an optical axis of the optical unit;

a secondary determining unit configured to determine a positional relation between the preceding image and the current image based on a difference between the optical axis direction signal output by the sensor with respect to the current image and the optical axis direction signal output by the sensor with respect to the preceding image; and

an image storing unit configured to store an image of the object taken by the optical unit in an image memory, wherein the image storing unit stores auxiliary data, which contains information indicating the positional relation from the secondary determining unit, in the image memory, in addition to the image stored therein.

17. The camera system of claim 16, wherein the display monitor is configured to display a displacement vector, which corresponds to the positional relation between the preceding image and the current image, during movement of an optical axis of the optical unit.

18. The camera system of claim 16, wherein the determining unit is configured to perform a matching between corresponding blocks taken from an overlapping portion of the preceding image and the current image, so that a maximum-similarity common part in the preceding image and the current image is identified.

19. The camera system of claim 16, wherein the determining unit is configured to perform a matching between corresponding blocks taken from an overlapping portion of the preceding image and the current image by checking intensities of individual pixels of the corresponding blocks.

20. The camera system of claim 16, wherein the determining unit is configured to determine both coordinates of a central pixel of a maximum-similarity common part in the preceding image and coordinates of a central pixel of the maximum-similarity common part in the current image.

21. The camera system of claim 16, wherein the determining unit is configured to determine a displacement vector, which indicates a positional relation between the preceding image and the current image, based on a difference between coordinates of a central pixel of a maximum-similarity common part in the preceding image and coordinates of a central pixel of the maximum-similarity common part in the current image, and wherein the displaying unit is configured to display an image of the displacement vector on the display monitor with the current image concurrently displayed on the display monitor.

22. A computer program product for use with a camera system having a display monitor, an image storing unit, and an optical unit, the computer program product comprising:

a computer storage medium; and

a computer program code mechanism embedded in the computer storage medium for causing the camera system to display at least two partially overlapping images taken by said optical unit, wherein the at least two partially overlapping images include a preceding image and a current image, the computer program code mechanism including;

a first computer code means for causing the display monitor to display a part of the preceding image when the display monitor displays the current image, the displayed part of the preceding image indicating a common boundary portion between a current image and the preceding image at a shifted position, wherein the current image is taken at a position at which a portion of the current image and the displayed part of the preceding image are overlapped and displayed on a screen of the display monitor; and

further computer code means for causing the image storing unit to store the preceding image in an image memory.

23. The computer program product of claim 22, wherein the computer program code mechanism further includes:

a second computer code means for causing a determining unit included in the camera system to determine a relation between a preceding image and a current image based on a common part in the preceding image and the current image.

24. The computer program product of claim 23, wherein the determining unit performs matching between corresponding blocks taken from an overlapping portion of the preceding image and the current image, so that a maximum-similarity common part in the preceding image and the current image is identified.

25. The computer program product of claim 23, wherein the determining unit performs matching between corresponding blocks taken from an overlapping portion of the preceding image and the current image by checking intensities of individual pixels of the corresponding blocks.

26. The computer program product of claim 23, wherein the determining unit determines both coordinates of a central pixel of a maximum similarity common part in the preceding image and coordinates of a central pixel of the maximum-similarity common part in the current image.

27. The computer program product of claim 23, wherein the computer program code mechanism further includes:

third computer code means for causing a sensor included in the camera system to output an optical axis direction signal indicating a direction of an optical axis of the optical unit; and

fourth computer code means for causing a secondary determining unit to determine a positional relation between the preceding image and the current image based on a difference between the optical axis direction signal output by the sensor with respect to the current image and the optical axis direction signal output by the sensor with respect to the preceding image.

28. The computer program product of claim 27, wherein the computer program code mechanism further includes:

sixth computer code means for causing the image storing unit to store an image of the object taken by the optical unit in the image memory, wherein auxiliary data, which contains information indicating the positional relation from the secondary determining unit, is stored in the image memory in addition to the image stored therein.

29. The computer program product of claim 23, wherein the determining unit determines a displacement vector, which indicates a positional relation between the preceding image and the current image, based on a difference between coordinates of a central pixel of a maximum-similarity common part in the preceding image and coordinates of a central pixel of the maximum-similarity common part in the current image, and the display monitor displays an image of the displacement vector with the current image concurrently displayed.

30. The computer program product of claim 23, wherein the computer program code mechanism further includes:

fifth computer code means for causing the image storing unit to store an image of the object taken by the optical unit in the image memory, wherein auxiliary data, which contains information indicating the positional relation from the determining unit, is stored in the image memory in addition to the image stored therein.

31. The computer program product of claim 23, wherein the display monitor displays a displacement vector, which corresponds to the positional relation between the preceding image and the current image, during movement of an optical axis of the optical unit.

32. The computer program product of claim 22, wherein the computer storage medium is arranged as a read-only memory.

33. The computer program product of claim 22, wherein the computer program code mechanism is arranged as a plurality of programs.

34. A method of capturing images in which a preceding image and a current image are displayed, comprising:

storing the preceding image;

displaying a part of the preceding image when the current image is displayed, the displayed part of the preceding image indicating a relation between the current image and the preceding image at a shifted position; and

capturing the current image by referring to the displayed part of the preceding image.

35. The method of claim 34, further comprising:

determining a relation between the preceding image and the current image based on a common part in the preceding image and the current image, wherein,

the displayed part of the preceding image indicates a common boundary portion between the current image and the preceding image at a shifted position, and

the current image is taken at a position at which a portion of the current image and the common part of the preceding image are overlapped and displayed.

36. The method of claim 35, wherein, in the determining step, a matching between corresponding blocks taken from an overlapping portion of the preceding image and the current image is performed by checking intensities of individual pixels of the corresponding blocks.

37. The method of claim 35, wherein, in the determining step, both coordinates of a central pixel of a maximum-similarity common part in the preceding image and coordinates of a central pixel of the maximum-similarity common part in the current image are determined.

38. The method of claim 35, further comprising:

outputting an optical axis direction signal indicating a direction of an optical axis; and

determining a positional relation between the preceding image and the current image based on a difference between the optical axis direction signal with respect to the current image and the optical axis direction signal with respect to the preceding image.

39. The method of claim 38, wherein the storing step includes:

storing auxiliary data, which contains information indicating the positional relation from the positional relation determining step.

40. The method of claim 35, wherein, in the determining step, a displacement vector, which indicates a positional relation between the preceding image and the current image, is determined based on a difference between coordinates of a central pixel of a maximum-similarity common part in the preceding image and coordinates of a central pixel of the maximum-similarity common part in the current image, and wherein, in the displaying step, an image of the displacement vector is displayed with the current image concurrently displayed.

41. The method of claim 35, wherein the storing step includes:

storing auxiliary data, which contains information indicating the positional relation from the determining step.

42. The method of claim 35, further comprising:

displaying a displacement vector, corresponding to the positional relation between the preceding image and the current image, during movement of an optical axis.

43. The method of claim 35, wherein, in the determining step, a matching between corresponding blocks taken from an overlapping portion of the preceding image and the current image is performed, so that a maximum-similarity common part in the preceding image and the current image is identified.

44. A camera comprising:

a display unit configured to display at least one of a current image and a preceding image taken by an optical unit;

a storage unit configured to store the preceding image; and

means for controlling the display unit to display a part of the preceding image when the display unit displays the current image, the displayed part of the preceding image indicating a common boundary portion between the current image and the preceding image on the displaying means at a shifted position,

wherein the current image is taken at a position at which a portion of the current image and the displayed part of the preceding image are overlapped and displayed on the display unit.

45. The camera of claim 44, further comprising:

means for determining a positional relation between the preceding image and the current image based on a common part in the preceding image and the current image.

46. The camera of claim 45, wherein the display unit is configured to display a displacement vector, corresponding to the positional relation between the preceding image and the current image, during movement of an optical axis of the optical unit.

47. The camera of claim 45, wherein the determining means performs a matching between corresponding blocks taken from an overlapping portion of the preceding image and the current image, so that a maximum-similarity common part in the preceding image and the current image is identified.

48. The camera of claim 45, wherein the determining means performs a matching between corresponding blocks taken from an overlapping portion of the preceding image and the current image by checking intensities of individual pixels of the corresponding blocks.

49. The camera of claim 45, wherein the determining means determines both coordinates of a central pixel of a maximum-similarity common part in the preceding image and coordinates of a central pixel of the maximum similarity common part in the current image.

50. The camera of claim 45, wherein the determining means determines a displacement vector, indicating a positional relation between the preceding image and the current image, based on a difference between coordinates of a central pixel of a maximum-similarity common part in the preceding image and coordinates of a central pixel of the maximum-similarity common part in the current image, and wherein the display unit is configured to display an image of the displacement vector with the current image concurrently displayed on the displaying means.

51. The camera of claim 45, further comprising:

means for outputting an optical axis direction signal indicating a direction of an optical axis of the optical unit; and

second means for determining a positional relation between the preceding image and the current image based on a difference between the optical axis direction signal output by the sensor with respect to the current image and the optical axis direction signal output by the outputting means with respect to the preceding image.

52. A camera system comprising:

a display monitor configured to display on a screen of the display monitor at least one of a current image and a preceding image taken by an optical unit;

a determining unit configured to display a relation between the preceding image and the current image based on a common part in the preceding image and the current image;

a control unit configured to display a part of the preceding image on the screen when the current image is displayed on the screen, the displayed part of the preceding image indicating a positional relation between the current image and the preceding image at a shifted position,

wherein the current image is taken by referring to the displayed part of the preceding image on the screen of the display monitor, and

an image storing unit configured to store images taken by the optical unit in an image memory, wherein the image storing unit stores auxiliary data, which contains information indicating the positional relation from the determining unit, in the image memory, in addition to the images stored therein.

53. A camera system comprising:

a display monitor configured to display on a screen of the display monitor at least one of a current image and a preceding image taken by an optical unit;

a determining unit configured to determine a relation between the preceding image and the current image based on a common part in the preceding image and the current image;

a control unit configured to display a part of the preceding image on the screen when the current image is displayed on the screen, the displayed part of the preceding image indicating a positional relation between the current image and the preceding image at a shifted position,

wherein the current image is taken by referring to the displayed part of the preceding image on the screen of the display monitor;

a sensor configured to output an optical axis direction signal indicating a direction of an optical axis of the optical unit;

a secondary determining unit configured to determine a positional relation between the preceding image and the current image based on a difference between the optical axis direction signal output by the sensor with respect to the current image and the optical axis direction signal output by the sensor with respect to the preceding image; and

an image storing unit configured to store an image of the object taken by the optical unit in an image memory, wherein the image storing unit stores auxiliary data, which contains information indicating the positional relation from the secondary determining unit, in the image memory, in addition to the image stored therein.

54. A computer program product for use with a camera system having a display monitor, an image storing unit, and an optical unit, the computer program product comprising:

a computer storage medium; and

a computer program code mechanism embedded in the computer storage medium for causing the camera system to display at least two images taken by said optical unit, wherein the at least two images include a preceding image and a current image, the computer program code mechanism including;

a first computer code means for causing the display monitor to display a part of the preceding image when the display monitor displays the current image, the displayed part of the preceding image indicating a positional relation between the current image and the preceding image at a shifted position, wherein the current image is taken by referring to the displayed part of the preceding image on a screen of the display monitor; and

further computer code means for causing the image storing unit to store the preceding image in an image memory.

55. A method of capturing images in which a preceding image and a current image are displayed, comprising the steps of:

storing the preceding image and auxiliary data corresponding to the preceding image, wherein the auxiliary data includes a part of the preceding image;

displaying the part of the preceding image when the current image is displayed, the displayed part of the preceding image indicating a relation between the current image and the preceding image at a shifted position; and

capturing the current image by referring to the displayed part of the preceding image.

56. A camera comprising:

a display unit configured to display at least one of a current image and a preceding image taken by an optical unit;

a storage unit configured to store the preceding image; and

means for controlling the displaying means to display a part of the preceding image when the displaying means displays the current image, the displayed part of the preceding image indicating a positional relation between the current image and the preceding image on the displaying means at a shifted position,

wherein the current image is taken by referring to the diplayed part of the preceding image on the displaying means.