Remote image display method, image capturing device, and method and program therefor

• US 7,385,624 B2
• Filed: 01/15/2004
• Issued: 06/10/2008
• Est. Priority Date: 01/17/2003
• Status: Active Grant

• Current Assignee: Nippon Telegraph and Telephone Corporation
• Original Assignee: Nippon Telegraph and Telephone Corporation
• Inventors: Watanabe, Tomoki, Kishida, Katsumi, Takehara, Nobuhiko, Anzai, Hiroki
• Primary Examiner(s): Vu; Ngoc-Yen
• Assistant Examiner(s): Meyers; James A
• Application Number:
  US10/510,954
• Publication Number:
  US 20050212909A1
• Time in Patent Office: 1,608 Days
• Field of Search: 348/36, 348/39, 348/155, 348/211.3, 348/211.11, 348/218.1, 348/222.1, 348/552
• US Class Current: 348/36
• CPC Class Codes: H04N 21/21805 enabling multiple viewpoints, e.g. using a plurality of cameras
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/6587 Control parameters, e.g. trick play commands, viewpoint selection
• CPC Class Codes: H04N 23/63 by using electronic viewfinders
• CPC Class Codes: H04N 23/661 Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources
• CPC Class Codes: H04N 7/185 from a mobile camera, e.g. for remote control

Abstract

An omnidirectional camera performs image sensing of a panoramic object 6 surrounding it, and sends the video signal to an image capturing device 1, then the image capturing device 1 extracts a video signal of a partial object 61 and sends the extracted video signal to a camera-equipped portable terminal 3 carried in user'"'"'s hand at a place distant from the omnidirectional camera 2 to display the image of the video signal on a display surface 32a of the portable terminal, and the user shoots a surrounding object 8 with the portable terminal 3 and sends the surrounding video signal to the device 1. When the shooting direction of the portable terminal 3 is turned as indicated by the broken line, the device 1 detects a change in the shooting direction of he portable terminal 3 from a difference between the surrounding video signals generated before and after the turning of the portable terminal, and extracts, from a panorama video signal, a video signal of a partial object 62 turned from the partial object 61 in correspondence to the difference and sends the extracted video signal to the portable terminal 3 to display the image of the video signal.

Claims

1. A remote image display method, comprising:

image sensing a panoramic object by a remote image sensing device;

sending a video signal of a part of said image-sensed panoramic object via an image capturing device to an image display portion at a place different from said remote image sensing device;

playing back and displaying said received video signal by said image display portion as an image of a part of said panoramic object;

image sensing surroundings by a camera-equipped portable terminal at a position where a display on said image display portion can be seen, and sending a surrounding video signal to said image capturing device;

obtaining variation information about surrounding image by a variation detecting portion in said image capturing devices, based on a previous surrounding video signal and a current surrounding video signal received from said camera-equipped portable terminal;

obtaining from said remote image sensing device a video signal of a part of said panoramic object changed corresponding to a change of said surrounding image based on said variation information; and

sending said obtained video signal to said image display portion.

2. An image capturing device comprising:

a surrounding image receiving portion for receiving a surrounding video signal from a camera-equipped portable terminal;

a variation detecting portion for detecting, from said received surrounding video signal, variation information of an image of the received surrounding video signal from an image of a previous surrounding video signal having been previously received by said surrounding image receiving portion, the image of the received surrounding video signal hereinafter being referred to as the current image, the image of the previous surrounding video signal hereinafter being referred to as the previous image;

a capturing signal generating portion for generating a capturing signal from said variation information which is used to obtain a video signal of a part of a panoramic object;

a signal sending portion for sending said capturing signal to a remote image sensing device that performs image sensing of a panoramic object and outputs a video signal; and

an image relay portion for receiving said video signal from said remote image sensing device and for sending the received video signal to an image display portion at the same position as said camera-equipped portable terminal.

3. The device of claim 2, wherein:

said variation detecting portion detects, as said variation information, direction information corresponding to a direction of movement of said current image with respect to said previous image or/and zoom information corresponding to a change in a size of a part of said current image corresponding to a part of said previous image; and

said capturing signal generating portion generates, as said capturing signal, an extracting reference pixel position signal obtained by correcting a previous extracting reference pixel position signal according to said direction information or/and an image zoom in/out signal corresponding to said zoom information.

4. The device of claim 2, further comprising:

a camera direction storage portion which stores identification information of each of plural camera devices provided in said remote image sensing device and angularly spaced apart in a shooting direction of the plural camera devices; and

wherein;

said variation detecting portion detects, as said variation information, direction information corresponding to a direction of movement of said current image with respect to said previous image; and

said capturing signal generating portion determines the shooting direction by referring to the identification information stored in said camera direction storage portion based on a previously generated transmitting camera identification information signal and said direction information, for generating, as said capturing signal, a transmitting camera identification information signal of the camera identification information corresponding to said shooting direction determined by said capturing signal generating portion.

5. The device of claim 2, wherein:

said variation detecting portion detects, as said variation information, direction information corresponding to a direction of movement of said current image with respect to said previous image or/and zoom information corresponding to a change in a size of said current image with respect to said previous image; and

said capturing signal generating portion generates, as said capturing signal, a shooting direction change signal corresponding to said direction information or/and zoom change signal corresponding to said zoom information.

6. The device of claim 2, wherein:

said surrounding image receiving portion receives the surrounding video signal from each of a plurality of camera-equipped portable terminals; and

said variation detecting portion is provided with a previous frame memory for storing a surrounding video signal used for detecting the previous variation information for each camera-equipped portable terminal, for detecting variation information corresponding to said each camera-equipped portable terminal having transmitted the surrounding video signal, based on the received surrounding video signal and the surrounding video signal stored in said previous frame memory corresponding to said each camera-equipped portable terminal having transmitted the surrounding video signal;

said capturing signal generating portion is provided with a previous signal memory for storing a previously generated capturing signal for said each camera-equipped portable terminal having transmitted the surrounding video signal, for generating a capturing signal corresponding to said each camera-equipped portable terminal from the variation information corresponding to said each camera-equipped portable terminal and said previously generated capturing signal stored in said previous signal memory corresponding to said each camera-equipped portable terminal, and for updating corresponding one of capturing signals in said previous signal memory with said generated capturing signal;

said signal portion sends said generated capturing signal and camera-equipped portable terminal identification information for identifying the camera-equipped portable terminal corresponding thereto; and

said image relay portion sends the received video signal to said image display means portion of the camera-equipped portable terminal indicated by the camera-equipped portable terminal identification information received together with said video signal.

12. The device of claim 2 or 7, further comprising:

a variation hysteresis storage portion for storing hysteresis information of said variation information;

a decision portion for deciding whether the variation information detected by said variation detecting portion is abnormal or not by referring to the hysteresis information stored in said variation hysteresis storage portion; and

a portion which, when said decision portion decides that said variation information is abnormal, inhibits a supply of said detected variation information to said capturing signal generating portion.

7. An image capturing device comprising:

a surrounding image receiving portion for receiving a surrounding video signal from a camera-equipped portable terminal;

a variation detecting portion for detecting, from said received surrounding video signal, variation information of an image of the received surrounding video signal from an image of a previous surrounding video signal having been previously received by said surrounding image receiving means portion, the image of the received surrounding video signal hereinafter being referred to as the current image, the image of the previous surrounding video signal hereinafter being referred to as the previous image;

a capturing signal generating portion for generating, from said variation information and a previously generated capturing signal, a capturing signal that is used to obtain a video signal of a part of a panoramic object;

a remote image receiving portion for receiving a video signal sent from a remote image sensing device having performed image sensing of a panoramic object;

an image capturing portion for capturing the video signal of a part of the panoramic object from said received video signal based on said capturing signal; and

an image sending portion for sending said captured video signal to an image display portion at a same position as that of said camera-equipped portable terminal.

8. The device of claim 7, wherein:

the video signal received by said remote image receiving portion is a panorama video signal;

said variation detecting portion detects, as said variation information, direction information corresponding to a direction of movement of said current image with respect to said previous image or/and zoom information corresponding to a change in a size of a part of said current image corresponding to a part of said previous image;

said capturing signal generating portion generates, as said capturing signal, an extracting reference pixel position signal obtained by correcting a previous extracting reference pixel position signal according to said direction information or/and an image zoom in/out signal corresponding to said zoom information; and

said image capturing portion captures, from said panorama video signal, one frame of video signal produced by zooming in/out, by said image zoom in/out signal relative to the image of the previously sent captured video signal, a partial image of the panorama image of said panorama video signal that is defined by one frame of video signal or/and the previously generated extracting reference pixel position signal, or said corrected extracting reference pixel position signal with reference to the position on the panorama image that is determined by said corrected extracting reference pixel position signal.

9. The device of claim 7, wherein:

the video signal received by said remote image receiving portion is video signals from a plurality of camera devices of said remote image sensing device with their shooting directions angularly spaced apart; and

said image capturing device further comprises;

a camera direction storage portion having stored herein identification information about said plurality of camera devices and information corresponding to the shooting directions; and

said variation detecting portion detects, as said variation information, direction information corresponding to a direction of movement of said current image with respect to said previous image;

said capturing signal generating portion determines a shooting direction by referring to the identification information stored in said camera direction storage portion based on a previously generated transmitting camera identification information signal and said direction information, for generating, as said capturing signal, a transmitting camera identification information signal of the camera identification information corresponding to said determined shooting direction; and

said image capturing portion captures one of a plurality of video signals corresponding to said generated transmitting camera identification information signal.

10. The device of claim 9, wherein:

said variation detecting portion includes a portion for detecting, as said variation information, second direction information corresponding to the direction of movement of said current image with respect to said previous image in a direction perpendicular to the direction of movement of said detected direction information or/and zoom information corresponding to a change in the size of the part of said current image corresponding to a part of said previous image;

said capturing signal generating portion generates, as said capturing signal, an extracting reference pixel position signal obtained by correcting a previous extracting reference pixel position signal according to said second direction information or/and an image zoom in/out signal corresponding to said zoom information; and

said image capturing portion captures, from said captured video signal, one frame of video signal produced by zooming in/out, by said image zoom in/out signal relative to the image of the previously sent captured video signal, a partial image of the image of said captured video signal that is defined by one frame of video signal or/and the previously generated extracting reference pixel position signal, or said corrected extracting reference pixel position signal with reference to the position on the panorama image that is determined by said corrected extracting reference pixel position signal.

11. The device of claim 7, wherein:

said surrounding image receiving portion receives a surrounding video signal from each of a plurality of camera-equipped portable terminals;

said variation detecting portion is provided with a previous frame memory for storing a surrounding video signal used for detecting the previous variation information for said each camera-equipped portable terminal having transmitted the surrounding video signal, for detecting variation information corresponding to said each camera-equipped portable terminal, based on the received surrounding video signal and the surrounding video signal stored in said previous frame memory corresponding to said camera-equipped portable terminal;

said capturing signal generating portion is provided with a previous signal memory for storing a previously generated capturing signal for said each camera-equipped portable terminal having transmitted the surrounding video signal, for generating a capturing signal corresponding to said each detected camera-equipped portable terminal from the detected variation information corresponding to said each camera-equipped portable terminal and said previously generated capturing signal having been stored in said previous signal memory in correspondence to said each camera-equipped portable terminal, and for updating the corresponding to signal in said previous signal memory with said generated capturing signal;

said image capturing portion captures a video signal, for each capturing signal stored in said previous signal memory in correspondence to each camera-equipped portable terminal; and

said image sending portion sends said captured video signal for said each camera-equipped portable terminal to an image display portion at a same position as that of said each camera-equipped portable terminal.

13. A processing method of an image capturing device, comprising:

a first step of deciding whether an operation command for changing the shooting direction of a camera or/and a lens field angle is received or not, the operation hereinafter being referred to as camera operation, the lens field angle being referred to as a zoom amount;

a second step of receiving a surrounding video signal of a surrounding object from a camera-equipped portable terminal when it is decided in said first step that said command is received;

a third step of detecting, from the received surrounding video signal, variation information indicating a variation of an image of the surrounding video signal from an image of the previous surrounding video signal;

a fourth step of generating a capturing signal for capturing a video signal of a part of a panoramic object based on a previous capturing signal and said variation information, the part of a panoramic object hereinafter being referred to as a partial object;

a fifth step of sending said generated capturing signal to a remote image sensing device which performs image sensing of said panoramic object;

a sixth step of receiving a video signal from the remote image sensing device;

a seventh step of sending said received video signal to an image display portion at a same position as that of said camera-equipped portable terminal; and

an eighth step of deciding whether a camera operation stop command is received;

wherein when the operation command is not received in said third step, processing proceeds to said sixth step, and when the stop command is not received in said eighth step, processing returns to said first step.

14. A processing method of an image capturing device, comprising:

a first step of deciding whether an operation command for changing the shooting direction of a camera or/and a lens field angle is received or not, the operation hereinafter being referred to as camera operation, the lens field angle being referred to as a zoom amount;

a second step of receiving a surrounding video signal of a surrounding object from a camera-equipped portable terminal when it is decided in said first step that said command is received;

a third step of detecting, from the received surrounding video signal, variation information indicating a variation of an image of the surrounding video signal from an image of the previous surrounding video signal;

a fourth step of generating a capturing signal for capturing a video signal of a part of a panoramic object based on a previous capturing signal and said variation information, the part of a panoramic object hereinafter being referred to as a partial object;

a fifth step of receiving a video signal from a remote image sensing device;

a sixth step of capturing a video signal of a part of a panoramic object from said received video signal based on said generate capturing signal;

a seventh step of sending said captured video signal to an image display means portion at a same position as that of said camera-equipped portable terminal; and

an eighth step of deciding whether a camera operation stop command is received;

wherein when the operation command is not received in said first step, processing proceeds to said fifth step, and when the stop command is not received in said eighth step, processing returns to said third step.

15. An image capturing device comprising:

a surrounding image receiving portion for receiving a surrounding video signal from each of a plurality of camera-equipped portable terminals;

a variation detecting portion, provided with a previous frame memory for storing a surrounding video signal, for detecting variation information corresponding to said each camera-equipped portable terminal having transmitted the surrounding video signal, based on the received surrounding video signal and the surrounding video signal stored in said previous frame memory corresponding to said each camera-equipped portable terminal having transmitted the surrounding video signal;

a capturing signal generating portion provided with a previous signal memory for storing a previously generated capturing signal for each piece of user terminal identification information identifying the camera-equipped portable terminal having sent the said variation information signal, for generating, from the detected variation information signal and the previously generated capturing signal of the user terminal identification information stored in the previously memory, a capturing signal to be used for capturing a video signal of a part of a panoramic object, the part of the panoramic object hereinafter being referred to as a partial object;

a signal sending portion for sending said generated capturing signal and the corresponding user terminal identification information to a remote image sensing device which performs image sensing of the panoramic object and outputs its video signal; and

an image relay portion for receiving the video signal and the corresponding user terminal identification information from said remote image sensing device and sending the video signal to an image display portion of the camera-equipped portable terminal corresponding to the user terminal identification information.

16. The device of claim 15, wherein:

said variation information signal represents direction information or/and zoom information of the partial object desired to switch from a previous partial object; and

said capturing signal generating portion generates, as said capturing signal, an extracting reference pixel position signal obtained by correcting a previous extracting reference pixel position signal according to said direction information or/and an image zoom in/out signal corresponding to said zoom information.

17. The device of claim 15, further comprising:

a camera direction storage portion which stores identification information of each of plural camera devices provided in said remote image sensing device and angularly spaced apart in their shooting direction;

wherein;

said variation information signal represents first direction information of the partial object desired to change from the previous partial object in a first direction, and/or second direction information of the partial object desired to change from the previous partial object in a second direction perpendicular to the first direction information signal and zoom information of the desired partial object; and

said capturing signal generating portion determines the shooting direction by referring to the identification information stored in said camera direction storage portion based on a previously generated transmitting camera identification information signal and said first direction information, for generating, as said capturing signal, a transmitting camera identification information signal of the camera identification information corresponding to said determined shooting direction, and/or a reference pixel position signal obtained by correcting said generated reference pixel position corresponding to said second direction information or/and an image zoom in/out signal corresponding to said zoom information.

21. A computer readable recording medium embedded with a program for functioning a computer as said image capturing device described in any one of claims 2, 7, 15 and 18.

18. An image capturing device comprising:

a surrounding image receiving portion for receiving a surrounding video signal from each of a plurality of user terminals;

a variation detecting portion, provided with a previous frame memory for storing a surrounding video signal, for detecting variation information corresponding to said each camera-equipped portable terminal having transmitted the surrounding video signal, based on the received surrounding video signal and the surrounding video signal stored in said previous frame memory corresponding to said each camera-equipped portable terminal having transmitted the surrounding video signal;

a capturing signal generating portion provided with a previous signal memory for storing previously generated capturing signals for each piece of user terminal identification information identifying the user terminal having sent the said variation information signal, for generating, from the received variation information signal and the previously generate capturing signal of the user terminal identification information stored in the previously memory, a capturing signal to be used for capturing a video signal of a part of a panoramic object, the part of the panoramic object hereinafter being referred to as a partial object;

a signal sending portion for sending said generated capturing signal to a remote image sensing device which performs image sensing of the panoramic object and outputs its video signal;

a remote image portion for receiving the video signal sent from the remote image sensing device having performed image sensing of a panoramic object;

an image capturing portion for capturing the video signal of a part of the panoramic object from said received video signal based on the capturing signal for each user terminal identification information stored in said previous signal memory; and

an image sending portion for sending said captured video signal to the user terminal corresponding to the user terminal identification information.

19. The device of claim 18, wherein:

the video signal received by said remote image receiving portion is a panorama video signal;

said variation information signal represents direction information or/and zoom information for the partial object desired to switch from the previous partial object;

said capturing signal generating portion generates, as said capturing signal, an extracting reference pixel position signal obtained by correcting a previous extracting reference pixel position signal according to said direction information or/and an image zoom in/out signal corresponding to said zoom information; and

said image capturing portion captures, from said panorama video signal, one frame of video signal produced by zooming in/out, by said image zoom in/out signal relative to the image of the previously sent captured video signal, a partial image of the panorama image of said panorama video signal that is defined by one frame of video signal or/and the previously generated extracting reference pixel position signal, or said corrected extracting reference pixel position signal with reference to the position on the panorama image that is determined by said corrected extracting reference pixel position signal.

20. The device of claim 18, wherein:

the video signal received by said remote image receiving portion is video signals from a plurality of camera devices of said remote image sensing device with their shooting directions angularly spaced apart; and

the device further comprises;

a camera direction storage portion having stored herein identification information about said plurality of camera devices and information corresponding to their shooting directions;

wherein;

said variation information signal represents first direction information of the partial object desired to change from the previous partial object or/and a second direction information of the desired partial object in a direction perpendicular to that of the first direction information or/and zoom information of the desired partial object;

said capturing signal generating portion determines the shooting direction by referring to the identification information stored in said camera direction storage means based on a previously generated transmitting camera identification information signal and said first direction information, for generating, as said capturing signal, a transmitting camera identification information signal of the camera identification information corresponding to said determined shooting direction, and/or a reference pixel position signal obtained by correcting said generated reference pixel position corresponding to said second direction information or/and an image zoom in/out signal corresponding to said zoom information; and

said image capturing portion captures that one of a plurality of video signals which corresponds to said generated camera identification signal, and for capturing, from said captured video signal, one frame of video signal produced by zooming in/out, by said image zoom in/out signal relative to the image of the previously sent captured video signal, a partial image of the image of said captured video signal that is defined by one frame of video signal or/and the previously generated extracting reference pixel position signal, or said corrected extracting reference pixel position signal with reference to the position on the panorama image that is determined by said corrected extracting reference pixel position signal.