Remote video display method, video acquisition device, method thereof, and program thereof

• US 20050212909A1
• Filed: 01/15/2004
• Published: 09/29/2005
• Est. Priority Date: 01/17/2003
• Status: Active Grant

• Current Assignee: Nippon Telegraph and Telephone Corporation
• Original Assignee: Nippon Telegraph and Telephone Corporation
• Inventors: Kishida, Katsumi, Takehara, Nobuhiko, Anzai, Hiroki
• Granted Patent: US 7,385,624 B2
• Time in Patent Office: Days
• US Class Current: 348/36
• CPC Class Codes: H04N 21/21805 enabling multiple viewpoints, e.g. using a plurality of cameras
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/6587 Control parameters, e.g. trick play commands, viewpoint selection
• CPC Class Codes: H04N 23/63 by using electronic viewfinders
• CPC Class Codes: H04N 23/661 Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources
• CPC Class Codes: H04N 7/185 from a mobile camera, e.g. for remote control

Abstract

An omnidirectional camera performs image sensing of a panoramic object 6 surrounding it, and sends the video signal to an image capturing device 1, then the image capturing device 1 extracts a video signal of a partial object 61 and sends the extracted video signal to a camera-equipped portable terminal 3 carried in user'"'"'s hand at a place distant from the omnidirectional camera 2 to display the image of the video signal on a display surface 32a of the portable terminal, and the user shoots a surrounding object 8 with the portable terminal 3 and sends the surrounding video signal to the device 1. When the shooting direction of the portable terminal 3 is turned as indicated by the broken line, the device 1 detects a change in the shooting direction of he portable terminal 3 from a difference between the surrounding video signals generated before and after the turning of the portable terminal, and extracts, from a panorama video signal, a video signal of a partial object 62 turned from the partial object 61 in correspondence to the difference and sends the extracted video signal to the portable terminal 3 to display the image of the video signal.

Claims

1. A remote image display method, characterized by the steps of:

image sensing a panoramic object by a remote image sensing device;

sending a video signal of a part of said image-sensed panoramic object via an image capturing device to image display means at a place different from said remote image sensing device;

playing back and displaying said received video signal by said image display means as an image of a part of said panoramic object;

image sensing its surroundings by a camera-equipped portable terminal at a position where the display on said image display means can be seen, and sending the surrounding video signal to said image capturing device;

obtaining variation information about the surrounding image from said camera-equipped portable terminal by said image capturing device based on a previous received surrounding video signal and the current received surrounding video signal;

obtaining from said remote image sensing device a video signal of a part of said panoramic object changed corresponding to a change of said surrounding image based on said variation information; and

sending said obtained image signal to said image display means.

2. An image capturing device comprising:

surrounding image receiving means for receiving a surrounding video signal from a camera-equipped portable terminal;

variation detecting means for detecting, from said received surrounding video signal, information about a variation of the image of the current surrounding video signal (hereinafter referred to as the current image) from the image of a previous surrounding video signal (hereinafter referred to as the previous image) received by said surrounding image receiving means;

capturing signal generating means for generating a capturing signal from said variation information which is used to obtain the video signal of a part of a panoramic object;

signal sending means for said capturing signal to a remote image sensing device that performs image sensing of a panoramic object and outputs a video signal; and

image relay means for receiving said video signal from said remote image sensing device and for sending the received video signal to image display means at the same position as said camera-equipped portable terminal.

3. The device of claim 2, wherein:

said variation detecting means is a means for detecting, as said variation information, direction information corresponding to the direction of movement of said current image with respect to said previous image or/and zoom information corresponding to a change in the size of the part of said current image corresponding to a part of said previous image; and

said capturing signal generating means is a means for generating, as said capturing signal, an extracting reference pixel position signal obtained by correcting a previous extracting reference pixel position signal according to said direction information or/and an image zoom in/out signal corresponding to said zoom information.

4. The device of claim 2, which further comprises:

camera direction storage means in which there is stored identification information of each of plural camera devices provided in said remote image sensing device and angularly spaced apart in their shooting direction; and

wherein;

said variation detecting means is a means for detecting, as said variation information, direction information corresponding to the direction of movement of said current image with respect to said previous image; and

said capturing signal generating means is a means which determines the shooting direction by referring to said camera direction storage means based on a previously generated transmitting camera identification information signal and said direction information, for generating, as said capturing signal, a transmitting camera identification information signal of the camera identification information corresponding to said determined shooting direction.

5. The device of claim 2, wherein:

said variation detecting means is a means for detecting, as said variation information, direction information corresponding to the direction of movement of said current image with respect to said previous image or/and zoom information corresponding to a change in the size of said current image with respect to said previous image; and

said capturing signal generating means is a means for generating, as said capturing signal, a shooting direction change signal corresponding to said direction information or/and zoom change signal corresponding to said zoom information.

6. The device of claim 2, wherein:

said surrounding image receiving means is a means for receiving the surrounding video signal from each of a plurality of camera-equipped portable terminals; and

said variation detecting means is a means provided with a previous frame memory for storing a surrounding video signal used for detecting the previous variation information for each camera-equipped portable terminal having transmitted its received surrounding video signal, for detecting variation information corresponding to said camera-equipped portable terminal having transmitted the received surrounding video signal, based on the received surrounding video signal and the surrounding video signal stored in said previous frame memory corresponding to said camera-equipped portable terminal having transmitted the surrounding video signal;

said capturing signal generating means is a means provided with a previous signal memory for storing a previously generated capturing signal for each camera-equipped portable terminal having transmitted its received surrounding video signal, for generating a capturing signal corresponding to said detected camera-equipped portable terminal from the variation information corresponding to said detected camera-equipped portable terminal and said previously generated capturing signal stored in said previous signal memory corresponding to said detected camera-equipped portable terminal, and for updating the corresponding to signal in said previous signal memory with said generated capturing signal;

said signal sending means is a means for sending said generated capturing signal and information for identifying the camera-equipped portable terminal corresponding thereto; and

said image relay means is a means for sending its received video signal to said image display means of the camera-equipped portable terminal indicated by camera-equipped portable terminal identification information received together with said video signal.

12. The device of claim 2 or 7, characterized by the provision of:

variation hysteresis storage means for storing the hysteresis of said variation information;

decision means for deciding whether the variation information detected by said variation detecting means is abnormal or not by referring to hysteresis information stored in said variation hysteresis storage means; and

means which, when said decision means decides that said variation information is abnormal, inhibits the supply of said detected variation information to said capturing signal generating means.

7. An image capturing device comprising:

surrounding image receiving means for receiving a surrounding vide signal from a camera-equipped portable terminal;

variation detecting means for detecting, from said received surrounding video signal, information about a variation of the image of the current surrounding video signal (hereinafter referred to as the current image) from the image of a previous surrounding video signal (hereinafter referred to as the previous image) received by said surrounding image receiving means;

capturing signal generating means for generating, from said variation information and a previously generated capturing signal, a capturing signal that is used to obtain a video signal of a part of a panoramic object;

remote image receiving means for receiving a video signal sent from a remote image sensing device having performed image sensing of a panoramic object;

image capturing means for capturing the video signal of a part of the panoramic object from said received video signal based on said capturing signal; and

image sending means for sending said captured video signal to image display means at the same position as that of said camera-equipped portable terminal.

8. The device of claim 7, wherein:

the video signal received by said remote image receiving means is a panorama video signal;

said variation detecting means is a means for detecting, as said variation information, direction information corresponding to the direction of movement of said current image with respect to said previous image or/and zoom information corresponding to a change in the size of the part of said current image corresponding to a part of said previous image;

said capturing signal generating means is a means for generating, as said capturing signal, an extracting reference pixel position signal obtained by correcting a previous extracting reference pixel position signal according to said direction information or/and an image zoom in/out signal corresponding to said zoom information; and

said image capturing means is a means for capturing, from said panorama video signal, one frame of video signal produced by zooming in/out, by said image zoom in/out signal relative to the image of the previously sent captured video signal, a partial image of the panorama image of said panorama video signal that is defined by one frame of video signal or/and the previously generated extracting reference pixel position signal, or said corrected extracting reference pixel position signal with reference to the position on the panorama image that is determined by said corrected extracting reference pixel position signal.

9. The device of claim 7, wherein:

the video signal received by said remote image receiving means is video signals from a plurality of camera devices of said remote image sensing device with their shooting directions angularly spaced apart; and

which further comprises;

camera direction storage means having stored herein identification information about said plurality of camera devices and information corresponding to their shooting directions; and

wherein;

said variation detecting means is a means for detecting, as said variation information, direction information corresponding to the direction of movement of said current image with respect to said previous image;

said capturing signal generating means is a means which determines the shooting direction by referring to said camera direction storage means based on a previously generated transmitting camera identification information signal and said direction information, for generating, as said capturing signal, a transmitting camera identification information signal of the camera identification information corresponding to said determined shooting direction; and

said image capturing means is a means for capturing that one of a plurality of video signals which corresponding to said generated camera identification signal.

10. The device of claim 9, wherein:

said variation detecting means includes a means for detecting, as said variation information, second direction information corresponding to the direction of movement of said current image with respect to said previous image in a direction perpendicular to the direction of movement of said detected direction information or/and zoom information corresponding to a change in the size of that part of said current image corresponding to a part of said previous image;

said capturing signal generating means is a means for generating, as said capturing signal, an extracting reference pixel position signal obtained by correcting a previous extracting reference pixel position signal according to said second direction information or/and an image zoom in/out signal corresponding to said zoom information; and

said image capturing means is a means for capturing, from said captured video signal, one frame of video signal produced by zooming in/out, by said image zoom in/out signal relative to the image of the previously sent captured video signal, a partial image of the image of said captured video signal that is defined by one frame of video signal or/and the previously generated extracting reference pixel position signal, or said corrected extracting reference pixel position signal with reference to the position on the panorama image that is determined by said corrected extracting reference pixel position signal.

11. The device of claim 7, wherein:

said surrounding image receiving means is a means for receiving a surrounding video signal from each of a plurality of camera-equipped portable terminals;

said variation detecting means is a means provided with a previous frame memory for storing a surrounding video signal used for detecting the previous variation information for said each camera-equipped portable terminal having transmitted its received surrounding video signal, for detecting variation information corresponding to said each camera-equipped portable terminal, based on the received surrounding video signal and the surrounding video signal stored in said previous frame memory corresponding to said camera-equipped portable terminal;

said capturing signal generating means is a means provided with a previous signal memory for storing a previously generated capturing signal for each camera-equipped portable terminal having transmitted its received surrounding video signal, for generating a capturing signal corresponding to said detected camera-equipped portable terminal from the variation information corresponding to said detected camera-equipped portable terminal and said previously generated capturing signal stored in said previous signal memory corresponding to said detected camera-equipped portable terminal, and for updating the corresponding to signal in said previous signal memory with said generated capturing signal;

said image capturing mean is a means for capturing a video signal for each capturing signal corresponding to each camera-equipped portable terminal stored in said previous signal memory; and

said image sending means is a means for sending said captured video signal for said each camera-equipped portable terminal to image display means at the same position as that of said each camera-equipped portable terminal.

22. A program for functioning a computer as said image capturing device described in any one of claims 2 to 12 and 16 to 21.

13. A processing method of an image capturing device, characterized by:

a first step of deciding whether an operation (hereinafter referred to as camera operation) command for changing the shooting direction of a camera or/and the lens field angle (zoom amount) is received or not;

a second step of receiving a surrounding video signal of a surrounding object from a camera-equipped portable terminal when it is decided in said first step that said command is received;

a third step of detecting, from the received surrounding video signal, variation information indicating a variation of the image of the surrounding video signal from the image of the previous surrounding video signal;

a fourth step of generating a capturing signal for capturing a video signal of a part of a panoramic object based on the previous capturing signal and said variation information (which part will hereinafter be referred to as a partial object);

a fifth step of sending said generated capturing signal to a remote image sensing device which performs image sensing of said panoramic object;

a sixth step of receiving a video signal from the remote image sensing device;

a seventh step of sending said received video signal to image display means at the same position as that of said camera-equipped portable terminal; and

an eighth step of deciding whether a camera operation stop command is received;

wherein if the operation command is not received in said first step, processing proceeds to said sixth step, and if the stop command is not received in said eighth step, processing returns to said first step.

14. A processing method of an image capturing device, characterized by:

a first step of deciding whether an operation (hereinafter referred to as camera operation) command for changing the shooting direction of a camera or/and the lens field angle (zoom amount) is received or not;

a second step of receiving a surrounding video signal of a surrounding object from a camera-equipped portable terminal when it is decided in said first step that said command is received;

a third step of detecting, from the received surrounding video signal, variation information indicating a variation of the image of the surrounding video signal from the image of the previous surrounding video signal;

a fourth step of generating a capturing signal for capturing a video signal of a part of a panoramic object based on the previous capturing signal and said variation information (which part will hereinafter be referred to as a partial object);

a fifth step of receiving a video signal from a remote image sensing device;

a sixth step of capturing a video signal of a part of a panoramic object from said received video signal based on said generate capturing signal;

a seventh step of sending said captured video signal to image display means at the same position as that of said camera-equipped portable terminal; and

an eighth step of deciding whether a camera operation stop command is received;

wherein if the operation command is not received in said first step, processing proceeds to said fifth step, and if the stop command is not received in said eighth step, processing returns to said first step.

15. (cancel)

16. An image capturing device comprising:

signal receiving means for receiving variation information signals from a plurality of user terminals;

capturing signal generating means provided with a previous signal memory for storing previously generated capturing signals for each piece of information (hereinafter referred to as user terminal identification information) identifying the user terminal having sent the said variation information signal, for generating, from the received variation information signal and the previously generate capturing signal of the user terminal identification information stored in the previously memory, a capturing signal to be used for capturing a video signal of a part of a panoramic object (which part will hereinafter be referred to as a partial object);

signal sending means for sending said generated capturing signal and the corresponding user terminal identification information to a remote image sensing device which performs image sensing of the panoramic object and outputs its video signal; and

image relay means for receiving the video signal and the corresponding user terminal identification information from said remote image sensing device and sending the video signal to image display means of the user terminal corresponding to the user terminal identification information.

17. The device of claim 16, wherein:

said variation information signal is a direction information signal or/and a zoom signal of the partial object desired to switch from the previous partial object; and

said capturing signal generating means is a means for generating, as said capturing signal, an extracting reference pixel position signal obtained by correcting a previous extracting reference pixel position signal according to said direction information or/and an image zoom in/out signal corresponding to said zoom information.

18. The device of claim 16, which further comprises:

camera direction storage means in which there is stored identification information of each of plural camera devices provided in said remote image sensing device and angularly spaced apart in their shooting direction; and

wherein;

said variation information signal is a first direction information signal of the partial object desired to change from the previous partial object and/or a second direction information signal of the first direction information signal or/and a zoom signal of the desired partial object; and

said capturing signal generating means is a means which determines the shooting direction by referring to said camera direction storage means based on a previously generated transmitting camera identification information signal and said direction information, for generating, as said capturing signal, a transmitting camera identification information signal of the camera identification information corresponding to said determined shooting direction, and/or a reference pixel position signal obtained by correcting said generated reference pixel position corresponding to said second direction information signal or/and an image zoom in/out signal corresponding to said zoom information signal.

19. An image capturing device comprising:

signal receiving means for receiving variation information signals from a plurality of user terminals;

capturing signal generating means provided with a previous signal memory for storing previously generated capturing signals for each piece of information (hereinafter referred to as user terminal identification information) identifying the user terminal having sent the said variation information signal, for generating, from the received variation information signal and the previously generate capturing signal of the user terminal identification information stored in the previously memory, a capturing signal to be used for capturing a video signal of a part of a panoramic object (which part will hereinafter be referred to as a partial object);

signal sending means for sending said generated capturing signal and the corresponding user terminal identification information to a remote image sensing device which performs image sensing of the panoramic object and outputs its video signal;

remote image means for receiving the video signal sent from the remote image sensing device having performed image sensing of a panoramic object;

image capturing means for capturing the video signal of a part of the panoramic object from said received video signal based on the capturing signal for each user terminal identification information stored in said previous signal memory; and

image sending means for sending said captured video signal to the user terminal of th user terminal identification information.

20. The device of claim 19, wherein:

the video signal received by said remote image receiving means is a panorama video signal;

said variation information signal is a direction signal or/and a zoom information signal for the partial object desired to switch from the previous partial object;

said capturing signal generating means is a means for generating, as said capturing signal, an extracting reference pixel position signal obtained by correcting a previous extracting reference pixel position signal according to said direction information or/and an image zoom in/out signal corresponding to said zoom information; and

said image capturing means is a means for capturing, from said panorama video signal, one frame of video signal produced by zooming in/out, by said image zoom in/out signal relative to the image of the previously sent captured video signal, a partial image of the panorama image of said panorama video signal that is defined by one frame of video signal or/and the previously generated extracting reference pixel position signal, or said corrected extracting reference pixel position signal with reference to the position on the panorama image that is determined by said corrected extracting reference pixel position signal.

21. The device of clam in 19, wherein:

the video signal received by said remote image receiving means is video signals from a plurality of camera devices of said remote image sensing device with their shooting directions angularly spaced apart; and

which further comprises;

camera direction storage means having stored herein identification information about said plurality of camera devices and information corresponding to their shooting directions;

wherein;

said variation information signal is a first direction information signal of the partial object desired to change from the previous partial object or/and a second direction information signal of the desired partial object in a direction perpendicular to that of the first direction information signal or/and a zoom signal of the desired partial object;

said capturing signal generating means is a means which determines the shooting direction by referring to said camera direction storage means based on a previously generated transmitting camera identification information signal and said direction information, for generating, as said capturing signal, a transmitting camera identification information signal of the camera identification information corresponding to said determined shooting direction, and/or a reference pixel position signal obtained by correcting said generated reference pixel position corresponding to said second direction information signal or/and an image zoom in/out signal corresponding to said zoom information signal; and

said image capturing means is a means for capturing that one of a plurality of video signals which corresponds to said generated camera identification signal, and for capturing, from said captured video signal, one frame of video signal produced by zooming in/out, by said image zoom in/out signal relative to the image of the previously sent captured video signal, a partial image of the image of said captured video signal that is defined by one frame of video signal or/and the previously generated extracting reference pixel position signal, or said corrected extracting reference pixel position signal with reference to the position on the panorama image that is determined by said corrected extracting reference pixel position signal.