REMOTE OPERATION SYSTEM

• US 20150261218A1
• Filed: 03/15/2013
• Published: 09/17/2015
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Hitachi, Ltd.
• Original Assignee: Hitachi, Ltd.
• Inventors: Nakamura, Ryosuke, Amino, Azusa, Ueda, Taishi
• Granted Patent: US 9,317,035 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: B25J 9/1689 Teleoperation
• CPC Class Codes: G05D 1/0038 by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246
• CPC Class Codes: G05D 1/0214 in accordance with safety or protection criteria, e.g. avoiding hazardous areas monitoring the location of vehicles within a certain area, e.g. forbidden or allowed areas, in traffic control systems for road vehicles G08G1/13
• CPC Class Codes: G05D 1/024 in combination with a laser lasers per se H01S
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/0248 in combination with a laser lasers per se H01S
• CPC Class Codes: G06T 1/0014 Image feed-back for automatic industrial control, e.g. robot with camera robots B25J19/023
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/80 Recognising image objects characterised by unique random patterns

Abstract

When a moving body is remotely operated, a delay in transmission of an image and information acquired from the moving body becomes a problem since immediate transmission of an operation to the moving body cannot be expected. When a frame rate is decreased or a resolution is reduced to decrease the amount of image data to prevent transmission delay of the image data, it is difficult to grasp motions of other moving bodies. As a result, when a certain amount of time is required to transmit a camera image and a moving obstacle, the movement of the obstacle and the movement of the moving body are estimated. The estimated image is generated based on a current image, and is presented to an operator via a monitor. Accordingly, it is possible to reduce instability and a risk associated with transmission delay to safely operate the moving body.

Claims

1. A remote operation system comprising:

a moving body; and

a remote operation apparatus,

wherein the moving body includes;

a camera configured to capture an image of the front of the moving body;

an environment sensor configured to measure the distance from the moving body to a surrounding obstacle;

a movement speed detection unit configured to measure the movement speed and the turning speed of the moving body; and

a wireless device,

wherein the remote operation apparatus includes;

a monitor configured to display an image from the camera;

a calculating machine; and

a wireless device,

wherein information is transmitted and received between the wireless device of the moving body and the wireless device of the remote operation apparatus, and

wherein the calculating machine includes;

a database in which image information regarding an obstacle and a moving body model are recorded;

an obstacle motion estimation unit configured to recognize an obstacle captured in the image from the camera and a moving body model for the obstacle, based on information from the environment sensor, the image from the camera, and information in the database, and to estimate the position of the obstacle after a set amount of time elapses;

an estimated image generation unit configured to generate a background image after the set amount of time elapses, based on the movement speed and the turning speed of the moving body, the background image being obtained by removing an image of the obstacle from the image from the camera; and

an image output unit configured to synthesize an image after the predetermined set amount of time elapses, based on outputs from the estimated image generation unit and the obstacle motion estimation unit.

2. The remote operation system according to claim 1,

wherein the obstacle motion estimation unit outputs a plurality of candidate positions and the intensity of estimation of a single obstacle after the predetermined set amount of time elapses, and

wherein the image output unit weights transparency and clearness for the plurality of candidate positions of the single obstacle in response to the intensity of estimation, and synthesizes an image.

3. The remote operation system according to claim 1,

wherein the obstacle motion estimation unit records estimated results for obstacles in time series, and adds weighting factors to the estimated results in such a manner that the recorded estimated results are easily selected in subsequent estimation.

4. The remote operation system according to claim 1,

wherein a stochastic process model is adopted as the moving body model,

wherein the obstacle motion estimation unit expresses an estimated result of movement of the obstacle using a position and a covariance matrix, and

wherein the image output unit gradates, and overlaps the images of the obstacles on top of each other according to the covariance matrix when synthesizing an output image.

5. The remote operation system according to claim 1,

wherein the moving body further includes a wheel drive device and wheels, and

wherein the movement speed detection unit obtains the rotational angular speed of each of the wheels via the wheel drive device, and measures the movement speed and the turning speed of the moving body.

6. The remote operation system according to claim 1,

wherein the remote operation apparatus further includes an operation input device,

wherein the calculating machine further includes an operation analysis unit, and

wherein the operation input device generates a command for movement of the moving body which is sent to the operation analysis unit.

7. A remote operation system comprising:

a moving body; and

a remote operation apparatus,

wherein the moving body includes;

a camera configured to capture an image of the front of the moving body;

an environment sensor configured to measure the distance from the moving body to a surrounding obstacle;

a control device;

a wireless device;

wheels; and

a wheel drive device configured to drive the wheels,

wherein the remote operation apparatus includes;

a monitor configured to display an image from the camera in the moving body;

a calculating machine; and

a wireless device,

wherein information is transmitted and received between the wireless device of the moving body and the wireless device of the remote operation apparatus,

wherein the control device has a movement speed detection unit configured to measure the movement speed and the turning angular speed of the moving body based on information from the wheel drive device, and

wherein the calculating machine includes;

a database in which image information regarding an obstacle and a moving body model are recorded;

an obstacle motion estimation unit configured to recognize an obstacle captured in the image from the camera and a moving body model for the obstacle, based on information from the environment sensor, the image from the camera, and information in the database, and to estimate the position of the obstacle after a set amount of time elapses;

an estimated image generation unit configured to generate a background image when the set amount of time elapses, based on the movement speed and the turning angular speed of the moving body, the background image being obtained by removing an image of the obstacle from the image from the camera; and

an image output unit configured to synthesize an image after the predetermined set amount of time elapses, based on outputs from the estimated image generation unit and the obstacle motion estimation unit.

8. The remote operation system according to claim 7,

wherein the obstacle motion estimation unit outputs a plurality of candidate positions and the intensity of estimation of a single obstacle after the predetermined set amount of time elapses, and

wherein the image output unit weights transparency and clearness for the plurality of candidate positions of the single obstacle in response to the intensity of estimation, and synthesizes an image.

9. The remote operation system according to claim 7,

wherein the obstacle motion estimation unit records estimated results for obstacles in time series, and adds weighting factors to the estimated results in such a manner that the recorded estimated results are easily selected in subsequent estimation.

10. The remote operation system according to claim 7,

wherein a stochastic process model is adopted as the moving body model,

wherein the obstacle motion estimation unit expresses an estimated result of movement of the obstacle using a position and a covariance matrix, and

wherein the image output unit gradates, and overlaps the images of the obstacles on top of each other according to the covariance matrix when synthesizing an output image.

11. The remote operation system according to claim 7,

wherein the remote operation apparatus further includes an operation input device,

wherein the calculating machine further includes an operation analysis unit, and

wherein the operation input device generates and sends a command for movement of the moving body to the operation analysis unit.