Mobile body system, control apparatus and method for controlling a mobile body

• US 10,237,518 B2
• Filed: 02/29/2016
• Issued: 03/19/2019
• Est. Priority Date: 06/12/2015
• Status: Active Grant

• Current Assignee: Sharp Kabushiki Kaisha (Hon Hai Precision Industry Co., Ltd.)
• Original Assignee: Sharp Kabushiki Kaisha (Hon Hai Precision Industry Co., Ltd.)
• Inventors: Kuroda, Tatsuro
• Primary Examiner(s): Elahi, Shan E
• Application Number:
  US15/055,675
• Publication Number:
  US 20160366372A1
• Time in Patent Office: 1,114 Days
• Field of Search: 348116
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: H04N 5/93 Regeneration of the television signal or of selected parts thereof
• CPC Class Codes: H04N 7/185 from a mobile camera, e.g. for remote control
• CPC Class Codes: H04N 9/8205 involving the multiplexing of an additional signal and the colour video signal

Abstract

A mobile body photographs an image with a camera while moving, and transmits to a control apparatus image data representing the photographed image and state information indicating the state thereof including the location thereof. The control apparatus displays a map including a moving range of the mobile body on a display unit and overlays a route of the mobile body on the map to display the route. The control apparatus detects an occurrence of an event such as finding of an intruder and the location of the mobile body at the time of the occurrence of the event, based on the received image data and state information, and displays a marker indicating the detected location over the map. Whenever an event occurs, an additional marker is displayed over the map and the history of the past events is recorded on the map.

Claims

1. A mobile body system, comprising:

an autonomous vehicle travelling autonomously;

a plurality of control apparatuses located outside the autonomous vehicle; and

a storage apparatus,

wherein the autonomous vehicle includes;

a camera for capturing images of the surrounding of the autonomous vehicle;

a sensor for detect condition of the autonomous vehicle when moving;

a control unit for generating state information indicating the condition detected the sensor; and

a transmitting part for transmitting image data representing the images captured by the camera and the state information to the control apparatus through a communication network,

the control apparatus includes;

a communication unit for receiving the image data and the state information through the communication network;

a display unit for displaying a map including a moving range of the autonomous vehicle;

a processor; and

a storage unit for storage the image data and instructions to be executed by the processor,

the processor executes instructions to determine whether an event occurs to the autonomous vehicle by determining whether specific change in the image data or the state information occurs,

the sensor detect a location of the autonomous vehicle,

the state information includes information indicating the location detected by the sensor,

the processor executes instructions to obtain a location of the autonomous vehicle at the time of the occurrence of the event from the state information,

the processor executes instructions to generate an image including the map and a marker indicating the location on the map,

the processor executes instructions to send the image including the map and the marker to the display unit and to display the image on the display unit,

the processor executes instructions to display a different marker according to details of the event on the display unit,

the storage apparatus stores event information including details of the event and the location, the event information is related to the plurality of control apparatuses,

each of the plurality of control apparatuses controls the autonomous vehicle with a different moving range,

the storage apparatus includes;

a statistical processing part for executing statistical processing for the entire event information related to all of the plurality of control apparatuses or for the event information related to each of the control apparatuses,

the communication unit receives from the storage apparatus a statistical result of the statistical processing executed by the statistical processing part, and

the processor executes instructions to display the statistical result on the display unit.

2. A method for controlling an autonomous vehicle including a camera and a sensor, comprising:

displaying on a display unit a map including a moving range of the autonomous vehicle;

determining whether an event occurs to the autonomous vehicle by determining whether specific change in image data representing images captured by the camera or in state information indicating condition of the autonomous vehicle when moving detected by the sensor;

obtaining a location of the autonomous vehicle at the time of the occurrence of the event from the state information;

displaying on the display unit the map and a marker indicating the location on the map;

displaying a different marker according to details of the event on the display unit;

storing event information including details of the event and the location;

executing statistical processing for the entire event information related to all of a plurality of autonomous vehicles or for the event information related to each of the autonomous vehicles; and

displaying statistical result of the statistical processing on the display unit.

3. The method according to claim 2, further comprising:

dividing a screen on the display unit;

displaying the map on one part of the divided screen on the display unit; and

displaying an image represented by the image data on another part of the divided screen.

4. The method according to claim 2, further comprising:

detecting as an event that the autonomous vehicle fails to pass a part of a predetermined route; and

generating a route for detouring around a specific part of the predetermined route, when the number of occurrences of events that the autonomous vehicle fails to pass the specific part is a predetermined number or more.

5. A non-transitory computer-readable medium having stored therein a program for causing a computer to execute a process of controlling an autonomous vehicle including a camera and a sensor, the process comprising:

a step of displaying a map including a moving range of the autonomous vehicle on a display unit of the computer;

a step of determining whether an event occurs to the autonomous vehicle by determining whether specific change in image data representing images captured by the camera or in state information indicating condition of the autonomous vehicle when moving detected by the sensor;

a step of obtaining a location of the autonomous vehicle at the time of the occurrence of the event from the state information;

a step of displaying on the display unit the map and a marker indicating the location on the map;

a step of displaying a different marker according to details of the event on the display unit;

a step of storing event information including details of the event and the location in a storage unit of the computer;

a step of obtaining statistical result of statistical processing for the entire event information related to all of a plurality of autonomous vehicles or for the event information related to each of the autonomous vehicles; and

a step of displaying the statistical result on the display unit.

6. A mobile body system, comprising:

an autonomous vehicle travelling autonomously; and

a control apparatus located outside the autonomous vehicle,

wherein the autonomous vehicle includes;

a camera for capturing images of the surrounding of the autonomous vehicle;

a sensor for detect condition of the autonomous vehicle when moving;

a control unit for generating state information indicating the condition detected the sensor; and

a transmitting part for transmitting image data representing the images captured by the camera and the state information to the control apparatus through a communication network,

the control apparatus includes;

a communication unit for receiving the image data and the state information through the communication network;

a display unit for displaying a map including a moving range of the autonomous vehicle;

a processor; and

a storage unit for storage the image data and instructions to be executed by the processor,

the processor executes instructions to determine whether an event occurs to the autonomous vehicle by determining whether specific change in the image data or the state information occurs,

the sensor detect a location of the autonomous vehicle,

the state information includes information indicating the location detected by the sensor,

the processor executes instructions to obtain a location of the autonomous vehicle at the time of the occurrence of the event from the state information,

the processor executes instructions to generate an image including the map and a marker indicating the location on the map,

the processor executes instructions to send the image including the map and the marker to the display unit and to display the image on the display unit,

the processor executes instructions to display a different marker according to details of the event on the display unit,

the processor executes instructions to determine a route of the autonomous vehicle,

the processor executes instructions to generate an image including the map and the route overlaid on the map, and

the processor executes instructions to display the image on the display unit.

7. A mobile body system, comprising:

an autonomous vehicle travelling autonomously; and

a control apparatus located outside the autonomous vehicle,

wherein the autonomous vehicle includes;

a camera for capturing images of the surrounding of the autonomous vehicle;

a sensor for detect condition of the autonomous vehicle when moving;

a control unit for generating state information indicating the condition detected the sensor; and

a transmitting part for transmitting image data representing the images captured by the camera and the state information to the control apparatus through a communication network,

the control apparatus includes;

a communication unit for receiving the image data and the state information through the communication network;

a display unit for displaying a map including a moving range of the autonomous vehicle;

a processor; and

a storage unit for storage the image data and instructions to be executed by the processor,

the processor executes instructions to determine whether an event occurs to the autonomous vehicle by determining whether specific change in the image data or the state information occurs,

the sensor detect a location of the autonomous vehicle,

the state information includes information indicating the location detected by the sensor,

the processor executes instructions to obtain a location of the autonomous vehicle at the time of the occurrence of the event from the state information,

the processor executes instructions to generate an image including the map and a marker indicating the location on the map,

the processor executes instructions to send the image including the map and the marker to the display unit and to display the image on the display unit,

the processor executes instructions to display a different marker according to details of the event on the display unit, and

the storage unit stores event information including details of the event and the location.

8. The mobile body system according to claim 7,

wherein the event information includes a time of the occurrence of the event.

9. The mobile body system according to claim 7,

wherein the processor executes instructions to execute statistical processing for the event information, and

the processor executes instructions to display statistical result of the statistical processing on the display unit.

10. A mobile body system, comprising:

an autonomous vehicle travelling autonomously; and

a control apparatus located outside the autonomous vehicle,

wherein the autonomous vehicle includes;

a camera for capturing images of the surrounding of the autonomous vehicle;

a sensor for detect condition of the autonomous vehicle when moving;

a control unit for generating state information indicating the condition detected the sensor; and

a transmitting part for transmitting image data representing the images captured by the camera and the state information to the control apparatus through a communication network,

the control apparatus includes;

a communication unit for receiving the image data and the state information through the communication network;

a display unit for displaying a map including a moving range of the autonomous vehicle;

a processor; and

a storage unit for storage the image data and instructions to be executed by the processor,

the processor executes instructions to determine whether an event occurs to the autonomous vehicle by determining whether specific change in the image data or the state information occurs,

the sensor detect a location of the autonomous vehicle,

the state information includes information indicating the location detected by the sensor,

the processor executes instructions to obtain a location of the autonomous vehicle at the time of the occurrence of the event from the state information,

the processor executes instructions to generate an image including the map and a marker indicating the location on the map,

the processor executes instructions to send the image including the map and the marker to the display unit and to display the image on the display unit,

the processor executes instructions to display a different marker according to details of the event on the display unit,

the processor executes instructions to generate an image including the map and an captured image represented by the image data overlaid on the map, and

the processor executes instructions to display the image on the display unit.

11. The mobile body system according to claim 10,

wherein the processor executes instructions to generate an image that is divided to plurality of parts and that includes the map on one part and the captured image on another part.

12. The mobile body system according to claim 10,

wherein the processor executes instructions to display the image including the map and the captured image on the display unit, when the occurrence of the event is determined.

13. A mobile body system, comprising:

an autonomous vehicle travelling autonomously; and

a control apparatus located outside the autonomous vehicle,

wherein the autonomous vehicle includes;

a camera for capturing images of the surrounding of the autonomous vehicle;

a sensor for detect condition of the autonomous vehicle when moving;

a control unit for generating state information indicating the condition detected the sensor; and

a transmitting part for transmitting image data representing the images captured by the camera and the state information to the control apparatus through a communication network,

the control apparatus includes;

a communication unit for receiving the image data and the state information through the communication network;

a display unit for displaying a map including a moving range of the autonomous vehicle;

a processor; and

a storage unit for storage the image data and instructions to be executed by the processor,

the processor executes instructions to determine whether an event occurs to the autonomous vehicle by determining whether specific change in the image data or the state information occurs,

the sensor detect a location of the autonomous vehicle,

the state information includes information indicating the location detected by the sensor,

the processor executes instructions to obtain a location of the autonomous vehicle at the time of the occurrence of the event from the state information,

the processor executes instructions to generate an image including the map and a marker indicating the location on the map,

the processor executes instructions to send the image including the map and the marker to the display unit and to display the image on the display unit,

the processor executes instructions to display a different marker according to details of the event on the display unit,

the processor executes instructions to generate a route of the autonomous vehicle for detouring around a part of a predetermined route,

the processor executes instructions to determine occurrence of an event that the autonomous vehicle fails to pass a part of the predetermined route, and

the processor executes instructions to generates the route for detouring around a specific part of the predetermined route, when the number of occurrences of events that the autonomous vehicle fails to pass the specific part is a predetermined number or more.

14. A mobile body system, comprising:

an autonomous vehicle travelling autonomously; and

a control apparatus located outside the autonomous vehicle,

wherein the autonomous vehicle includes;

a camera for capturing images of the surrounding of the autonomous vehicle;

a sensor for detect condition of the autonomous vehicle when moving;

a control unit for generating state information indicating the condition detected the sensor; and

a transmitting part for transmitting image data representing the images captured by the camera and the state information to the control apparatus through a communication network,

the control apparatus includes;

a communication unit for receiving the image data and the state information through the communication network;

a display unit for displaying a map including a moving range of the autonomous vehicle;

a processor; and

a storage unit for storage the image data and instructions to be executed by the processor,

the processor executes instructions to determine whether an event occurs to the autonomous vehicle by determining whether specific change in the image data or the state information occurs,

the sensor detect a location of the autonomous vehicle,

the state information includes information indicating the location detected by the sensor,

the processor executes instructions to obtain a location of the autonomous vehicle at the time of the occurrence of the event from the state information,

the processor executes instructions to generate an image including the map and a marker indicating the location on the map,

the processor executes instructions to send the image including the map and the marker to the display unit and to display the image on the display unit,

the processor executes instructions to display a different marker according to details of the event on the display unit,

the processor executes instructions to generate an image indicating the occurrence of the event, and

the processor executes instructions to display the image on the display unit, when the occurrence of the event is determined.