Human-machine interface (HMI) architecture

• US 10,452,065 B2
• Filed: 07/03/2017
• Issued: 10/22/2019
• Est. Priority Date: 07/03/2017
• Status: Active Grant

• Current Assignee: Baidu USA LLC (Baidu Incorporated)
• Original Assignee: Baidu USA LLC (Baidu Incorporated)
• Inventors: Xiao, Xiangquan, Wang, Lei, Hu, Jiangtao, Gao, Li
• Primary Examiner(s): Badii, Behrang
• Assistant Examiner(s): Greene, Daniel L
• Application Number:
  US15/640,867
• Publication Number:
  US 20190004510A1
• Time in Patent Office: 841 Days
• CPC Class Codes: B60W 30/00 Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units
• CPC Class Codes: G05D 1/0022 characterised by the communication link data switching networks in general H04L12/00
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G07C 5/008 communicating information to a remotely located station transmission systems for measured values G08C
• CPC Class Codes: G07C 5/0841 Registering performance data recording measured values G01D; information storage G11B
• CPC Class Codes: H04L 67/01 Protocols
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/12 specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04W 4/48 for in-vehicle communication

Abstract

An autonomous driving vehicle (ADV) is operated using a human-machine interface (HMI). The web server provides the HMI to a computing device in response to an input received from the computing device. An ADV command is entered into the HMI and passed to an interface of the web server. In response to receiving the ADV command, the web server calls a remote procedure call to a proxy server in a backend server for processing by a perception and control module of the ADV. Results of the ADV command are received by the web server interface and stored in results memory with a unique identifier. The web server interface opens a socket that accesses the results memory. In response to a change in the results memory, the socket reads the results memory and provides the ADV command results to the HMI. Multiple HMIs can simultaneously communicate with the web server interface and socket.

Claims

1. A computer-implemented method of operating an autonomous driving vehicle (ADV), the method comprising:

receiving, at a Web interface of a web server, an ADV command from a first client application, the ADV command being received using a Web communication protocol, wherein the first client application is one of a plurality of client applications hosted by the Web server;

in response to the ADV command, transmitting the ADV command via a remote procedure call (RPC) to a proxy server of a robotic operating system of the ADV, the proxy server configured to route the ADV command to an ADV perception and control system via a Controller Area Network (CAN) message bus; and

in response to receiving an ADV command response from the ADV perception and control system, wherein the ADV command response is stored in a results memory of the web server, multicasting the ADV command response to the first client application via a first socket connection and to a second client application of the plurality of client applications via a second socket connection.

2. The method of claim 1, further comprising:

storing the ADV command response in the results memory of the web server, in association with a unique identifier of the ADV command response; and

opening a socket to read from the results memory at least a portion of the ADV command response that includes the unique identifier in response to a change in the results, wherein the socket retrieves the ADV command response from the results memory and transmit the ADV command to the first and second client applications via the first and second socket connections respectively.

3. The method of claim 2, wherein the unique identifier of the ADV response comprises one or more of:

a hash of the ADV command response, a checksum of the ADV command response, or a unique identifier associated with the ADV command response generated by the ADV perception and control module.

4. The method of claim 1, wherein the ADV command comprises one of:

start, or stop, recording of an ADV driving session;

playback a recorded ADV driving session;

turn on, or off, a module of the ADV;

or

check, and return a status of, a module of the ADV, wherein a module of the ADV comprises one or more of;

a global positioning system (GPS), a control system of the ADV, a control area network (CAN) communication bus of the ADV, a localization module of the ADV, or a data recording system of the ADV.

5. The method of claim 1, wherein the Web communication protocol comprises a representational state transfer (REST) protocol.

6. The method of claim 1, wherein the ADV command response is received by the web server via an application programming interface (API) of the web server called by the ADV perception and control system of the robotic operating system.

7. The method of claim 1, further comprising:

reading, by the socket, the unique identifier of the ADV command response; and

determining that the unique identifier associated with the ADV command response has changed.

8. A non-transitory machine-readable medium having executable instructions stored therein, which when executed by a processor, cause the processor to perform operations for operating an autonomous driving vehicle (ADV), the operations comprising:

receiving, at a Web interface of a web server, an ADV command from a first client application, the ADV command being received using a Web communication protocol, wherein the first client application is one of a plurality of client applications hosted by the Web server;

in response to the ADV command, transmitting the ADV command via a remote procedure call (RPC) to a proxy server of a robotic operating system of the ADV, the proxy server configured to route the ADV command to an ADV perception and control system via a Controller Area Network (CAN) message bus; and

in response to receiving an ADV command response from the ADV perception and control system, wherein the ADV command response is stored in a results memory of the web server, multicasting the ADV command response to the first client application via a first socket connection and to a second client application of the plurality of client applications via a second socket connection.

9. The medium of claim 8, wherein the operations further comprise:

storing the ADV command response in the results memory of the web server, in association with a unique identifier of the ADV command response; and

opening a socket to read from the results memory at least a portion of the ADV command response that includes the unique identifier in response to a change in the results, wherein the socket retrieves the ADV command response from the results memory and transmit the ADV command to the first and second client applications via the first and second socket connections respectively.

10. The medium of claim 9, wherein the unique identifier of the ADV response comprises one or more of:

a hash of the ADV command response, a checksum of the ADV command response, or a unique identifier associated with the ADV command response generated by the ADV perception and control module.

11. The medium of claim 8, wherein the ADV command comprises one of:

start, or stop, recording of an ADV driving session;

playback a recorded ADV driving session;

turn on, or off, a module of the ADV;

or

check, and return a status of, a module of the ADV, wherein a module of the ADV comprises one or more of;

a global positioning system (GPS), a control system of the ADV, a control area network (CAN) communication bus of the ADV, a localization module of the ADV, or a data recording system of the ADV.

12. The medium of claim 8, wherein the Web communication protocol comprises a representational state transfer (REST) protocol.

13. The medium of claim 8, wherein the ADV command response is received by the web server via an application programming interface (API) of the web server called by the ADV perception and control system of the robotic operating system.

14. The medium of claim 8, wherein the operations further comprise:

reading, by the socket, the unique identifier of the ADV command response; and

determining that the unique identifier associated with the ADV command response has changed.

15. A web server comprising:

a processor; and

a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to perform operations for operating an autonomous driving vehicle (ADV), the operations comprising;

receiving, at a Web interface of a web server, an ADV command from a first client application, the ADV command being received using a Web communication protocol, wherein the first client application is one of a plurality of client applications hosted by the Web server,

in response to the ADV command, transmitting the ADV command via a remote procedure call (RPC) to a proxy server of a robotic operating system of the ADV, the proxy server configured to route the ADV command to an ADV perception and control system via a Controller Area Network (CAN) message bus, and

in response to receiving an ADV command response from the ADV perception and control system, wherein the ADV command response is stored in a results memory of the web server, multicasting the ADV command response to the first client application via a first socket connection and to a second client application of the plurality of client applications via a second socket connection.

16. The web server of claim 15, wherein the operations further comprise:

storing the ADV command response in the results memory of the web server, in association with a unique identifier of the ADV command response; and

opening a socket to read from the results memory at least a portion of the ADV command response that includes the unique identifier in response to a change in the results, wherein the socket retrieves the ADV command response from the results memory and transmit the ADV command to the first and second client applications via the first and second socket connections respectively.

17. The web server of claim 16, wherein the unique identifier of the ADV response comprises one or more of:

a hash of the ADV command response, a checksum of the ADV command response, or a unique identifier associated with the ADV command response generated by the ADV perception and control module.

18. The web server of claim 15, wherein the ADV command comprises one of:

start, or stop, recording of an ADV driving session;

playback a recorded ADV driving session;

turn on, or off, a module of the ADV;

or

check, and return a status of, a module of the ADV, wherein a module of the ADV comprises one or more of;

a global positioning system (GPS), a control system of the ADV, a control area network (CAN) communication bus of the ADV, a localization module of the ADV, or a data recording system of the ADV.

19. The web server of claim 15, wherein the Web communication protocol comprises a representational state transfer (REST) protocol.

20. The web server of claim 15, wherein the ADV command response is received by the web server via an application programming interface (API) of the web server called by the ADV perception and control system of the robotic operating system.

21. The web server of claim 15, wherein the socket reads the unique identifier of the ADV command response and determine that the unique identifier associated with the ADV command response has changed.