Method and system for enabling interaction in a test environment

• US 10,444,826 B2
• Filed: 03/10/2017
• Issued: 10/15/2019
• Est. Priority Date: 03/18/2016
• Status: Active Grant

• Current Assignee: Volvo Car Corporation (AB Volvo)
• Original Assignee: Volvo Car Corporation (AB Volvo)
• Inventors: Andersson, Patrik
• Primary Examiner(s): Dalbo, Michael J
• Application Number:
  US15/455,783
• Publication Number:
  US 20170269681A1
• Time in Patent Office: 949 Days
• CPC Class Codes: B60W 40/08 related to drivers or passengers
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 11/3664 Environments for testing or debugging software
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G07C 5/008 communicating information to a remotely located station transmission systems for measured values G08C
• CPC Class Codes: G09B 9/04 for teaching control of land vehicles
• CPC Class Codes: G09B 9/042 providing simulation in a real vehicle G09B9/052, G09B9/058 take precedence

Abstract

An interaction test system provides a first virtual reality layer associated with a first test environment to a first test object and a second virtual reality layer associated with a second test environment to a second test object, such that a mixed reality perceived by the first and second test objects correspond. The system derives first object data including status information and/or actions of the first test object, provides first object behavior data to the second virtual reality layer such that status information and/or actions of the first object are included in the mixed reality perceived by the second object, derives second object data including status information and/or actions of the second test object, and provides second object behavior data to the first virtual reality layer such that status information and/or actions of the second object are included in the mixed reality perceived by the first object.

Claims

1. A method performed by an interaction test system for enabling interaction in a test environment between a first test object comprising a vehicle, and at least a second test object adapted to communicate directly or indirectly with the first test object, wherein during a test session, the first test object acts within a first physical test environment and the at least second test object acts within at least a second physical test environment physically separated from the first test environment, the method comprising:

providing a first virtual reality layer associated with the first test environment to the first test object, and at least a second virtual reality layer associated with the at least second test environment to the at least second test object, such that a mixed reality perceived by the first test object corresponds to a mixed reality perceived by the at least second test object; and

continuously, periodically, or intermittently;

deriving first object data associated with the first test object, the first object data comprising status information and/or actions of the first test object;

providing first object behavior data based on the first object data to the at least second virtual reality layer, such that at least a portion of the status information and/or actions of the first test object are comprised in the mixed reality perceived by the at least second test object;

deriving at least second object data associated with the at least second test object, the at least second object data comprising status information and/or actions of the at least second test object; and

providing at least second object behavior data based on the at least second object data to the first virtual reality layer, such that at least a portion of the status information and/or actions of the at least second test object are comprised in the mixed reality perceived by the first test object.

2. The method according to claim 1 further comprising:

determining an interaction behavior of the first test object and/or the at least second test object, based on the derived first object data and/or the derived at least second object data; and

based on the interaction behavior;

providing an action instruction to the first and/or the at least second test object, and/or

adapting a data parameter of the first and/or at least second test object, the data parameter being associated with driving assistance functionality.

3. The method according to claim 1 wherein the first test object additionally comprises a vehicle occupant;

wherein the providing the first virtual reality layer comprises providing a first vehicle virtual reality layer to the vehicle and/or a first occupant virtual reality layer to a head-mounted display (HMD) carried by the vehicle occupant;

wherein the deriving first object data comprises deriving vehicle data from the vehicle, and/or occupant data from a first motion capture system carried by, and/or visually sensing, the vehicle occupant, the vehicle data comprising status information and/or actions of the vehicle and the occupant data comprising status information and/or actions of the vehicle occupant;

wherein the providing first object behavior data comprises providing vehicle behavior data based on the vehicle data and/or occupant behavior data based the occupant data, to the at least second virtual reality layer, such that at least a portion of the status information and/or actions of the vehicle and/or the vehicle occupant, are comprised in the mixed reality perceived by the at least second test object; and

wherein the providing at least second object behavior data comprises providing at least second object behavior data based on the at least second object data, to the first vehicle virtual reality layer and/or the first occupant virtual reality layer, such that at least a portion of the status information and/or actions of the at least second test object are comprised in the mixed reality perceived by the vehicle and/or in the mixed reality perceived by the vehicle occupant.

4. The method according to claim 1 wherein the second test object or a further test object comprises a vulnerable road user (VRU);

wherein the providing the at least second virtual reality layer comprises providing a VRU virtual reality layer to an HMD carried by the VRU; and

wherein the deriving at least second object data comprises deriving the second object data from a VRU motion capture system carried by, and/or visually sensing, the VRU.

5. The method according to claim 1 wherein the second test object or a further test object comprises a second vehicle and a vehicle occupant thereof;

wherein the providing the at least second virtual reality layer comprises providing a second vehicle virtual reality layer to the second vehicle and/or a second occupant virtual reality layer to an HMD carried by the vehicle occupant of the second vehicle;

wherein the providing first object behavior data comprises providing first object behavior data based on the first object data, to the second vehicle virtual reality layer and/or the second occupant virtual reality layer, such that at least a portion of the status information and/or actions of the first test object, is comprised in the mixed reality perceived by the second vehicle and/or in the mixed reality perceived by the HMD carried by the vehicle occupant of the second vehicle; and

wherein the deriving at least second object data comprises deriving second vehicle data from the second vehicle, and/or second occupant data from a second motion capture system carried by, and/or visually sensing, the second vehicle occupant, the second vehicle data comprising status information and/or actions of the second vehicle and the second occupant data comprising status information and/or actions of the second vehicle occupant; and

wherein the providing at least second object behavior data comprises providing second vehicle behavior data based on the second vehicle data and/or occupant behavior data based on the second occupant data, to the first virtual reality layer, such that at least a portion of the status information and/or actions of the second vehicle and/or the second vehicle occupant, are comprised in the mixed reality perceived by the first test object.

6. The method according to claim 1 wherein:

during the test session, at least a first auxiliary object acts within the first and/or the at least second physical test environment, the at least first auxiliary object being adapted to communicate directly or indirectly with the first test object and/or the at least second test object;

the method further comprising, continuously, periodically, or intermittently;

deriving auxiliary object data associated with the at least first auxiliary object, the auxiliary object data comprising status information and/or actions of the auxiliary object; and

providing auxiliary object behavior data based on the auxiliary object data, to the first and/or the at least second virtual reality layer, such that at least a portion of the status information and/or actions of the at least first auxiliary object are comprised in the mixed reality perceived by the first test object and/or in the mixed reality perceived by the at least second test object.

7. The method according to claim 1 further comprising:

deriving environment conditions data from a cloud service, the environment conditions data comprising one or more environment conditions associated with the first and/or at least second physical test environment; and

providing the environment conditions data to the first test object, the first virtual layer, the at least second test object and/or the at least second virtual layer, such that the environment conditions data are comprised in the mixed reality perceived by the first test object and/or in the mixed reality perceived by the at least second test object.

8. An interaction test system adapted for enabling interaction in a test environment between a first test object comprising a vehicle, and at least a second test object adapted to communicate directly or indirectly with the first test object, wherein during a test session, the first test object acts within a first physical test environment and the at least second test object acts within at least a second physical test environment physically separated from the first test environment, the interaction test system comprising:

a virtual reality layer providing unit adapted for providing a first virtual reality layer associated with the first test environment to the first test object, and at least a second virtual reality layer associated with the at least second test environment, to the at least second test object, such that a mixed reality perceived by the first test object corresponds to a mixed reality perceived by the at least second test object; and

a data deriving unit adapted for deriving continuously, periodically, or intermittently first object data associated with the first test object, the first object data comprising status information and/or actions of the first test object;

a behavior data providing unit adapted for providing continuously, periodically, or intermittently first object behavior data based on the first object data to the at least second virtual reality layer, such that at least a portion of the status information and/or actions of the first test object are comprised in the mixed reality perceived by the at least second test object;

wherein the data deriving unit further is adapted for deriving at least second object data associated with the at least second test object, the at least second object data comprising status information and/or actions of the at least second test object; and

wherein the behavior data providing unit further is adapted for providing continuously, periodically, or intermittently at least second object behavior data based on the at least second object data to the first virtual reality layer, such that at least a portion of the status information and/or actions of the at least second test object are comprised in the mixed reality perceived by the first test object.

9. The interaction test system according to claim 8 further comprising:

an interaction behavior determining unit adapted for determining an interaction behavior of the first test object and/or the at least second test object, based on the derived first object data and/or the derived at least second object data; and

an actions instruction providing unit adapted for providing, based on the interaction behavior, an action instruction to the first and/or the at least second test object, and/or

a data parameter adapting unit adapted for adapting, based on the interaction behavior, a data parameter of the first and/or at least second test object, the data parameter being associated with driving assistance functionality.

10. The interaction test system according to claim 8 wherein the first test object additionally comprises a vehicle occupant;

wherein the virtual reality layer providing unit further is adapted for providing a first vehicle virtual reality layer to the vehicle and/or a first occupant virtual reality layer to a head-mounted display (HMD) carried by the vehicle occupant;

wherein the data deriving unit further is adapted for deriving vehicle data from the vehicle, and/or occupant data from a first motion capture system carried by, and/or visually sensing, the vehicle occupant, the vehicle data comprising status information and/or actions of the vehicle and the occupant data comprising status information and/or actions of the vehicle occupant;

wherein the behavior data providing unit further is adapted for providing vehicle behavior data based on the vehicle data and/or occupant behavior data based on the occupant data, to the at least second virtual reality layer, such that at least a portion of the status information and/or actions of the vehicle and/or the vehicle occupant, are comprised in the mixed reality perceived by the at least second test object; and

wherein the behavior data providing unit further is adapted for providing at least second object behavior data based on the at least second object data, to the first vehicle virtual reality layer and/or the first occupant virtual reality layer, such that at least a portion of the status information and/or actions of the at least second test object are comprised in the mixed reality perceived by the vehicle and/or in the mixed reality perceived by the vehicle occupant.

11. The interaction test system according to claim 8 wherein the second test object or a further test object comprises a vulnerable road user (VRU);

wherein the virtual reality layer providing unit further is adapted for providing a VRU virtual reality layer to an HMD carried by the VRU; and

wherein the data deriving unit further is adapted for deriving the second object data from a VRU motion capture system carried by, and/or visually sensing, the VRU.

12. The interaction test system according to claim 8 wherein the second test object or a further test object comprises a second vehicle and a vehicle occupant thereof;

wherein the virtual reality layer providing unit further is adapted for providing a second vehicle virtual reality layer to the second vehicle and/or a second occupant virtual reality layer to an HMD carried by the second vehicle occupant;

wherein the behavior data providing unit further is adapted for providing first object behavior data based on the first object data, to the second vehicle virtual reality layer and/or the second occupant virtual reality layer, such that at least a portion of the status information and/or actions of the first test object, is comprised in the mixed reality perceived by the second vehicle and/or in the mixed reality perceived by the HMD carried by the second vehicle occupant; and

wherein the data deriving unit further is adapted for deriving second vehicle data from the second vehicle, and/or second occupant data from a second motion capture system carried by, and/or visually sensing, the second vehicle occupant, the second vehicle data comprising status information and/or actions of the second vehicle and the second occupant data comprising status information and/or actions of the second vehicle occupant; and

wherein the behavior data providing unit further is adapted for providing second vehicle behavior data based on the second vehicle data and/or second occupant behavior data based on the second occupant data, to the first virtual reality layer, such that at least a portion of the status information and/or actions of the second vehicle and/or the second vehicle occupant, are comprised in the mixed reality perceived by the first test object.

13. The interaction test system according to claim 8 further comprising at least a first auxiliary object adapted to during the test session act within the first and/or the at least second test environment, the at least first auxiliary object being adapted to communicate directly or indirectly with the first test object and/or the at least second test object;

wherein the data deriving unit further is adapted for deriving, continuously, periodically, or intermittently, auxiliary object data associated with the at least first auxiliary object, the auxiliary object data comprising status information and/or actions of the auxiliary object; and

wherein the behavior data providing unit further is adapted for providing, continuously, periodically, or intermittently, auxiliary object behavior data based on the auxiliary object data, to the first and/or the at least second virtual reality layer, such that at least a portion of the status information and/or actions of the at least first auxiliary object are comprised in the mixed reality perceived by the first test object and/or in the mixed reality perceived by the at least second test object.

14. The interaction test system according to claim 8 further comprising:

an environment conditions data deriving unit adapted for deriving environment conditions data from a cloud service, the environment conditions data comprising one or more environment conditions associated with the first and/or at least second test environment; and

an environment conditions providing unit adapted for providing the environment conditions data to the first test object, the first virtual layer, the at least second test object and/or the at least second virtual layer, such that the environment conditions data are comprised in the mixed reality perceived by the first test object and/or in the mixed reality perceived by the at least second test object.

15. A computer readable storage medium having non-transitory computer executable instructions recorded thereon for use in an interaction test system for enabling interaction in a test environment between a first test object comprising a vehicle, and at least a second test object adapted to communicate directly or indirectly with the first test object, wherein during a test session, the first test object acts within a first physical test environment and the at least second test object acts within at least a second physical test environment physically separated from the first test environment, the computer executable instructions comprising instructions for:

providing a first virtual reality layer associated with the first test environment to the first test object, and at least a second virtual reality layer associated with the at least second test environment to the at least second test object, such that a mixed reality perceived by the first test object corresponds to a mixed reality perceived by the at least second test object; and

continuously, periodically, or intermittently;

deriving first object data associated with the first test object, the first object data comprising status information and/or actions of the first test object;

providing first object behavior data based on the first object data to the at least second virtual reality layer, such that at least a portion of the status information and/or actions of the first test object are comprised in the mixed reality perceived by the at least second test object;

deriving at least second object data associated with the at least second test object, the at least second object data comprising status information and/or actions of the at least second test object; and

providing at least second object behavior data based on the at least second object data to the first virtual reality layer, such that at least a portion of the status information and/or actions of the at least second test object are comprised in the mixed reality perceived by the first test object.