CONTEXTUAL AUGMENTED REALITY DEVICES COLLABORATION

• US 20180047216A1
• Filed: 10/26/2017
• Published: 02/15/2018
• Est. Priority Date: 06/30/2016
• Status: Active Grant

• Current Assignee: META Platforms Technologies LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Original Assignee: DAQRI, LLC
• Inventors: Mullins, Brian
• Granted Patent: US 10,445,937 B2
• Time in Patent Office: Days
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 2203/0383 Remote input, i.e. interface arrangements in which the signals generated by a pointing device are transmitted to a PC at a remote location, e.g. to a PC in a LAN
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: G06F 8/38 for implementing user interfaces
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G06Q 10/10 Office automation; Time management
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A first display device determines a first device configuration of the first display device and a first context of a first AR application operating on the first display device. The first display device detects a second display device that comprises a head up display (HUD) system of a vehicle. A second device configuration of the second display device and a second context of a second AR application operating on the second display device are determined. The first display device generates a first collaboration configuration for the first display device and a second collaboration configuration for the second display device. A task of the first AR application is allocated to the second AR application based on the first collaboration configuration. The second display device performs the allocated task based on the second collaboration configuration.

Claims

1. A first display device comprising:

a head-mounted device;

one or more hardware processors; and

a memory storing instructions that, when executed by the one or more hardware processors, cause the first display device to perform operations comprising;

determining a first device configuration of the first display device and a first context of a first AR application operating on the first display device;

detecting a second display device that comprises a head up display (HUD) system of a vehicle;

determining a second device configuration of the second display device and a second context of a second AR application operating on the second display device;

generating a first collaboration configuration for the first display device and a second collaboration configuration for the second display device, the first and second collaboration configurations being generated based on the first device configuration, the first context, the second device configuration, and the second context; and

allocating a task of the first AR application to the second AR application based on the first collaboration configuration, the second display device being configured to perform the allocated task based on the second collaboration configuration.

2. The first display device of claim 1, wherein the first device configuration identifies a first display size, wherein the second device configuration identifies a second display size,

wherein the operations further comprise:

determining that the second display size is larger than the first display size; and

the allocating of the task includes allocating a display operation of the first AR application to the second AR application in response to determining that the second display size is larger than the first display size.

3. The first display device of claim 1, wherein the first device configuration identifies a first computing capacity, wherein the second device configuration identifies a second computing capacity,

wherein the operations further comprise:

determining that the second computing capacity is larger than the first computing capacity; and

the allocating of the task includes allocating a computation operation from the first AR application to be performed by the second AR application in response to determining that the second computing capacity is larger than the first computing capacity.

4. The first display device of claim 1, wherein the operations further comprise:

modifying a first task of the first AR application in the first display device based on the first collaboration configuration; and

modifying a second task of the second AR application in the second display device based on the second collaboration configuration.

5. The first display device of claim 1, wherein the operations further comprise:

identifying a common task executed by the first and second display devices based on the common task identified in the first and second configurations and in the first and second contexts;

determining that the first display device is configured to perform the common task more efficiently than the second display device;

assigning the common task to the first display device in the first collaboration configuration; and

removing the common task from the second display device in the second collaboration configuration.

6. The first display device of claim 1, wherein the operations further comprise:

disabling a first display of the first display device; and

causing a second display of the second display device to display content from the first AR application using the second collaboration configuration on the second AR application of the second display device.

7. The first display device of claim 1, wherein the first device configuration identifies hardware and software specifications of the first display device, and hardware and software resource capacities and availabilities of the first display device, and

wherein the second device configuration identifies hardware and software specifications of the second display device, and hardware and software resource capacities and availabilities of the second display device.

8. The first display device of claim 1, wherein the first context includes a user-based context and an ambient-based context of the first display device,

the user-based context being generated based on a user profile of a first user of the first display device, a user credential of the first user, and biometric data from the first user, the biometric data including at least one of a heart rate, a blood pressure, or brain activity, and

the ambient-based context indicating at least one of a geographic location of the first display device, an orientation and a position of the first display device, an ambient pressure, an ambient humidity level, or an ambient light level.

9. The first display device of claim 1, wherein the first collaboration configuration includes an allocation instruction of tasks from the first AR application to the second AR application.

10. The first display device of claim 1, wherein the operations further comprise:

generating first AR content based on the first collaboration configuration;

generating second AR content based on the second collaboration configuration;

causing a display of the first AR content in the display of the first display device; and

causing a display of the second AR content in the display of the second display device.

11. A method comprising:

determining a first device configuration of the first display device and a first context of a first AR application operating on the first display device, the first display device comprising a head mounted device;

detecting a second display device that comprises a head up display (HUD) system of a vehicle;

determining a second device configuration of the second display device and a second context of a second AR application operating on the second display device;

generating a first collaboration configuration for the first display device and a second collaboration configuration for the second display device, the first and second collaboration configurations being generated based on the first device configuration, the first context, the second device configuration, and the second context; and

allocating a task of the first AR application to the second AR application based on the first collaboration configuration, the second display device being configured to perform the allocated task based on the second collaboration configuration.

12. The method of claim 11, wherein the first device configuration identifies a first display size, wherein the second device configuration identifies a second display size,

wherein the method further comprises:

determining that the second display size is larger than the first display size; and

the allocating of the task includes allocating a display operation of the first AR application to the second AR application in response to determining that the second display size is larger than the first display size.

13. The method of claim 11, wherein the first device configuration identifies a first computing capacity, wherein the second device configuration identifies a second computing capacity,

wherein the method further comprises:

determining that the second computing capacity is larger than the first computing capacity; and

the allocating of the task includes allocating a computation operation from the first AR application to be performed by the second AR application in response to determining that the second computing capacity is larger than the first computing capacity.

14. The method of claim 11, further comprising:

modifying a first task of the first AR application in the first display device based on the first collaboration configuration; and

modifying a second task of the second AR application in the second display device based on the second collaboration configuration.

15. The method of claim 11, further comprising:

identifying a common task executed by the first and second display devices based on the common task identified in the first and second configurations and in the first and second contexts;

determining that the first display device is configured to perform the common task more efficiently than the second display device;

assigning the common task to the first display device in the first collaboration configuration; and

removing the common task from the second display device in the second collaboration configuration.

16. The method of claim 11, further comprising:

disabling a first display of the first display device; and

causing a second display of the second display device to display content from the first AR application using the second collaboration configuration on the second AR application of the second display device.

17. The method of claim 11, wherein the first device configuration identifies hardware and software specifications of the first display device, and hardware and software resource capacities and availabilities of the first display device, and

wherein the second device configuration identifies hardware and software specifications of the second display device, and hardware and software resource capacities and availabilities of the second display device.

18. The method of claim 11, wherein the first context includes a user-based context and an ambient-based context of the first display device,

the user-based context being generated based on a user profile of a first user of the first display device, a user credential of the first user, and biometric data from the first user, the biometric data including at least one of a heart rate, a blood pressure, or brain activity, and

the ambient-based context indicating at least one of a geographic location of the first display device, an orientation and a position of the first display device, an ambient pressure, an ambient humidity level, or an ambient light level.

19. The method of claim 11, wherein the first collaboration configuration includes an allocation instruction of tasks from the first AR application to the second AR application.

20. A non-transitory machine-readable medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:

determining a first device configuration of the first display device and a first context of a first AR application operating on the first display device, the first display device comprising a head mounted device;

detecting a second display device that comprises a head up display (HUD) system of a vehicle;

determining a second device configuration of the second display device and a second context of a second AR application operating on the second display device;

generating a first collaboration configuration for the first display device and a second collaboration configuration for the second display device, the first and second collaboration configurations being generated based on the first device configuration, the first context, the second device configuration, and the second context; and

allocating a task of the first AR application to the second AR application based on the first collaboration configuration, the second display device being configured to perform the allocated task based on the second collaboration configuration.