Contextual augmented reality devices collaboration

• US 9,865,093 B1
• Filed: 06/30/2016
• Issued: 01/09/2018
• Est. Priority Date: 06/30/2016
• Status: Active Grant

• Current Assignee: META Platforms Technologies LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Original Assignee: DAQRI, LLC
• Inventors: Mullins, Brian
• Primary Examiner(s): Buttram, Todd
• Application Number:
  US15/199,570
• Publication Number:
  US 20180005442A1
• Time in Patent Office: 558 Days
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 2203/0383 Remote input, i.e. interface arrangements in which the signals generated by a pointing device are transmitted to a PC at a remote location, e.g. to a PC in a LAN
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: G06F 8/38 for implementing user interfaces
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G06Q 10/10 Office automation; Time management
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A first display device determines a first device configuration of the first display device and a first context of a first AR application operating on the first display device. The first display device connects to a second display device and determines a second device configuration of the second display device and a second context of a second AR application operating on the second display device. The first display device generates a first collaboration configuration for the display device and a second collaboration configuration for the second display device. The first and second collaborations are based on the first device configuration, the first context, the second device configuration, and the second context. The second display device modifies an operation of the second AR application based on the second collaboration configuration. The first display device modifies an operation of the first AR application based on the first collaboration configuration.

Claims

1. A first display device comprising:

one or more hardware processors; and

a memory storing instructions that, when executed by the one or more hardware processors, configure the first display device to perform operations comprising;

determining a first device configuration of the first display device and a first context of a first AR application operating on the first display device;

connecting to a second display device;

determining a second device configuration of the second display device and a second context of a second AR application operating on the second display device;

generating a first collaboration configuration for the first display device and a second collaboration configuration for the second display device, the first and second collaboration configurations being generated based on the first device configuration, the first context, the second device configuration, and the second context, the second display device being configured to modify an operation of the second AR application based on the second collaboration configuration; and

modifying an operation of the first AR application in the first display device based on the first collaboration configuration.

2. The display device of claim 1, wherein the operations further comprise:

identifying a common process executed by the first and second display devices based on a common operation identified in the first and second configurations and in the first and second contexts;

determining that the first display device is configured to perform the common process more efficiently than the second display device;

assigning the common process to the first display device in the first collaboration configuration; and

removing the common process from the second display device in the second collaboration configuration.

3. The display device of claim 1, wherein the modifying of the operation of the first AR application in the first display device comprises:

disabling a first display of the first display device; and

causing a second display of the second display device to display content from the first AR application using the second collaboration configuration on the second AR application of the second display device.

4. The display device of claim 1, wherein the operations further comprise:

determining that a first task of a first user of the first display device and a second task of a second user of the second display device are part of a general workflow;

generating first content and a first user interface for the first user of the first display device based on the first task, the first collaboration configuration including the first content and the first user interface; and

generating a second content and a second user interface for the second user of the first display device based on the second task, the second collaboration configuration including the second content and the second user interface.

5. The display device of claim 1, wherein the first configuration includes hardware and software specifications of the first display device, and hardware and software resource capacities and availabilities.

6. The display device of claim 1, wherein the first context includes a user-based context and an ambient-based context of the display device,

the user-based context being generated based on a user profile of a first user of the first display device, a user credential of the first user, and biometric data from the first user, the biometric data including at least one of a heart rate, a blood pressure, or brain activity,

the ambient-based context indicating at least one of a geographic location of the display device, an orientation and a position of the first display device, an ambient pressure, an ambient humidity level, or an ambient light level.

7. The display device of claim 1, wherein the first collaboration configuration includes allocation instructions of operations and tasks from the first AR application and the second AR application between the first display device and the second display device.

8. The display device of claim 1, further comprising:

a display; and

a plurality of sensors including a first set of sensors configured to measure first sensor data related to a user of the HMD, and a second set of sensors configured to measure second sensor data related to the HMD,

wherein the operations further comprise;

determining a third context including a user-based context based on the first set of sensors,

determining a fourth context including an ambient-based context based on the second set of sensors, and

generating AR content based on the third context and the fourth context, the first AR application being configured to display the AR content in the display.

9. The display device of claim 8, wherein the operations further comprises:

modifying the AR content based on the first collaboration configuration; and

causing a display of the modified AR content in the display of the first display device.

10. The display device of claim 8, wherein the operations further comprise:

determining a first authorization level of the first user based on the third context and the fourth context; and

determining a second authorization level of a second user of the second display device based on a fifth context including a user-based context of the second user and a sixth context including an ambient-based context of the second display device,

wherein the first and second collaboration configurations are based on the first authorization level of the first user and the second authorization level of the second user.

11. A method comprising:

determining a first device configuration of a first display device and a first context of a first AR application operating on the first display device;

connecting to a second display device;

determining a second device configuration of the second display device and a second context of a second AR application operating on the second display device;

generating a first collaboration configuration for the first display device and a second collaboration configuration for the second display device, the first and second collaboration configurations being generated based on the first device configuration, the first context, the second device configuration, and the second context, the second display device being configured to modify an operation of the second AR application based on the second collaboration configuration; and

modifying an operation of the first AR application in the first display device based on the first collaboration configuration.

12. The method of claim 11, further comprising:

identifying a common process executed by the first and second display devices based on a common operation identified in the first and second configurations and in the first and second contexts;

determining that the first display device is configured to perform the common process more efficiently than the second display device;

assigning the common process to the first display device in the first collaboration configuration; and

removing the common process from the second display device in the second collaboration configuration.

13. The method of claim 11, wherein the modifying of the operation of the first AR application in the first display device comprises:

disabling a first display of the first display device; and

causing a second display of the second display device to display content from the first AR application using the second collaboration configuration on the second AR application of the second display device.

14. The method of claim 11, further comprising:

determining that a first task of a first user of the first display device and a second task of a second user of the second display device are part of a general workflow;

generating first content and a first user interface for the first user of the first display device based on the first task, the first collaboration configuration including the first content and the first user interface; and

generating a second content and a second user interface for the second user of the first display device based on the second task, the second collaboration configuration including the second content and the second user interface.

15. The method of claim 11, wherein the first configuration includes hardware and software specifications of the first display device, and hardware and software resource capacities and availabilities.

16. The method of claim 11, wherein the first context includes user-based context and ambient-based context of the first display device,

the user-based context being generated based on a user profile of a first user of the first display device, a user credential of the first user, and biometric data from the first user, the biometric data including at least one of a heart rate, a blood pressure, or brain activity,

the ambient-based context indicating at least one of a geographic location of the display device, an orientation and a position of the first display device, an ambient pressure, an ambient humidity level, or an ambient light level.

17. The method of claim 11, wherein the first collaboration configuration includes allocation instructions of operations and tasks from the first AR application and the second AR application between the first display device and the second display device.

18. The method of claim 11, wherein the first display device comprises a display, and a plurality of sensors including a first set of sensors configured to measure first sensor data related to a user of the HMD, and a second set of sensors configured to measure second sensor data related to the HMD,

wherein the method further comprises:

determining a third context including a user-based context based on the first set of sensors,

determining a fourth context including an ambient-based context based on the second set of sensors,

generating AR content based on the third context and the fourth context, the first AR application being configured to display the AR content in the display,

modifying the AR content based on the first collaboration configuration; and

causing a display of the modified AR content in the display of the first display device.

19. The method of claim 18, further comprising:

determining a first authorization level of the first user based on the third context and the fourth context; and

determining a second authorization level of a second user of the second display device based on a fifth context including a user-based context of the second user and a sixth context including an ambient-based context of the second display device,

wherein the first and second collaboration configurations are based on the first authorization level of the first user and the second authorization level of the second user.

20. A non-transitory machine-readable medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:

determining a first device configuration of a first display device and a first context of a first AR application operating on the first display device;

connecting to a second display device;

determining a second device configuration of the second display device and a second context of a second AR application operating on the second display device;

generating a first collaboration configuration for the first display device and a second collaboration configuration for the second display device, the first and second collaboration configurations being generated based on the first device configuration, the first context, the second device configuration, and the second context, the second display device being configured to modify an operation of the second AR application based on the second collaboration configuration; and

modifying an operation of the first AR application in the first display device based on the first collaboration configuration.