Dynamic sensor array for augmented reality system

• US 9,934,754 B2
• Filed: 09/30/2015
• Issued: 04/03/2018
• Est. Priority Date: 09/30/2015
• Status: Active Grant

• Current Assignee: RPX Corporation
• Original Assignee: DAQRI, LLC
• Inventors: Mullins, Brian, Kammerait, Matthew
• Primary Examiner(s): WANG, JIN CHENG
• Application Number:
  US14/870,701
• Publication Number:
  US 20170092220A1
• Time in Patent Office: 916 Days
• Field of Search: 345 7, 345 8
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 2200/1637 Sensing arrangement for detection of housing movement or orientation, e.g. for controlling scrolling or cursor movement on the display of an handheld computer
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: G06F 3/1423 controlling a plurality of local displays, e.g. CRT and flat panel display
• CPC Class Codes: G06F 3/147 using display panels
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09G 2370/022 Centralised management of display operation, e.g. in a server instead of locally
• CPC Class Codes: G09G 2370/16 Use of wireless transmission of display information
• CPC Class Codes: G09G 3/001 using specific devices not provided for in groups G09G3/02 - G09G3/36, e.g. using an intermediate record carrier such as a film slide; Projection systems; Display of non-alphanumerical information, solely or in combination with alphanumerical information, e.g. digital display on projected diapositive as background slide projectors per se G03B23/00 = 42 HP
• CPC Class Codes: G09G 5/003 Details of a display terminal, the details relating to the control arrangement of the display terminal and to the interfaces thereto specific for a CRT G09G1/165; for a flat panel G09G3/2092
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/383 for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
• CPC Class Codes: H04N 2013/0074 : Stereoscopic image analysis

Abstract

A system and method for generating a dynamic sensor array for an augmented reality system is described. A head mounted device includes one or more sensors, an augmented reality (AR) application, and a sensor array module. The sensor array module identifies available sensors from other head mounted devices that are geographically located within a predefined area. A dynamic sensor array is formed based on the available sensors and the one or more sensors. The dynamic sensor array is updated based on an operational status of the available sensors and the one or more sensors. The AR application generates AR content based on data from the dynamic sensor array. A display of the head mounted device displays the AR content.

Claims

1. A head mounted device comprising:

one or more sensors;

one or more processors comprising an augmented reality (AR) application and a sensor array module;

the sensor array module configured to;

identify a sensor, in the one or more sensors of the head mounted device, as being a defective sensor;

in response to the identifying the defective sensor, identify sensors included by a plurality of other head mounted devices that are geographically located within a predefined area and that are operational;

form a dynamic sensor array that includes the identified sensors of the plurality of other head mounted devices and includes the one or more sensors of the head mounted device;

identify an operating sensor, in the dynamic sensor array, that is configured to perform a function similar to the defective sensor;

update the dynamic sensor array by excluding the defective sensor from the dynamic sensor array;

the AR application configured to generate AR content based on data from the dynamic sensor array; and

a display configured to display the AR content.

2. The head mounted device of claim 1, wherein the sensor array module is configured to:

identify an additional sensor from the dynamic sensor array, the additional sensor including a sensor of the plurality of other head mounted devices, the additional sensor performing a function similar to the one or more sensors in the head mounted device, and

access data from the additional sensor,

wherein the AR application is configured to generate the AR content based on data from the additional sensor and the one or more sensors of the head mounted device.

3. The head mounted device of claim 1, wherein the sensor array module is configured to:

identify a complimentary sensor from the dynamic sensor array, the complimentary sensor including a sensor of the plurality of other head mounted devices, the complimentary sensor performing a function different from the one or more sensors in the head mounted device, and

access data from the complimentary sensor,

wherein the AR application is configured to generate the AR content based on data from the complimentary sensor and the one or more sensors of the head mounted device.

4. The head mounted device of claim 1, wherein the sensor array module is configured to:

identify a context of the head mounted device,

identify a context sensor from the dynamic sensor array, the context sensor including a sensor of the plurality of other head mounted devices, the context sensor identified based on the context of the head mounted device, and

access data from the context sensor,

wherein the AR application is configured to generate the AR content based on data from the context sensor and the one or more sensors of the head mounted device.

5. The head mounted device of claim 4, wherein the context include a predefined geographic location,

wherein the AR application is configured to generate the AR content in response to detecting that the head mounted device is located or approaching the predefined geographic location.

6. The head mounted device of claim 4, wherein the context include a task of the AR application,

wherein the AR application is configured to generate the AR content in response to detecting a selection of the task in the head mounted device.

7. The head mounted device of claim 4, wherein the context include a user privilege,

wherein the AR application is configured to generate the AR content based on the user privilege.

8. The head mounted device of claim 1, wherein the sensor array module is configured to identify a plurality of stationary sensors located within the predefined area, the plurality of stationary sensors being external to any head mounted device, and to form the dynamic sensor array based on the plurality of stationary sensors, the available sensors from the plurality of other head mounted devices, and the one or more sensors of the head mounted device.

9. The head mounted device of claim 1, wherein the sensor array module is configured to identify one or more sensors from the dynamic sensor array based on a physical object captured with a camera of the head mounted device, the dynamic array including a plurality of stationary sensors being external to any head mounted device, and to access data from the identified one or more sensors from the dynamic sensor array,

wherein the AR application is configured to generate the AR content based on the data from the identified one or more sensors from the dynamic sensor array.

10. A server comprising:

one or more processors comprising a sensor array module;

the sensor array module configured to;

identify a sensor, in the one or more sensors of the head mounted device, as being a defective sensor;

in response to the identifying the defective sensor, identify sensors included by a plurality of other head mounted devices that are geographically located within a predefined area and that are operational;

form a dynamic sensor array that includes the identified sensors of the plurality of other head mounted devices and includes the one or more sensors of the head mounted device;

identify an operating sensor, in the dynamic sensor array, that is configured to perform a function similar to the defective sensor;

update the dynamic sensor array by excluding the defective sensor from the dynamic sensor array;

provide data from the dynamic sensor array to a first head mounted device, the first head mounted device configured to generate AR content based on the data from the dynamic sensor array.

11. The server of claim 10, wherein the sensor array module is configured to:

identify an additional sensor from the dynamic sensor array, the additional sensor including a sensor of the plurality of head mounted devices, the additional sensor performing a function similar to a sensor in the first head mounted device, and

access and provide data from the additional sensor to the first head mounted device.

12. The server of claim 10, wherein the sensor array module is configured to:

identify a complimentary sensor from the dynamic sensor array, the complimentary sensor including a sensor of the plurality of head mounted devices, the complimentary sensor performing a function different from a sensor in the first head mounted device, and

access and provide data from the complimentary sensor to the first head mounted device.

13. The server of claim 10, wherein the sensor array module is configured to:

identify a context of the first head mounted device,

identify a context sensor from the dynamic sensor array, the context sensor including a sensor of the plurality of head mounted devices, the context sensor identified based on the context of the first head mounted device, and

access and provide data from the context sensor to the first head mounted device.

14. A computer-implemented method comprising:

identifying a plurality of head mounted devices geographically located within a predefined area;

identifying sensors included by the plurality of head mounted devices;

forming, using a hardware processor of a computer, a dynamic sensor array that includes the identified sensors;

identifying a non-operating sensor and an operating sensor in the dynamic sensor array;

determining that the operating sensor and the non-operating sensor each include a same sensor that is configured to generate a same type of data;

updating the dynamic sensor array by excluding the non-operating sensor from the dynamic sensor array in response to the determining; and

providing data from the dynamic sensor array to a first head mounted device, the first head mounted device configured to generate AR content based on the data from the dynamic sensor array.

15. The computer-implemented method of claim 14, further comprising:

identifying a defective sensor in the first head mounted device;

identifying a replacement sensor from the dynamic sensor array, the replacement sensor including a sensor of the plurality of head mounted devices, the replacement sensor performing a function similar to the defective sensor; and

accessing and providing data from the replacement sensor to the first head mounted device.

16. The computer-implemented method of claim 14, further comprising:

identifying an additional sensor from the dynamic sensor array, the additional sensor including a sensor of the plurality of head mounted devices, the additional sensor performing a function similar to a sensor in the first head mounted device; and

accessing and providing data from the additional sensor to the first head mounted device.

17. The computer-implemented method of claim 14, further comprising:

identifying a complimentary sensor from the dynamic sensor array, the complimentary sensor including a sensor of the plurality of head mounted devices, the complimentary sensor performing a function different from a sensor in the first head mounted device; and

accessing and providing data from the complimentary sensor to the first head mounted device.

18. The computer-implemented method of claim 14, further comprising:

identifying a context of the first head mounted device;

identifying a context sensor from the dynamic sensor array, the context sensor including a sensor of the plurality of head mounted devices, the context sensor identified based on the context of the first head mounted device; and

accessing and providing data from the context sensor to the first head mounted device.

19. A non-transitory machine-readable medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:

identifying a sensor, in one or more sensors of a head mounted device, as being a defective sensor;

in response to the identifying the defective sensor, identifying sensors included by a plurality of other head mounted devices that are geographically located within a predefined area and that are operational;

forming a dynamic sensor array that includes the identified sensors of the plurality of other head mounted devices and includes the one or more sensors of the head mounted device;

identifying an operating sensor, in the dynamic sensor array, that is configured to perform a function similar to the defective sensor;

updating the dynamic sensor array by excluding the defective sensor from the dynamic sensor array;

providing data from the dynamic sensor array to a first head mounted device, the first head mounted device configured to generate AR content based on the data from the dynamic sensor array.