UBIQUITOUS NATURAL USER SYSTEM

• US 20140354528A1
• Filed: 05/28/2013
• Published: 12/04/2014
• Est. Priority Date: 05/28/2013
• Status: Active Grant

• Current Assignee: The Boeing Co.
• Original Assignee: The Boeing Co.
• Inventors: Laughlin, Brian Dale, Kelsey, William David, Blair, Richard N.
• Granted Patent: US 9,395,810 B2
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00

Abstract

A system is provided for that includes sensor(s) configured to provide sensed input characteristic of a user, the user'"'"'s environment or the user'"'"'s interaction with their environment, and including from a 3D scanner, measurements of points on a surface of an object in the user'"'"'s environment. A front-end system may receive and process the sensed input including the measurements to identify a known pattern that indicates a significance of the sensed input from which to identify operations of electronic resource(s). The front-end system may form and communicate an input to the electronic resource(s) to cause the resource(s) to perform the operations, including generation of a point cloud from the measurements, and transformation of the point cloud to a 3D model of the object. And the front-end system may receive an output including the 3D model from the resource(s), and communicate the output for display by a display device.

Claims

1. A ubiquitous natural user system comprising:

one or more sensors configured to sense a user, an environment of the user or an interaction of the user with the environment, and provide sensed input characteristic thereof, the sensors including a three-dimensional scanner configured to provide measurements of points on a surface of an object in the environment, and the sensed input including the measurements provided by the three-dimensional scanner; and

a front-end system coupled to the one or more sensors, the front-end system being configured to receive and process the sensed input including the measurements to identify a known pattern that indicates a significance of the sensed input from which to identify operations of one or more electronic resources,

wherein the front-end system is configured to form and communicate an input to cause the one or more electronic resources to perform the operations, including generation of a point cloud from the measurements, and transformation of the point cloud to a three-dimensional model of the object, and

wherein the front-end system is configured to receive an output including the three-dimensional model from the one or more electronic resources, and communicate the output for display by a display device.

2. The ubiquitous natural user system of claim 1, wherein the front-end system is configured to communicate the output for display by a personal display system that includes the display device, the personal display system being configured to generate or enable a live or direct view of the environment, augmented by the output displayed by the display device.

3. The ubiquitous natural user system of claim 2, wherein the front-end system is configured to communicate the output for display by a plurality of personal display systems that include respective display devices and are wearable by a respective plurality of users in the environment, and

wherein the three-dimensional model displayed by the respective display devices is anchored to the environment, but varied for each of the personal display systems depending on viewpoints of the respective users within the environment.

4. The ubiquitous natural user system of claim 1, wherein the object in the environment in which the three-dimensional scanner is configured to provide measurements includes a mockup of a component of a complex system, and

wherein the front-end system is configured to communicate the output for display to the user or another user in an instance in which the environment of the user or other user includes an empty space of the complex system designed for placement of the component, a live or direct view of the empty space being augmented by the three-dimensional model.

5. The ubiquitous natural user system of claim 1, wherein the one or more sensors further include a sensor configured to capture a gesture of the user indicating alteration to a portion of the three-dimensional model, and the sensed input includes the gesture captured by the sensor,

wherein the front-end system is configured to receive and process the sensed input including the gesture captured by the sensor, and form and communicate another input to cause the same or another one or more electronic resources to carry out the alteration indicated by the gesture, and

wherein the front-end system is configured to receive an output including at least the altered portion of the three-dimensional model, and communicate the output for display of the three-dimensional model including the altered portion.

6. The ubiquitous natural user system of claim 1, wherein the object in the environment in which the three-dimensional scanner is configured to provide measurements includes a mockup of a component of a complex system, and

wherein the front-end system is further configured to receive an output from an electronic resource simulation of operation of the complex system, and communicate the output for display by the display device.

7. The ubiquitous natural user system of claim 1, wherein the front-end system is configured to communicate the output for display by a plurality of display devices of a respective plurality of users, the output for each of the display devices being varied depending on roles of the respective users.

8. A method comprising:

sensing by one or more sensors, a user, an environment of the user or an interaction of the user with the environment, and providing sensed input characteristic thereof, the sensors including a three-dimensional scanner providing measurements of points on a surface of an object in the environment, and the sensed input including the measurements provided by the three-dimensional scanner;

receiving and processing the sensed input including the measurements to identify a known pattern that indicates a significance of the sensed input from which to identify operations of one or more electronic resources;

forming and communicating an input to cause the one or more electronic resources to perform the operations, including generation of a point cloud from the measurements, and transformation of the point cloud to a three-dimensional model of the object; and

receiving an output including the three-dimensional model from the one or more electronic resources, and communicating the output for display by a display device.

9. The method of claim 8, wherein the output is communicated for display by a personal display system that includes the display device, the personal display system being configured to generate or enable a live or direct view of the environment, augmented by the output displayed by the display device.

10. The method of claim 9, wherein the output is communicated for display by a plurality of personal display systems that include respective display devices and are wearable by a respective plurality of users in the environment, and

wherein the three-dimensional model displayed by the respective display devices is anchored to the environment, but varied for each of the personal display systems depending on viewpoints of the respective users within the environment.

11. The method of claim 8, wherein the object in the environment in which the three-dimensional scanner provides measurements includes a mockup of a component of a complex system, and

wherein the output is communicated for display to the user or another user in an instance in which the environment of the user or other user includes an empty space of the complex system designed for placement of the component, a live or direct view of the empty space being augmented by the three-dimensional model.

12. The method of claim 8, wherein the one or more sensors further include a sensor capturing a gesture of the user indicating alteration to a portion of the three-dimensional model, and the sensed input includes the gesture captured by the sensor,

wherein receiving and processing the sensed input includes the receiving and processing the gesture captured by the sensor, and forming and communicating the input includes forming and communicating another input to cause the same or another one or more electronic resources to carry out the alteration indicated by the gesture, and

wherein receiving an output includes receiving an output including at least the altered portion of the three-dimensional model, the output being communicated for display of the three-dimensional model including the altered portion.

13. The method of claim 8, wherein the object in the environment in which the three-dimensional scanner provides measurements includes a mockup of a component of a complex system, and wherein the method further comprises:

receiving an output from an electronic resource simulation of operation of the complex system, and communicating the output for display by the display device.

14. The method of claim 8, wherein the output is communicated for display by a plurality of display devices of a respective plurality of users, the output for each of the display devices being varied depending on roles of the respective users.

15. A computer-readable storage medium having computer-readable program code stored therein that, in response to execution by a processor, cause an apparatus to at least:

receive from one or more sensors, sensed input characteristic of a user, an environment of the user or an interaction of the user with the environment, the sensors including a three-dimensional scanner configured to provide measurements of points on a surface of an object in the environment, and the sensed input including the measurements provided by the three-dimensional scanner;

process the sensed input including the measurements to identify a known pattern that indicates a significance of the sensed input from which to identify operations of one or more electronic resources;

form and communicate an input to cause the one or more electronic resources to perform the operations, including generation of a point cloud from the measurements, and transformation of the point cloud to a three-dimensional model of the object; and

receive an output including the three-dimensional model from the one or more electronic resources, and communicate the output for display by a display device.

16. The computer-readable storage medium of claim 15, wherein the output is communicated for display by a personal display system that includes the display device, the personal display system being configured to generate or enable a live or direct view of the environment, augmented by the output displayed by the display device.

17. The computer-readable storage medium of claim 16, wherein the output is communicated for display by a plurality of personal display systems that include respective display devices and are wearable by a respective plurality of users in the environment, and

wherein the three-dimensional model displayed by the respective display devices is anchored to the environment, but varied for each of the personal display systems depending on viewpoints of the users within the respective environment.

18. The computer-readable storage medium of claim 15, wherein the object in the environment in which the three-dimensional scanner is configured to provide measurements includes a mockup of a component of a complex system, and

wherein the apparatus is caused to communicate the output for display to the user or another user in an instance in which the environment of the user or other user includes an empty space of the complex system designed for placement of the component, a live or direct view of the empty space being augmented by the three-dimensional model.

19. The computer-readable storage medium of claim 15, wherein the one or more sensors further include a sensor configured to capture a gesture of the user indicating alteration to a portion of the three-dimensional model, and the sensed input includes the captured gesture captured by the sensor,

wherein the apparatus is caused to receive and process the sensed input including the gesture captured by the sensor, and form and communicate another input to cause the same or another one or more electronic resources to carry out the alteration indicated by the gesture, and

wherein the apparatus is caused to receive an output including at least the altered portion of the three-dimensional model, and communicate the output for display of the three-dimensional model including the altered portion.

20. The computer-readable storage medium of claim 15, wherein the object in the environment in which the three-dimensional scanner is configured to provide measurements includes a mockup of a component of a complex system, and

wherein the computer-readable storage medium has further computer-readable program code stored therein that, in response to execution by the processor, cause the apparatus to receive an output from an electronic resource simulation of operation of the complex system, and communicate the output for display by the display device.

21. The computer-readable storage medium of claim 15, wherein the apparatus is caused to communicate the output for display by a plurality of display devices of a respective plurality of users, the output for each of the display devices being varied depending on roles of the respective users.