WEARABLE HEAD-MOUNTED DISPLAY AND CAMERA SYSTEM WITH MULTIPLE MODES

• US 20150362733A1
• Filed: 06/12/2015
• Published: 12/17/2015
• Est. Priority Date: 06/13/2014
• Status: Abandoned Application

• Current Assignee: Augmented Reality Holdings, LLC
• Original Assignee: Zambala LLLP
• Inventors: Spivack, Nova T.
• Application Number:
  US14/738,182
• Publication Number:
  US 20150362733A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/26 having at least one additional display device, e.g. on the game controller or outside a game booth
• CPC Class Codes: A63F 13/53 involving additional visual information provided to the game scene, e.g. by overlay to simulate a head-up display [HUD] or displaying a laser sight in a shooting game
• CPC Class Codes: A63F 13/92 Video game devices specially adapted to be hand-held while playing
• CPC Class Codes: G02B 2027/0123 comprising devices increasing the field of view
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0181 Adaptation to the pilot/driver
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042

Abstract

Embodiments of the present disclosure include systems and methods for a wearable head-mounted display and camera system with multiple modes of user interaction including at in some embodiments a natural reality mode, an augmented reality mode, and a virtual reality mode.

Claims

1. A head-mounted device for providing multiple modes of user interaction, the head-mounted device comprising:

a display device configured to present visual information to the user; and

a user input device configured to receive an input from the user selecting one of a plurality of interaction modes, including at least a reality mode, an augmented reality mode, and a virtual reality mode;

wherein when in reality mode, the visual information includes a live video feed from an image capture device associated with the head-mounted display device, the image capture device configured to capture images of a physical environment;

wherein when in augmented reality mode, the visual information includes live video feed from the image capture device along with computer-generated simulated objects or computer image processing; and

wherein when in virtual reality mode, the visual information includes a computer-generated simulated environment.

2. The head-mounted device of claim 1, wherein the image capture device includes two forward facing cameras attached to the head-mounted device and the display device includes two display screens, and wherein the forward facing cameras and the two display screens are configured to provide the user with a stereoscopic view of the surrounding physical environment.

3. The head-mounted device of claim 2, wherein the image capture device further includes additional cameras configured to provide additional simultaneous viewing angles or a larger field of view.

4. The head-mounted device of claim 1, wherein the image capture device is remotely located from the head-mounted display device and in communication with the head-mounted display device via a wireless network connection.

5. The head-mounted device of claim 1, wherein the display device incorporates optical projection technology to project the visual information onto the retina of the user.

6. The head-mounted device of claim 1, wherein when in augmented reality mode, the live video feed from the image capture device is processed and adjusted to compensate for a visual impairment of the user.

7. The head-mounted device of claim 1, further comprising:

an audio output device configured to present audio information to the user;

wherein the audio information includes one or more of, an audio feed from a first audio input device or microphone or set of microphones configured to capture audio from the physical environment surrounding the head-mounted device, an audio feed from a second audio input device configured to capture audio at a remote location, a pre-recorded audio, or a computer-generated audio.

8. The head-mounted device of claim 1, further comprising:

a location sensor configured to gather location data including a current location of the head-mounted device in the physical environment;

a timer module configured to gather a time data including an absolute or relative time based on the current location of the head-mounted device; and

an object identifier module configured to detect and retrieve one or more simulated objects, based on the location data and/or the time data, the simulated objects for presentation via the display device when the head-mounted device is in augmented reality mode or virtual reality mode.

9. The head mounted device of claim 6, further comprising:

a network interface configured to receive simulated objects from one or more host servers, the simulated objects stored in a remote repository.

10. A system for providing multiple modes of user interaction via a head-mounted device, the system comprising:

a sensor unit configured to capture sensor data from the physical environment surrounding the head-mounted device;

a sensory output unit configured to present sensory output information to a user of the head-mounted device;

a user input unit configured to receive a user input;

a processor unit; and

a memory unit having instructions stored thereon, which when executed by the processor unit, cause the system to;

receive, via the user input unit, a user selection of a first, second, or third user interaction mode; and

if the user selection is the first user interaction mode;

capture a live sensor data feed via the sensor unit; and

present sensory information including the live sensor data feed via the sensory output unit;

if the user selection is the second user interaction mode;

capture a live sensor data feed via the sensor unit;

generate one or more simulated objects, the one or more simulated objects including user perceptible characteristics;

present sensory output information including the live sensor data feed and the one or more simulated objects via the sensory output unit; and

if the user selection is the third user interaction mode;

generate a simulated environment, the simulated environment including user perceptible characteristics; and

present sensory output information including the simulated environment via the sensory output unit.

11. The system of claim 10, wherein the first user interaction mode is a reality mode, the second user interaction mode is an augmented reality mode, and the third user interaction mode is a virtual reality mode.

12. The system of claim 10,

wherein the sensor unit includes one or more image capture devices;

wherein the sensor data includes video captured by the one or more image capture devices;

wherein the sensory output unit includes one or more displays;

wherein the sensory output information includes visual information configured for presentation via the one or more displays; and

wherein the one or more simulated objects include visual characteristics.

13. The system of claim 10,

wherein the sensor unit includes one or more audio capture devices;

wherein the sensor data includes audio captured by the one or more audio capture devices;

wherein the sensory output unit includes one or more speakers;

wherein the sensory output information includes audible information configured for presentation via the one or more speakers; and

wherein the one or more simulated objects include audible characteristics.

14. The system of claim 12, wherein the image capture unit includes two forward facing cameras and the display unit includes two display screens, and wherein the forward facing cameras and the two display screens are configured to provide the user with a stereoscopic view of the surrounding physical environment.

15. The system of claim 14, wherein the image capture unit further includes additional cameras configured to provide additional simultaneous viewing angles or a larger field of view.

16. The system of claim 12, wherein the display unit incorporates optical projection technology to project the visual information onto the retina of the user.

17. The system of claim 12, wherein the memory unit has further instructions stored thereon, which when executed by the processor unit, cause the system to further:

detect a vision impairment of the user; and

adjust presentation via the display unit to correct for the vision impairment.

18. The system of claim 10, further comprising:

a location sensor; and

an object identifier module;

wherein the memory unit has further instructions stored thereon, which when executed by the processor unit, cause the system to further;

gather, via the location sensor, a location data including a current location of the head-mounted device in the physical environment; and

detect and retrieve, via the object identifier module, one or more simulated objects based on the gathered location data.

19. The system of claim 10, further comprising:

a timer module; and

an object identifier module;

wherein the memory unit has further instructions stored thereon, which when executed by the processor unit, cause the system to further;

gather, via the timer module, a time data including an absolute or relative time based on the current location of the head-mounted device; and

detect and retrieve, via the object identifier module, one or more simulated objects based on the gathered time data.

20. The system of claim 10, further comprising:

a network interface;

wherein the memory unit has further instructions stored thereon, which when executed by the processor unit, cause the system to further;

receive simulated objects from one or more host servers, the simulated objects stored in a remote repository.

21. A method for providing multiple modes of user interaction via a head-mounted device, the head-mounted device associated with a sensor unit configured to capture sensor data from the physical environment surrounding the head-mounted device, a sensory output unit configured to present sensory output information to a user of the head-mounted device, and a user input unit configured to receive a user input, the method comprising:

receiving, via the user input unit, a user selection of a first, second, or third user interaction mode; and

if the user selection is the first user interaction mode;

capturing a live sensor data feed via the sensor unit; and

presenting sensory output information including the live sensor data feed via the sensory output unit;

if the user selection is the second user interaction mode;

capturing a live sensor data feed via the sensor unit;

generating one or more simulated objects, the one or more simulated objects including user perceptible characteristics;

presenting sensory output information including the live sensor data feed with the one or more simulated objects via the sensory output unit; and

if the user selection is the third user interaction mode;

generating a simulated environment, the simulated environment including user perceptible characteristics; and

presenting sensory output information including the simulated environment via the sensory output unit.

22. The method of claim 21, wherein the first user interaction mode is a reality mode, the second user interaction mode is an augmented reality mode, and the third user interaction mode is a virtual reality mode.

23. The method of claim 21,

wherein the sensor unit includes one or more image capture devices;

wherein the sensor data includes video captured by the one or more image capture devices;

wherein the sensory output unit includes one or more displays;

wherein the sensory output information includes visual information configured for presentation via the one or more displays; and

wherein the one or more simulated objects include visual characteristics.

24. The system of claim 21,

wherein the sensor unit includes one or more audio capture devices;

wherein the sensor data includes audio captured by the one or more audio capture devices;

wherein the sensory output unit includes one or more speakers;

wherein the sensory output information includes audible information configured for presentation via the one or more speakers; and

wherein the one or more simulated objects include audible characteristics.

25. The method of claim 23, further comprising:

detecting a vision impairment of the user; and

adjusting presentation via the display unit to correct for the vision impairment.

26. The method of claim 21, wherein the head-mounted device is further associated with a location sensor and an object identifier module, the method further comprising:

gathering, via the location sensor, a location data including a current location of the head-mounted device in the physical environment; and

detecting and retrieving, via the object identifier module, one or more simulated objects based on the gathered location data.

27. The method of claim 21, wherein the head-mounted device is further associated with a timer module and an object identifier module, the method further comprising:

gathering, via the timer module, a time data including an absolute or relative time; and

detecting and retrieving, via the object identifier module, one or more simulated objects based on the gathered time data.

28. The method of claim 21, wherein the head-mounted device is further associated with a network interface, the method further comprising:

receiving, via the network interface, simulated objects from one or more host servers, the simulated objects stored in a remote repository.