Reality mixer for mixed reality

• US 10,169,922 B2
• Filed: 10/21/2016
• Issued: 01/01/2019
• Est. Priority Date: 02/16/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Kipman, Alex, Rao, Purnima M., Haruyama, Rebecca, Chiu, Shih-Sang Carnaven, Mayhew, Stuart, Murillo, Oscar E., Costa, Carlos Fernando Faria
• Primary Examiner(s): CRADDOCK, ROBERT J
• Application Number:
  US15/331,746
• Publication Number:
  US 20170236332A1
• Time in Patent Office: 802 Days
• Field of Search: 345633
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0147 comprising a device modifying the resolution of the displayed image
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0362 with detection of 1D translations or rotations of an operating part of the device, e.g. scroll wheels, sliders, knobs, rollers or belts
• CPC Class Codes: G06F 3/041 Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A mixed-reality display device comprises an input system, a display, and a graphics processor. The input system is configured to receive a parameter value, the parameter value being one of a plurality of values of a predetermined range receivable by the input system. The display is configured to display virtual image content that adds an augmentation to a real-world environment viewed by a user of the mixed-reality display device. The graphics processor is coupled operatively to the input system and to the display; it is configured to render the virtual image content so as to variably change the augmentation, to variably change a perceived realism of the real world environment in correlation to the parameter value.

Claims

1. A mixed-reality display device comprising:

an input system configured to receive a parameter value, the parameter value being one of a plurality of values of a predetermined range receivable by the input system;

an electronic display configured to display virtual image content that adds an augmentation to a real-world environment viewed by a user of the mixed reality display device, the virtual image content being positioned persistently relative to the real-world environment, the display being configured to superpose the virtual image content on the real-world environment, so as to impart a variable unrealistic quality to the real-world environment; and

a graphics processor coupled operatively to the input system and to the electronic display, the graphics processor being configured to render the virtual image content so as to variably change the augmentation, to variably change a perceived realism of the real world environment in correlation to the parameter value.

2. The display device of claim 1 further comprising a user-interface component configured to present on the display device a user-interface element to enable selection of the parameter value.

3. The display device of claim 1 wherein the input system includes one or more of a camera coupled operatively to gesture-recognition componentry and a microphone coupled operatively to voice-recognition componentry.

4. The display device of claim 1 wherein the input system includes a touch sensor.

5. The display device of claim 1 wherein the virtual image content is displayed at a predetermined location of a fixed object sightable by a user of the display device, and wherein the input system is further configured to enable selection of the object from among a plurality of objects sighted by the user.

6. The display device of claim 1 wherein the parameter value is one of three or more values within the range.

7. The display device of claim 1 wherein the display is configured to project a secondary image of the object, in its entirety, and wherein the secondary image is sightable by the user via the display device.

8. The display device of claim 7 wherein the virtual image content is displayed at a predetermined location of an object sightable by a user of the display device, wherein the input system includes a world-facing camera configured to acquire a primary image of the object, and wherein the primary image is provided to the graphics processor.

9. The display device of claim 1, wherein the virtual image content is displayed at a predetermined location of a moving object in the real-world environment, and wherein the display is configured to superpose the virtual image content on the moving object, so as to impart a variable unrealistic quality to the moving object.

10. The display device of claim 1 further comprising:

a loudspeaker configured to emit audio; and

audio-mixing componentry coupled operatively to the input system and to the loudspeaker, the audio-mixing componentry being configured to compose an audio overlay to impart a variable unrealistic quality to real-world environment, the variable unrealistic quality varying in relation to the parameter value.

11. The display device of claim 1 wherein the augmentation includes one or more of color, texture, lighting, shadow, opacity, blur, and pixelization of a foreground or background object within the real-world environment.

12. The display device of claim 1 wherein the augmentation is configured to change an aesthetic presentation of an object within the real-world environment, and wherein the parameter value ranges from real aesthetic presentation to fantastical aesthetic presentation.

13. The display device of claim 1 wherein the augmentation is configured to change a virtualized mass and associated kinematics of an object within the real-world environment, and wherein the parameter value ranges from realistic virtualized mass to unrealistic virtualized mass.

14. The display device of claim 1 wherein the augmentation is configured to temporally evolve an appearance of an object within the real-world environment, and wherein the parameter value ranges from a past time to a future time.

15. The display device of claim 1 wherein the augmentation is configured to alter a relational aspect of an object relative to one or more nearby objects in the real-world environment.

16. A fully immersive, binocular, near-eye display device, comprising:

an input system configured to receive a parameter value, the parameter value being one of a plurality of values of a predetermined range receivable by the input system;

an electronic microdisplay configured to position virtual image content persistently relative to a location of an object in a virtual environment, to form an image of the object sightable by a user of the near-eye display device, the virtual image content adding a superposed augmentation to the object; and

a graphics processor coupled operatively to the input system and to the microdisplay, the graphics processor being configured to render the virtual image content so as to variably change the augmentation, to variably change a perceived realism of the object in correlation to the parameter value.

17. The display device of claim 16 wherein the input system includes a world-facing depth-imaging camera configured to acquire a depth image of the object, and wherein the depth image is provided to the graphics processor.

18. A near-eye display device comprising:

an input system configured to receive a parameter value, the parameter value being one of a plurality of values of a predetermined range receivable by the input system;

an electronic microdisplay configured to display virtual image content to a predetermined location sightable by a user of the near-eye display device, the virtual image content being displayed at an observer-invariant location of a fixed or moving real-world object in a real-world environment of the user, wherein the display is configured to superpose the virtual image content on the object, so as to impart a variable unrealistic quality to the object;

a loudspeaker configured to emit audio at the predetermined location;

a graphics processor coupled operatively to the input system and to the microdisplay, the graphics processor being configured to render the virtual image content so as to variably change the augmentation in correlation to the parameter value; and

audio-mixing componentry coupled operatively to the input system and to the loudspeaker, the audio-mixing componentry being configured to compose an audio overlay to impart the variable virtual quality in correlation to the parameter value.

19. The display device of claim 18 wherein the augmentation is configured to change one or more of loudness and pitch emitted by an object in the predetermined location, the display device further comprising a position sensor configured to sense a position and orientation of the display device relative to the object.

20. The display device of claim 18 wherein the augmentation is configured to change background imagery in the predetermined location.