Head mounted display, mobile information terminal, image processing apparatus, display control program, display control method, and display system

• US 10,620,699 B2
• Filed: 06/21/2019
• Issued: 04/14/2020
• Est. Priority Date: 10/22/2014
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Inventors: Kurihara, Yosuke
• Primary Examiner(s): Regn, Mark W
• Application Number:
  US16/448,434
• Publication Number:
  US 20190310703A1
• Time in Patent Office: 298 Days
• CPC Class Codes: A63F 13/214 for locating contacts on a surface, e.g. floor mats or touch pads
• CPC Class Codes: A63F 13/24 Constructional details thereof, e.g. game controllers with detachable joystick handles
• CPC Class Codes: A63F 13/25 Output arrangements for video game devices
• CPC Class Codes: A63F 13/426 involving on-screen location information, e.g. screen coordinates of an area at which the player is aiming with a light gun
• CPC Class Codes: A63F 13/533 for prompting the player, e.g. by displaying a game menu
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 1/169 the I/O peripheral being an integrated pointing device, e.g. trackball in the palm rest area, mini-joystick integrated between keyboard keys, touch pads or touch stripes G06F1/1643 takes precedence; constructional details of pointing devices G06F3/033
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/0412 Digitisers structurally integrated in a display
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 : for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/147 : using display panels
• CPC Class Codes: G06V 40/103 : Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G09G 2354/00 : Aspects of interface with display user
• CPC Class Codes: G09G 3/003 : to produce spatial visual effects
• CPC Class Codes: G09G 5/00 : Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators image data processing or generation, in general G06T

Abstract

Methods and apparatus provide for: acquiring data for, and generating, images based on: (i) input operations received from a hand-held input device, and (ii) at least one of a position and attitude of the input device; and displaying the images on a display device disposed on an inner surface of a head-mounted display worn by the user on the head.

Claims

1. A head mounted display system comprising:

a head-mounted display wearable on the head of a user;

a display device disposed on an inner surface of the head-mounted display so as to be seen by the user when the head-mounted display is worn;

a communication section receiving input operations from the user by way of a hand-held input device;

a display controlling unit configured to acquire data for, and generate, images based on;

(i) input operations which the communication section receives from the input device, and (ii) position and attitude of the input device, wherein the display controlling unit causes the display device to display the images;

an input apparatus disposed on an outer surface of the head-mounted display, and being operable to detect contact with or proximity of a body part of the user;

a body section connected detachably to a mobile information terminal, which displays images generated by the display controlling unit,

wherein the display controlling unit generates the images also based on the input operations of contact with or proximity of a body part of the user detected by the input apparatus.

2. The head-mounted display system according to claim 1, wherein:

the input apparatus constitutes a first input unit which receives the input from the user;

the first input unit includes at least one button or sensor to detect the input from the user; and

the display controlling unit generates the images also based on the input from the user received by the first input unit.

3. The head-mounted display system according to claim 2, further comprising:

a second input unit which is different from the first input unit;

wherein the second input unit includes at least one button or sensor to detect the input from the user; and

wherein the display controlling unit generates the images also based on the input from the user received by the second input unit.

4. The head-mounted display system according to claim 3, wherein the second input unit is part of the input device.

5. The head-mounted display system according to claim 3, wherein the first input unit has at least one function in common with the second input unit.

6. A head mounted display wearable on the head of a user, the head-mounted display comprising:

a display device disposed on an inner surface of the head-mounted display so as to be seen by the user when the head-mounted display is worn; and

a display controlling unit configured to acquire data for, and generate, images based on;

(i) input operations received from a hand-held input device, and (ii) position and attitude of the input device, wherein the display controlling unit causes the display device to display the images;

an input apparatus disposed on an outer surface of the head-mounted display, and being operable to detect contact with or proximity of a body part of the user;

a body section connected detachably to a mobile information terminal, which displays images generated by the display controlling unit,

wherein the display controlling unit generates the images also based on the input operations of contact with or proximity of a body part of the user detected by the input apparatus.

7. The head-mounted display according to claim 6, wherein:

the input apparatus constitutes a first input unit which receives the input from the user;

the first input unit includes at least one button or sensor to detect the input from the user; and

the display controlling unit generates the images also based on the input from the user received by the first input unit.

8. The head-mounted display according to claim 7, further comprising:

a second input unit which is different from the first input unit;

wherein the second input unit includes at least one button or sensor to detect the input from the user; and

wherein the display controlling unit generates the images also based on the input from the user received by the second input unit.

9. The head-mounted display according to claim 8, wherein the second input unit is part of the input device.

10. The head-mounted display according to claim 8, wherein the first input unit has at least one function in common with the second input unit.

11. A display control method comprising:

acquiring data for, and generating, images based on;

(i) input operations received from a hand-held input device, and (ii) position and attitude of the input device;

displaying the images on a display device disposed on an inner surface of a head-mounted display worn by the user on the head;

detecting contact with or proximity of a body part of the user via an input apparatus; and

generating the images also based on the input operations of contact with or proximity of a body part of the user detected by the input apparatus.