Computer and computer system for controlling object manipulation in immersive virtual space

• US 9,721,396 B2
• Filed: 10/05/2016
• Issued: 08/01/2017
• Est. Priority Date: 03/17/2015
• Status: Active Grant

• Current Assignee: Colopl, Inc.
• Original Assignee: Colopl, Inc.
• Inventors: Baba, Naruatsu
• Primary Examiner(s): Cheng, Joe H
• Application Number:
  US15/286,084
• Publication Number:
  US 20170024935A1
• Time in Patent Office: 300 Days
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/25 Output arrangements for video game devices
• CPC Class Codes: A63F 13/26 having at least one additional display device, e.g. on the game controller or outside a game booth
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/525 Changing parameters of virtual cameras
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0308 comprising a plurality of distinctive and separately oriented light emitters or reflectors associated to the pointing device, e.g. remote cursor controller with distinct and separately oriented LEDs at the tip whose radiations are captured by a photo-detector associated to the screen
• CPC Class Codes: G06F 3/033 Pointing devices displaced or positioned by the user , e.g. mice, trackballs, pens or joysticks; Accessories therefor digitisers characterised by the transducing means G06F3/041
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/0383 Signal control means within the pointing device
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/04842 : Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/0487 : using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
• CPC Class Codes: G06F 3/0488 : using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04886 : by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: G06T 19/006 : Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 : Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts

Abstract

Action of a head around which an HMD main body is worn is related to manipulation of an object in an immersive three-dimensional virtual space. Information on a sight in a virtual space may be determined based on information on head inclination sensed with an inclination sensor; an image of the sight in the virtual space may be generated based on the sight information in order to display the sight image in the head mounted display; an object placed on a reference sight line in the virtual space may be identified, the reference sight line being determined in correspondence with a predetermined position in the sight image; and an object the identified object may be manipulated in response to the identification of the object in accordance with action of inclining in a predetermined direction the head around which the head mounted display is worn.

Claims

1. A computer comprising a CPU and a memory having a computer program stored therein wherein the computer program, when executed by the CPU causes the CPU to perform operations comprising:

determining information on a sight in a three-dimensional virtual space based on three-dimensional virtual space information stored in the memory and information on inclination sensed with an inclination sensor provided in a head mounted display connected to the computer;

generating an image of the sight in the three-dimensional virtual space based on the sight information in order to display the sight image in the head mounted display;

identifying an object in the generated sight image placed on a reference sight line in the generated sight image in the three-dimensional virtual space, the reference sight line being determined in correspondence with a predetermined position in the generated sight image; and

manipulating the identified object in the three-dimensional virtual space in accordance with action of inclining the head mounted display in a predetermined direction.

2. The computer according to claim 1, wherein the determining sight information based further on information on a position of the head mounted display that is sensed with a position sensor connected to the computer and capable of sensing the head mounted display.

3. The computer according to claim 1, wherein in a case where a plurality of objects are placed on the reference sight line, the identifying an object in the generated sight image comprises selecting an object closest to a point of view in the three-dimensional virtual space.

4. The computer according to claim 1, wherein the predetermined position is set at a central point of the sight image.

5. The computer according to claim 1, wherein the predetermined direction is a vertical direction.

6. The computer according to claim 1, wherein the manipulating the identified object in the three-dimensional virtual space comprises manipulating the identified object in cooperation with an external controller connectable to the computer.

7. The computer according to claim 6, wherein the external controller includes a touch display, and the manipulation performed on the identified object in cooperation with the external controller is performed in accordance with touch action including any of tapping, swiping, and holding performed on the touch display.

8. A computer system comprising a computer and a head mounted display connected to the computer and including an inclination sensor,

wherein the computer is configured to perform operations comprising:

determining information on a sight in a three-dimensional virtual space based on stored three-dimensional virtual space information and information on inclination sensed with the inclination sensor,

generating an image of the sight in the three-dimensional virtual space based on the sight information in order to display the sight image in the head mounted display,

identifying an object in the generated sight image placed on a reference sight line in the generated sight image in the three-dimensional virtual space, the reference sight line being determined in correspondence with a predetermined position in the generated sight image, and

manipulating the identified object in the three-dimensional virtual space in accordance with action of inclining the head mounted display in a predetermined direction.

9. The computer system according to claim 8, further comprising a position sensor connected to the computer and capable of sensing the head mounted display,

wherein the sight information is determined based further on information on a position of the head mounted display sensed with the position sensor.

10. The computer system according to claim 8, further comprising an external controller connectable to the computer,

wherein the manipulation performed on the identified object is performed in accordance with touch action performed on a touch display provided in the external controller.

11. A non-transitory computer-readable medium having a computer program stored therein, the computer program when read and executed by a processor causing the processor to perform operations comprising:

determining information on a sight in a three-dimensional virtual space based on stored three-dimensional virtual space information and information on sensed inclination of a head mounted display;

generating an image of the sight in the three-dimensional virtual space based on the sight information in order to display the sight image in the head mounted display;

identifying an object in the generated sight image placed on a reference sight line in the generated sight image in the three-dimensional virtual space, the reference sight line being determined in correspondence with a predetermined position in the generated sight image; and

manipulating the identified object in the three-dimensional virtual space in accordance with action of inclining the head mounted display in a predetermined direction.