Information processing device, storage medium storing information processing program, information processing system, and information processing method

• US 10,025,975 B2
• Filed: 01/20/2016
• Issued: 07/17/2018
• Est. Priority Date: 02/10/2015
• Status: Active Grant

• Current Assignee: Nintendo Company Limited
• Original Assignee: Nintendo Company Limited
• Inventors: Ogasawara, Yoshiyasu, Osawa, Kazuyoshi
• Primary Examiner(s): Yentrapati, Avinash
• Application Number:
  US15/001,517
• Publication Number:
  US 20160232675A1
• Time in Patent Office: 909 Days
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/92 Video game devices specially adapted to be hand-held while playing
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06V 40/113 Recognition of static hand signs
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

An example information processing device includes a housing and a camera for capturing an image in a side surface direction of the housing. The information processing device calculates, based on a camera image obtained by the camera, a three-dimensional position of an object (e.g., a hand of a user) included in the camera image, and performs an information process in accordance with the three-dimensional position. Note that the information processing device may specify a shape of the object based on the camera image to perform an information process in accordance with the three-dimensional position and the shape of the object. Then, it is possible to realize various inputs using a captured image.

Claims

1. An information processing system comprising:

a housing having at least a front surface and side surfaces;

a camera for capturing an image in a side surface direction of the housing; and

,

one or more computer processors for executing instructions such that the information processing system is at least configured to;

calculate, based on a camera image obtained by the camera, a two-dimensional position of the object on the camera image and calculate a three-dimensional position of an object included in the camera image based on the two-dimensional position;

specify a shape of an image area of the object on the camera image based on the camera image based on the calculated two-dimensional position, the shape of the image area of the object being a two-dimensional shape;

perform an information process in accordance with the three-dimensional position and the shape of the image area of the object,

determine a determination range based on the position of the three-dimensional position of the object, wherein a size of the determination range is variable; and

determine the shape of the image area of the object based on a number of areas of the object outside of the determination range and on the camera image;

wherein

the image area of the object is the entirety of an area of the object on the camera image; and

the areas of the object outside of the determination range are partial areas of the entirety of the area of the object and are areas separate from each other.

2. The information processing system according to claim 1, wherein the information processing system, including the one or more processors, is further configured to specify a position in a three-dimensional virtual space based on the three-dimensional position to perform, for the specified position, a different information process depending on the shape of the image area of the object.

3. The information processing system according to claim 1, wherein the information processing system, including the one or more processors, is further configured to:

calculate the position of the object based on the image area of the object.

4. The information processing system according to claim 1, wherein:

the housing can be held in one hand of a user; and

the information processing system, including the one or more processors, is further configured to calculate a three-dimensional position of the other hand of the user included in the camera image.

5. The information processing system according to claim 1, wherein the information processing system, including the one or more processors, is further configured to perform an information process for specifying a position in a three-dimensional virtual space based on the three-dimensional position.

6. The information processing system according to claim 1, wherein the information processing system, including the one or more processors, is further configured to move an object in a three-dimensional virtual space based on the three-dimensional position.

7. The information processing system according to claim 1, further comprising

wherein the information processing system, including the one or more processors, is further configured to perform an information process for displaying, on a display device, an image in accordance with the three-dimensional position.

8. The information processing system according to claim 7, wherein:

the camera captures an image in a side surface direction with respect to a left side or a right side of the housing;

the information processing system, including the one or more processors, is further configured to calculate a three-dimensional position, the three-dimensional position including a first position with respect to a first direction that is parallel to the side surface direction, a second position with respect to a second direction that is perpendicular to the side surface direction and is perpendicular to a screen of the display device, and a third position with respect to a third direction that is perpendicular to the side surface direction and is parallel to the screen of the display device; and

the information processing system, including the one or more processors, is further configured to perform a process with respect to a direction in the three-dimensional virtual space corresponding to a left-right direction on the screen of the display device based on the first position, performs a process with respect to a direction in the three-dimensional virtual space corresponding to a depth direction on the screen of the display device based on the second position, and performs a process with respect to a direction in the three-dimensional virtual space corresponding to an up-down direction on the screen of the display device based on the third position.

9. A non-transitory computer-readable storage medium storing an information processing program to be performed by a computer of an information processing system including an information processing device comprising a housing having at least a front surface and side surfaces, and a camera for capturing an image in a side surface direction of the housing, the information processing program instructing the computer to execute:

calculate, based on a camera image obtained by the camera, a two-dimensional position of the object on the camera image and calculate a three-dimensional position of an object included in the camera image based on the two-dimensional position;

specify a shape of an image area of the object on the camera image based on the camera image based on the calculated two-dimensional position, the shape of the image area of the object being a two-dimensional shape, and

perform an information process in accordance with the three-dimensional position and the shape of the image area of the object;

determine a determination range based on the position of the three-dimensional position of the object, wherein a size of the determination range is variable; and

determine the shape of the image area of the object based on a number of areas of the object outside of the determination range and on the camera image;

wherein

the image area of the object is the entirety of an area of the object on the camera image; and

the areas of the object outside of the determination range are partial areas of the entirety of the area of the object and are areas separate from each other.

10. An information processing method to be performed on an information processing system including an information processing device comprising a housing having at least a front surface and side surfaces, and a camera for capturing an image in a side surface direction of the housing, the method comprising:

calculating, based on a camera image obtained by the camera, a two-dimensional position of the object on the camera image and calculate a three-dimensional position of an object included in the camera image based on the two-dimensional position;

specifying a shape of an image area of the object on the camera image based on the camera image based on the calculated two-dimensional position, the shape of the image area of the object being a two-dimensional shape; and

performing, an information process in accordance with the three-dimensional position and the shape of the image area of the object;

determine a determination range based on the position of the three-dimensional position of the object, wherein a size of the determination range is variable; and

determine the shape of the image area of the object based on a number of areas of the object outside of the determination range and on the camera image;

wherein;

the image area of the object is the entirety of an area of the object on the camera image; and

the areas of the object outside of the determination range are partial areas of the entirety of the area of the object and are areas separate from each other.