Information processing device, storage medium storing information processing program, information processing system, and information processing method

• US 9,824,293 B2
• Filed: 12/07/2015
• Issued: 11/21/2017
• Est. Priority Date: 02/10/2015
• Status: Active Grant

• Current Assignee: Nintendo Company Limited
• Original Assignee: Nintendo Company Limited
• Inventors: Aoyagi, Norihiro
• Primary Examiner(s): Azarian, Seyed
• Application Number:
  US14/961,210
• Publication Number:
  US 20160232673A1
• Time in Patent Office: 715 Days
• Field of Search: 382100, 382103, 382106-107, 382154, 382162, 382168, 382173, 382181, 382189, 382194, 382203, 382209, 382232, 382254, 382274, 382276, 382286-291, 382305, 382312, 345426, 345158, 345157, 345156, 463 37
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: A63F 13/92 Video game devices specially adapted to be hand-held while playing
• CPC Class Codes: G06F 1/1626 with a single-body enclosure integrating a flat display, e.g. Personal Digital Assistants [PDAs]
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06V 10/507 Summing image-intensity values; Histogram projection analysis
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

An example information processing system performs a process based on a captured image captured by an image-capturing device. The information processing system obtains information regarding brightness of each pixel based on the captured image. The information processing device calculates position information of the captured image, the position information representing deviation obtained with the brightness of each pixel being used as a weight. The position information is calculated for a predetermined area of the captured image. The information processing device performs a predetermined information process based on the position information.

Claims

1. A information processing system for performing a process, the information processing system comprising:

a camera configured to obtain a camera image;

one or more processor configured to execute;

obtaining information regarding brightness of each pixel based on the camera image;

calculating position information of the camera image, the position information representing deviation obtained with the brightness of each pixel being used as a weight; and

performing a predetermined information process based on the position information; and

a display configured to output an image based on results of the performed predetermined information process;

wherein;

the processor calculates, as the position information, a center-of-gravity position of a whole area or a predetermined partial area of the camera image, in which the brightness of each pixel is used as a weight; and

for a whole area or a predetermined partial area of the camera image, the processor obtains the center-of-gravity position by calculating, for each pixel, a coordinate value obtained by multiplying coordinates representing a position of the pixel by the brightness value of the pixel, and dividing a total sum of the calculated coordinate values by a total sum of the brightness values of the pixels.

2. The information processing system according to claim 1, wherein the processor calculates information representing a change in a position of an object included in the camera image based on the position information, and performs an information process based on the information.

3. The information processing system according to claim 1, wherein:

the camera is an infrared camera; and

the processor obtains information representing a brightness value of the infrared image.

4. The information processing system according to claim 1, wherein the processor calculates information representing a shape of an object included in the camera image based on the position information, and performs an information process based on the information.

5. The information processing system according to claim 4, wherein the processor sets a determination range on the camera image based on a position on the camera image specified by the position information, and calculates the shape of the object based on a relationship between the determination range and an area of an image of the object.

6. The information processing system according to claim 1, wherein:

the processor further calculates brightness of the camera image; and

the processor performs the information process based on the position information and the brightness.

7. The information processing system according to claim 6, wherein the processor calculates information representing a two-dimensional position or movement with respect to a direction generally perpendicular to the image-capturing direction based on the position information, and performs the information process in accordance with the information.

8. The information processing system according to claim 6, wherein the processor calculates information representing a two-dimensional position and movement with respect to a direction generally perpendicular to the image-capturing direction based on the position information, and performs the information process in accordance with the information.

9. The information processing system according to claim 1, wherein:

the processor calculates brightness of the camera image; and

the processor calculates information representing a position or movement with respect to an image-capturing direction of the camera based on the brightness, and performs the information process in accordance with the information.

10. The information processing system according to claim 9, wherein the processor calculates information representing a two-dimensional position or movement with respect to a direction generally perpendicular to the image-capturing direction based on the position information, and performs the information process in accordance with the information.

11. The information processing system according to claim 9, wherein the processor calculates information representing a two-dimensional position and movement with respect to a direction generally perpendicular to the image-capturing direction based on the position information, and performs the information process in accordance with the information.

12. The information processing system according to claim 1, wherein the processor calculates position information corresponding to a target area of the camera image, which is set independently of an area of an image of an object included in the camera image.

13. The information processing system according to claim 1, wherein a game process is performed as the predetermined information process based on the position information, and the image outputted based on results of the performed predetermined information process is a game image.

14. The information processing system according to claim 1, wherein:

the processor calculates brightness of the camera image; and

the processor calculates information representing a position and movement with respect to an image-capturing direction of the camera based on the brightness, and performs the information process in accordance with the information.

15. A non-transitory computer-readable storage medium storing an information process program to be performed by a computer of an information processing system for performing a process based on a camera image obtained by a camera, the information process program instructing the computer to execute:

obtaining information regarding brightness of each pixel based on the camera image;

calculating position information of the camera image, the position information representing deviation obtained with the brightness of each pixel being used as a weight;

performing a predetermined information process based on the position information; and

outputting an image based on results of the performed predetermined information process, wherein;

a center-of-gravity position of a whole area or a predetermined partial area of the camera image is calculated as the position information, in which the brightness of each pixel is used as a weight; and

for a whole area or a predetermined partial area of the camera image, the center-of-gravity position is obtained by calculating, for each pixel, a coordinate value obtained by multiplying coordinates representing a position of the pixel by the brightness value of the pixel, and dividing a total sum of the calculated coordinate values by a total sum of the brightness values of the pixels.

16. The non-transitory computer-readable storage medium according to claim 15, wherein a game process is performed as the predetermined information process based on the position information, and the image outputted based on results of the performed predetermined information process is a game image.

17. An information processing device for performing a process, the information processing device comprising:

a camera configured to obtain a camera image;

one or more processor at least configured to;

obtain information regarding brightness of each pixel based on the camera image;

calculate position information of the camera image, the position information representing deviation obtained with the brightness of each pixel being used as a weight; and

perform a predetermined information process based on the position information; and

a display configured to output an image based on results of the performed predetermined information process, wherein;

a center-of-gravity position of a whole area or a predetermined partial area of the camera image is calculated as the position information, in which the brightness of each pixel is used as a weight; and

for a whole area or a predetermined partial area of the camera image, the center-of-gravity position is obtained by calculating, for each pixel, a coordinate value obtained by multiplying coordinates representing a position of the pixel by the brightness value of the pixel, and dividing a total sum of the calculated coordinate values by a total sum of the brightness values of the pixels.

18. The information processing device according to claim 17, wherein a game process is performed as the predetermined information process based on the position information, and the image outputted based on results of the performed predetermined information process is a game image.

19. An information processing method to be performed on an information processing system for performing a process based on a camera image obtained by a camera, the method comprising:

obtaining information regarding brightness of each pixel based on the camera image;

calculating position information of the camera image, the position information representing deviation obtained with the brightness of each pixel being used as a weight;

performing a predetermined information process based on the position information; and

outputting an image based on results of the performed predetermined information process, wherein;

a center-of-gravity position of a whole area or a predetermined partial area of the camera image is calculated as the position information, in which the brightness of each pixel is used as a weight; and

for a whole area or a predetermined partial area of the camera image, the center-of-gravity position is obtained by calculating, for each pixel, a coordinate value obtained by multiplying coordinates representing a position of the pixel by the brightness value of the pixel, and dividing a total sum of the calculated coordinate values by a total sum of the brightness values of the pixels.

20. The method according to claim 19, wherein a game process is performed as the predetermined information process based on the position information, and the image outputted based on results of the performed predetermined information process is a game image.