Computer-readable storage medium, image processing apparatus, image processing system, and image processing method

• US 8,718,325 B2
• Filed: 07/29/2011
• Issued: 05/06/2014
• Est. Priority Date: 02/24/2011
• Status: Active Grant

• Current Assignee: Nintendo Company Limited
• Original Assignee: Nintendo Company Limited
• Inventors: Osako, Satoru
• Primary Examiner(s): Abdi, Amara
• Application Number:
  US13/193,895
• Publication Number:
  US 20120219178A1
• Time in Patent Office: 1,012 Days
• Field of Search: 382/103, 345/158, 345/560
• US Class Current: 382/103
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51

Abstract

A position of a predetermined object or a predetermined design is sequentially detected from images. Then, an amount of movement of the predetermined object or the predetermined design is calculated on the basis of: a position, in a first image, of the predetermined object or the predetermined design detected from the first image; and a position, in a second image, of the predetermined object or the predetermined design detected from the second image acquired before the first image. Then, when the amount of movement is less than a first threshold, the position, in the first image, of the predetermined object or the predetermined design detected from the first image is corrected to the position, in the second image, of the predetermined object or the predetermined design detected from the second image.

Claims

1. A non-transitory computer-readable storage medium having stored thereon an image processing program which, when executed by a computer of an information processing apparatus, causes the computer to perform operations comprising:

sequentially acquiring images;

sequentially detecting a position of a predetermined object or a predetermined design from the images;

calculating an amount of movement of the predetermined object or the predetermined design on the basis of;

a position, in a first image, of the predetermined object or the predetermined design detected from the first image; and

a position, in a second image, of the predetermined object or the predetermined design detected from the second image that has been acquired before the first image;

comparing the amount of movement with a first threshold;

when the amount of movement is less than the first threshold, correcting the position, in the first image, of the predetermined object or the predetermined design detected from the first image, to the position, in the second image, of the predetermined object or the predetermined design detected from the second image;

on the basis of the corrected position, setting a virtual camera placed in a virtual space; and

displaying on a display device a virtual space image obtained by capturing the virtual space with the virtual camera,

wherein

when the amount of movement is equal to or greater than the first threshold, and is less than a second threshold that is greater than the first threshold, the position, in the first image, of the predetermined object or the predetermined design detected from the first image, is corrected to a position internally dividing, in a predetermined ratio, a line segment connecting;

a position, in the first image, for the predetermined object or the predetermined design detected from the first image;

to a corresponding position, in the second image, for the predetermined object or the predetermined design detected from the second image.

2. The computer-readable storage medium according to claim 1, wherein

when the amount of movement is equal to or greater than the second threshold, setting the virtual camera on the basis of the position, in the first image, of the predetermined object or the predetermined design detected from the first image.

3. The computer-readable storage medium according to claim 1, the image processing program causing the computer to perform operations further comprising:

sequentially calculating motion vectors from the images, the motion vectors each indicating a moving direction of the predetermined object or the predetermined design; and

changing the predetermined ratio on the basis of the motion vectors.

4. The computer-readable storage medium according to claim 3, wherein the changing the predetermined ratio

includes determining, on the basis of the plurality of motion vectors, whether or not the predetermined object or the predetermined design is continuously moving in a constant direction, and changes the predetermined ratio in accordance with a determination result of the determining.

5. An image processing apparatus comprising:

a processing system comprising at least one processor, the processing system configured to perform operations comprising;

sequentially acquiring images;

sequentially detecting a position of a predetermined object or a predetermined design from the images;

calculating an amount of movement of the predetermined object or the predetermined design on the basis of;

a position, in a first image, of the predetermined object or the predetermined design detected from the first image; and

a position, in a second image, of the predetermined object or the predetermined design detected from the second image that has been acquired before the first image;

comparing the amount of movement with a first threshold;

when the amount of movement is less than the first threshold, correcting the position, in the first image, of the predetermined object or the predetermined design detected from the first image, to the position, in the second image, of the predetermined object or the predetermined design detected from the second image;

on the basis of the position, setting a virtual camera placed in a virtual space; and

displaying on a display device a virtual space image obtained by capturing the virtual space with the virtual camera, wherein

when the amount of movement is equal to or greater than the first threshold, and is less than a second threshold that is greater than the first threshold, the position, in the first image, of the predetermined object or the predetermined design detected from the first image, is corrected to a position internally dividing, in a predetermined ratio, a line segment connecting;

a position, in the first image, for the predetermined object or the predetermined design detected from the first image;

to a corresponding position, in the second image, for the predetermined object or the predetermined design detected from the second image.

6. An image processing method comprising:

sequentially acquiring images;

sequentially detecting a position of a predetermined object or a predetermined design from the images;

calculating an amount of movement of the predetermined object or the predetermined design on the basis of;

a position, in a first image, of the predetermined object or the predetermined design detected from the first image; and

a position, in a second image, of the predetermined object or the predetermined design detected from the second image that has been acquired before the first image;

comparing the amount of movement with a first threshold;

when the amount of movement is less than the first threshold, correcting the position, in the first image, of the predetermined object or the predetermined design detected from the first image, to the position, in the second image, of the predetermined object or the predetermined design detected from the second image;

on the basis of the position corrected in the correction step, setting a virtual camera placed in a virtual space; and

displaying on a display device a virtual space image obtained by capturing the virtual space with the virtual camera, wherein

when the amount of movement is equal to or greater than the first threshold, and is less than a second threshold that is greater than the first threshold, the position, in the first image, of the predetermined object or the predetermined design detected from the first image, is corrected to a position internally dividing, in a predetermined ratio, a line segment connecting;

a position, in the first image, for the predetermined object or the predetermined design detected from the first image;

to a corresponding position, in the second image, for the predetermined object or the predetermined design detected from the second image.

7. An image processing system comprising:

a camera; and

a processing system including at least one processor, the processing system configured to perform operations comprising;

sequentially acquiring images;

sequentially detecting a position of a predetermined object or a predetermined design from the images;

calculating an amount of movement of the predetermined object or the predetermined design on the basis of;

a position, in a first image, of the predetermined object or the predetermined design detected from the first image; and

a position, in a second image, of the predetermined object or the predetermined design detected from the second image that has been acquired before the first image;

comparing the amount of movement with a first threshold;

when the amount of movement is less than the first threshold, correcting the position, in the first image, of the predetermined object or the predetermined design detected from the first image, to the position, in the second image, of the predetermined object or the predetermined design detected from the second image;

on the basis of the corrected position, setting a virtual camera placed in a virtual space; and

displaying on a display device a virtual space image obtained by capturing the virtual space with the virtual camera

when the amount of movement is equal to or greater than the first threshold, and is less than a second threshold that is greater than the first threshold, the position, in the first image, of the object or the design detected from the first image, is corrected to a position internally dividing, in a predetermined ratio, a line segment connecting;

a position, in the first image, for the object or the design detected from the first image;

to a corresponding position, in the second image, for the object or the design detected from the second image.

8. An image processing system including an image processing apparatus and a marker in which a design is drawn, the image processing apparatus comprising:

a capturing section for capturing the marker;

a processing system including at least one processor, the processing system configured to perform operations comprising;

sequentially acquiring images from the capturing section;

sequentially detecting a position of a marker or a design from the images;

calculating an amount of movement of the marker or the design on the basis of;

a position, in a first image, of the marker or the design detected from the first image; and

a position, in a second image, of the marker or the design detected from the second image that has been acquired before the first image;

comparing the amount of movement with a first threshold;

when the amount of movement is less than the first threshold, correcting the position, in the first image, of the marker or the design detected from the first image, to the position, in the second image, of the marker or the design detected from the second image;

on the basis of the corrected position, setting a virtual camera placed in a virtual space; and

displaying on a display device a virtual space image obtained by capturing the virtual space with the virtual camera, wherein

when the amount of movement is equal to or greater than the first threshold, and is less than a second threshold that is greater than the first threshold, the position, in the first image, of the marker or the design detected from the first image, is corrected to a position internally dividing, in a predetermined ratio, a line segment connecting;

a position, in the first image, for the marker or the design detected from the first image;

to a corresponding position, in the second image, for the marker or the design detected from the second image.