IMAGE PROCESSING DEVICE, IMAGE PROCESSING SYSTEM, AND IMAGE PROCESSING METHOD

• US 20140362188A1
• Filed: 05/22/2014
• Published: 12/11/2014
• Est. Priority Date: 06/07/2013
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Incorporated (Sony Group Corp.)
• Inventors: Yokokawa, Yutaka, Doucet, Nicolas, Noda, Masafumi, Miyamae, Yuki
• Granted Patent: US 10,293,252 B2
• Time in Patent Office: Days
• US Class Current: 348/47
• CPC Class Codes: A63F 13/00 Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/42 by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
• CPC Class Codes: A63F 13/5372 for tagging characters, objects or locations in the game scene, e.g. displaying a circle under the character controlled by the player
• CPC Class Codes: A63F 13/843 involving concurrently two or more players on the same game device, e.g. requiring the use of a plurality of controllers or of a specific view of game data for each player
• CPC Class Codes: A63F 2009/2435 using a video camera
• CPC Class Codes: A63F 2300/10 characterized by input arrangements for converting player-generated signals into game device control signals
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/65 for computing the condition of a game character
• CPC Class Codes: A63F 9/24 Electric games; Games using electronic circuits not otherwise provided for video games A63F13/00; computerized gaming systems G07F17/32
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06F 3/13 Digital output to plotter ; Cooperation and interconnection of the plotter with other functional units
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 15/30 Clipping
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10024 : Color image
• CPC Class Codes: G06T 2207/30196 : Human being; Person
• CPC Class Codes: G06T 2207/30204 : Marker
• CPC Class Codes: G06T 7/11 : Region-based segmentation
• CPC Class Codes: G06T 7/194 : involving foreground-background segmentation
• CPC Class Codes: G06V 20/64 : Three-dimensional objects
• CPC Class Codes: G06V 20/647 : by matching two-dimensional images to three-dimensional objects
• CPC Class Codes: G06V 40/103 : Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/16 : Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/168 : Feature extraction; Face representation
• CPC Class Codes: H04N 13/366 : using viewer tracking

Abstract

An image processing device includes an image acquirer that acquires data of a shot image obtained by a camera, a depth image acquirer that acquires a depth image obtained by representing a distance of a subject from the camera in the depth direction as a pixel value, and a first object detector that detects a region of an image of a first object from the shot image or the depth image. The image processing device further includes a second object detector that detects a region of an image of a second object based on the region of the image of the first object and the pixel value in the depth image, a clipping region decider that decides a clipping region including at least the regions of the images of the first and second objects, and a display image generator that generates a display image by using the clipping region.

Claims

1. An image processing device comprising:

an image acquirer configured to acquire data of a shot image obtained by shooting by a camera;

a depth image acquirer configured to acquire a depth image obtained by representing a distance of a subject from the camera in depth direction as a pixel value on an image plane;

a first object detector configured to detect a region of an image of a first object from the shot image or the depth image based on a shape or a feature;

a second object detector configured to detect a region of an image of a second object based on the region of the image of the first object and the pixel value in the depth image;

a clipping region decider configured to decide a clipping region including at least the region of the image of the first object and the region of the image of the second object; and

a display image generator configured to generate a display image by using the clipping region in the shot image separately from the other region.

2. The image processing device according to claim 1, further comprising

a marker detector configured to detect, from the shot image, a region of an image of a marker with which a subject is equipped,

wherein the clipping region decider decides the clipping region in such a manner that the clipping region includes the region of the image of the marker in addition to the region of the image of the first object and the region of the image of the second object.

3. The image processing device according to claim 2, wherein

the image acquirer acquires, as the shot image, data of a moving image shot by the camera on each image frame basis,

the image frames cyclically include an image frame for marker detection shot under a shooting condition suitable for detection of the region of the image of the marker by the marker detector, and

the marker detector detects the region of the image of the marker by using the image frame for marker detection.

4. The image processing device according to claim 3, further comprising

an information processor configured to decide whether or not to shoot the image frame for marker detection and a cycle of shooting of the image frame for marker detection and control a shooting condition in the camera.

5. The image processing device according to claim 2, wherein

the display image generator generates the display image by combining the clipping region with another image and drawing a predetermined object at a position corresponding to at least any of the first object, the second object, and the marker.

6. The image processing device according to claim 1, wherein

the first object detector detects a region of an image of a head of a person by face detection processing for the shot image, and

the second object detector detects a region that is continuous from the region of the image of the head and has pixel values falling within a predetermined range in the depth image as a region of a body of the person.

7. The image processing device according to claim 1, wherein

the display image generator combines the clipping region with an image in which an object in a virtual space is drawn by giving the clipping region a depth value in the virtual space relative to the camera, and executes hidden surface removal processing by comparison of the depth value between the clipping region and the object.

8. The image processing device according to claim 1, wherein

the display image generator combines the clipping region with another image and generates, as the display image, a moving image in which a predetermined object appears when a user carries out predetermined operation to the image processing device and the object moves with motion decided based on the region of the image of the second object corresponding to the user who has carried out the operation in the clipping region.

9. An image processing system comprising:

an imaging device configured to shoot a moving image of a subject equipped with a marker; and

an image processing device configured to generate a display image that changes according to motion of the subject based on a position of the marker in an image frame of the shot moving image,

wherein the imaging device shoots, as the moving image, an image frame for displaying shot under a shooting condition suitable for displaying and an image frame for marker detection shot under a shooting condition suitable for detection of the marker by the image processing device in turn at a predetermined ratio, and

the image processing device includes

a marker detector that detects the marker by using the image frame for marker detection, and

a display image generator that generates the display image by executing image processing based on the position of the detected marker by using the image frame for displaying.

10. An image processing method by which an image processing device generates a display image by using a shot image obtained by shooting by a camera, the image processing method comprising:

acquiring data of the shot image and storing the data of the shot image in a storage device;

acquiring data of a depth image obtained by representing distance of a subject from the camera in depth direction as a pixel value on an image plane and storing the data of the depth image in the storage device;

reading out the data of the shot image or the depth image from the storage device and detecting a region of an image of a first object from the read image based on a shape or a feature;

reading out the data of the depth image from the storage device and detecting a region of an image of a second object from the depth image based on the region of the image of the first object and the pixel value;

deciding a clipping region including at least the region of the image of the first object and the region of the image of the second object; and

reading out the data of the shot image from the storage device and using the clipping region separately from the other region to generate the display image and output the display image to a display device.

11. A computer program for a computer, comprising:

acquiring data of a shot image obtained by shooting by a camera;

acquiring a depth image obtained by representing distance of a subject from the camera in depth direction as a pixel value on an image plane;

detecting a region of an image of a first object from the shot image or the depth image based on a shape or a feature;

detecting a region of an image of a second object based on the region of the image of the first object and the pixel value in the depth image;

deciding a clipping region including at least the region of the image of the first object and the region of the image of the second object; and

generating a display image by using the clipping region in the shot image separately from the other region.

12. A computer-readable recording medium in which a computer program is recorded, the computer program being for a computer, comprising:

acquiring data of a shot image obtained by shooting by a camera;

acquiring a depth image obtained by representing distance of a subject from the camera in depth direction as a pixel value on an image plane;

detecting a region of an image of a first object from the shot image or the depth image based on a shape or a feature;

detecting a region of an image of a second object based on the region of the image of the first object and the pixel value in the depth image;

deciding a clipping region including at least the region of the image of the first object and the region of the image of the second object; and

generating a display image by using the clipping region in the shot image separately from the other region.