Information processing apparatus, method of controlling information processing apparatus, and storage medium

• US 10,140,777 B2
• Filed: 07/26/2017
• Issued: 11/27/2018
• Est. Priority Date: 08/02/2016
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Nakagawa, Masashi
• Primary Examiner(s): Wu, Xiao
• Assistant Examiner(s): Akhavannik, Mohammad H
• Application Number:
  US15/660,532
• Publication Number:
  US 20180040169A1
• Time in Patent Office: 489 Days
• Field of Search: None
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 17/30 Polynomial surface description
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2219/2004 Aligning objects, relative positioning of parts
• CPC Class Codes: G06T 2219/2016 Rotation, translation, scaling
• CPC Class Codes: G06T 2219/2021 Shape modification
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80

Abstract

An information processing apparatus comprises an extraction unit configured to extract a finger region from a captured image; a generation unit configured to generate a finger model from the finger region; a determination unit configured to perform a collision determination for a collision between the finger model and a CG model superimposed on the captured image; and a control unit configured to control a display form of the finger model based on a result of the collision determination.

Claims

1. An information processing apparatus, comprising:

a memory configured to store instructions; and

a processor configured to execute the instructions to function as;

an extraction unit configured to extract a finger region from a captured image;

a generation unit configured to generate a finger model from the finger region;

a determination unit configured to perform a collision determination for a collision between the finger model and a CG model superimposed on the captured image; and

a control unit configured to;

when the collision determination indicates that the finger model has sunk into the CG model, calculate a movement amount of the finger model such that the finger model is not sunk into the CG model; and

control a display form of the finger model based on the determined movement amount of the finger model such that the finger model is not sunk into the CG model.

2. The information processing apparatus according to claim 1, wherein the control unit is configured to control at least one of a shape and a display position of the finger model based on the result of the collision determination.

3. The information processing apparatus according to claim 1, wherein the processor is further configured to execute instructions to function as a processing unit configured to perform processing that removes the finger region from the captured image, and inpaints the removed region.

4. The information processing apparatus according to claim 3, wherein the processing unit is configured to perform processing that supplements the removed region in accordance with an image of a previous frame.

5. The information processing apparatus according to claim 3, wherein the processing unit is configured to perform processing that supplements the removed region in accordance with a texture of a peripheral pixel of the finger region.

6. The information processing apparatus according to claim 1, wherein the processor is further configured to execute instructions to function as a processing unit configured to perform processing that displays a finger operation state on a surface of the CG model in the captured image.

7. The information processing apparatus according to claim 6, wherein the processing unit is configured to display the operation state by obtaining a surface of the CG model that is sandwiched by the finger model before and after the display form control by the control unit, and distinguishably display the finger model on the surface.

8. The information processing apparatus according to claim 6, wherein the processing unit is configured to display the finger region as semi-transparent, in addition to displaying of the operation state.

9. The information processing apparatus according to claim 1, wherein the processor is further configured to execute instructions to function as:

a reconfiguration unit configured to obtain three-dimensional reconfiguration data of a physical space, and

a removal unit configured to remove the finger region of the captured image, and supplement the finger region by using the three-dimensional reconfiguration data.

10. The information processing apparatus according to claim 1, wherein the control unit is further configured to:

calculate the movement amount of the finger model based on an interference depth vector corresponding to an interference between the finger model and the CG model;

control the display form of the finger model by deforming or moving the finger model to a surface of the CG model based on the interference depth vector.

11. A method of controlling an information processing apparatus, comprising:

extracting a finger region from a captured image;

generating a finger model from the finger region;

performing a collision determination for a collision between the finger model and a CG model superimposed on the captured image;

when the collision determination indicates that the finger model has sunk into the CG model, calculating a movement amount of the finger model such that the finger model is not sunk into the CG model; and

controlling a display form of the finger model based on the determined movement amount of the finger model such that the finger model is not sunk into the CG model.

12. A non-transitory computer-readable storage medium storing a computer program for causing a computer to execute a method of controlling an information processing apparatus, the method comprising:

extracting a finger region from a captured image;

generating a finger model from the finger region;

performing a collision determination for a collision between the finger model and a CG model superimposed on the captured image;

when the collision determination indicates that the finger model has sunk into the CG model, calculating a movement amount of the finger model such that the finger model is not sunk into the CG model; and

controlling a display form of the finger model based on the determined movement amount of the finger model such that the finger model is not sunk into the CG model.