Image processing method and image processing device

• US 9,968,176 B2
• Filed: 04/16/2014
• Issued: 05/15/2018
• Est. Priority Date: 04/17/2013
• Status: Active Grant

• Current Assignee: Panasonic Intellectual Property Management Co., Ltd. (Panasonic Holdings Corporation)
• Original Assignee: Panasonic Intellectual Property Management Co., Ltd. (Panasonic Holdings Corporation)
• Inventors: Tomita, Hiroto, Nguyen, Phu, Tezuka, Tadanori
• Primary Examiner(s): SUH, JOSEPH JINWOO
• Application Number:
  US14/784,743
• Publication Number:
  US 20160058158A1
• Time in Patent Office: 1,490 Days
• CPC Class Codes: A45D 44/005 for selecting or displaying personal cosmetic colours or hairstyle
• CPC Class Codes: G06F 3/0416 Control or interface arrangements specially adapted for digitisers
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: H04N 2005/2726 for simulating a person's appearance, e.g. hair style, glasses, clothes
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/64 Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof

Abstract

An image processing method including: determining whether an operation mode of a system is a control mode for still image display or a view mode for video display; and performing processing in accordance with the operation mode. Processing in the control mode includes: specifying a specific video frame image including a suitable image of a subject of image processing, from among video frame images of a video constituted of a plurality of video frame images each including an image of the subject; displaying the specific video frame image as a still image; and receiving control related to image processing from a user. Processing in the view mode includes: performing image processing with respect to each video frame image that is input, one video frame image at a time; and displaying the result of image processing for the user to view.

Claims

1. An image processing method usable in an image processing system, the image processing system performing image processing with respect to a video input thereto and displaying a result of the image processing, the image processing method comprising:

determining whether an operation mode of the system is a control mode or a view mode, the system performing still image display in the control mode and performing video display in the view mode; and

performing processing in accordance with the operation mode, wherein

when the system is in the control mode, the processing in accordance with the operation mode comprises;

each time a video frame image is input, performing frame determination processing of determining whether the video frame image includes a face image of a subject of the image processing; and

when the frame determination processing determines that the video frame image includes a face image of the subject,

performing reception processing of setting the one video frame image as an image to be displayed for receiving control related to the image processing, displaying the one video frame image as a still image, receiving a specification of at least one area of the one video frame image displayed as a still image and receiving user control for specifying operations of the image processing to be performed on the subject via the at least one area of the one video frame image displayed as a still image, and causing the system to refrain from performing the frame determination processing with respect to any video frame image subsequent to the one video frame image, and

when the system transitions to the view mode from the control mode, the processing in accordance with the operation mode comprises;

each time a video frame image is input, performing the image processing with respect to an image of the subject in the video frame image based on the specified operations of the image processing; and

displaying the result of the image processing for the user to view.

2. The image processing method of claim 1, wherein

the system comprises a frame memory configured to store a processing-target video frame image, and performs the processing in accordance with the operation mode with respect to the processing-target video frame image, which is a video frame image currently stored in the frame memory,

when the system is in the view mode, the processing-target video frame image is updated each time a new video frame image is input, by overwriting a video frame image currently stored in the frame memory with the new video frame image, and

when the system is in the control mode, once the one video frame image is determined and stored to the frame memory as the processing-target video frame image, the processing-target video frame image is not updated until the operation mode switches to the view mode from the control mode.

3. The image processing method of claim 2, wherein

the frame determination processing further comprises:

each time a video frame image is input,

detecting characteristic coordinates in the video frame image; and

performing examination of a position indicated by the characteristic coordinates, wherein the determination of whether the image of the subject in the one video frame image is suitable is performed based on a result of the examination.

4. The image processing method of claim 3, wherein

the subject is a face of a person,

the characteristic coordinates detected in the video frame image are characteristic coordinates of an eye, and

the examination comprises examining whether the eye is open or closed based on positions of the characteristic coordinates of the eye, wherein when the eye is open, the video frame image is determined as the one video frame image and remains stored in the frame memory, and when the eye is closed, the frame memory is updated by using a video frame image following the video frame image.

5. The image processing method of claim 3, wherein

the subject is a face of a person,

the characteristic coordinates detected in the video frame image indicate a direction in which the face is facing, and

the examination comprises examining whether or not the face is facing forward based on the direction, wherein when the face is facing forward, the video frame image is determined as the one video frame image and remains stored in the frame memory, and when the face is not facing forward, the frame memory is updated by using a video frame image following the video frame image.

6. The image processing method of claim 3, wherein

the processing in accordance with the operation mode, when the system is in the control mode, comprises storing characteristic points detected within a predetermined period, and performing characteristic coordinate filtering with respect to the stored characteristic coordinates, the characteristic coordinate filtering comprising filtering corresponding ones of the stored characteristic coordinates,

the processing in accordance with the operation mode, when the system is in the view mode, comprises performing characteristic coordinate detection of detecting characteristic coordinates in each video frame image that is input, and

when the system is in the control mode, the examination is performed by using characteristic coordinates yielded as a result of the characteristic coordinate filtering, in place of the characteristic coordinates yielded as a result of the detection of characteristic coordinate.

7. The image processing method of claim 6,

the filtering in the characteristic coordinate filtering comprises averaging corresponding ones of the stored characteristic coordinates.

8. The image processing method of claim 6, wherein

the characteristic coordinate filtering further comprises, after performing the filtering, comparing the stored characteristic coordinates and characteristic coordinates yielded as a result of the filtering, and performing the filtering once again after excluding ones of the stored characteristic coordinates differing from corresponding ones of the characteristic coordinates yielded as a result of the filtering by a value greater than a predetermined threshold value.

9. The image processing method of claim 6 further comprising:

receiving a setting of the operation mode from the user, wherein

the processing in accordance with the operation mode, when the system is in the view mode, comprises;

performing make-up processing with respect to a part of the subject, in each video frame image that is input, that is related to characteristic coordinates yielded as a result of the characteristic coordinate filtering; and

displaying the each video frame image with the make-up processing applied,

the user control for specifying the contents of the image processing includes selecting a sample color, and

the make-up processing comprises mapping, with respect to a color range of the sample color, values of a plurality of pixels composing the part of the subject.

10. The image processing method of claim 1, wherein

the processing in accordance with the operation mode, when the system is in the control mode, comprises detecting characteristic coordinates in the one video frame image, and

the processing in accordance with the operation mode, when the system is in the view mode, comprises make-up processing of applying virtual make-up with respect to a part of the subject, in each video frame image that is input, that is related to the characteristic coordinates, and displaying the each video frame image with the virtual-make up applied.

11. The image processing method of claim 10, wherein

the user control for specifying the contents the image processing includes selecting a sample color, and

the make-up processing comprises mapping, with respect to a color range of the sample color, values of a plurality of pixels composing the part of the subject in each video frame image that is input.

12. The image processing method of claim 10, wherein

the user control for specifying the contents of the image processing includes selecting a sample color and performing finger drawing control with respect to the image of the subject in the one video frame image, and

the make-up processing comprises mapping, with respect to the part of the subject in each video frame image that is input, a drawing pattern that has a shape pertaining to the finger drawing control and that is composed of pixels of the sample color.

13. The image processing method of claim 10, wherein

the user control for specifying the contents of the image processing includes performing hand drawing control with respect to the part of the subject in the one video frame image,

the characteristic coordinates define a shape of an outline of the part of the subject in the one video frame image, and

the make-up processing comprises searching for corresponding points in the one video frame image and each video frame image that is input so as to link the characteristic coordinates detected in the one video frame image with corresponding characteristic coordinates in the each video frame image, and mapping a hand-drawn image pertaining to the hard drawing control with respect to a part of the subject, in the each video frame image, that has been linked with the part of the subject in the one video frame image through the search for corresponding points.

14. The image processing method of claim 10, wherein

the processing in accordance with the operation mode, when the system is in the control mode, further comprises storing the characteristic coordinates, wherein

in the view mode, the part of the subject is located at a position of the subject, in each video frame image that is input, related to the stored characteristic coordinates.

15. The image processing method of claim 1 further comprising

receiving a setting of the operation mode from the user.

16. The image processing method of claim 15, wherein

the system comprises a touch panel, and

the user performs the setting of the operation mode by performing touch control with respect to a specific area of the touch panel.

17. The image processing method of claim 16 further comprising

when the user performs the setting of the operation mode, providing a notification that setting has been accepted.

18. The image processing method of claim 17, wherein

the notification is provided by displaying the operation mode that has been set by the user.

19. The image processing method of claim 17, wherein

the notification is provided by output of sound.

20. An image processing device performing image processing with respect to a video input thereto and displaying a result of the image processing, the image processing device comprising:

an operation mode determiner determining whether an operation mode of the image processing device is a control mode or a view mode, the image processing device performing still image display in the control mode and performing video display in the view mode;

an image processor performing image processing in accordance with the operation mode; and

a display displaying an image resulting from the image processing by the image processor, wherein

when the image processing device is in the control mode, the image processor;

each time a video frame image is input, performs frame determination processing of determining whether the video frame image includes a face image of a subject of the image processing; and

when the frame determination processing determines that the video frame image includes a face image of the subject,

performs reception processing of setting the one video frame image as an image to be displayed for receiving control related to the image processing, displays the one video frame image as a still image, receives a specification of at least one area of the one video frame image displayed as a still image and receives user control for specifying operations of the image processing to be performed on the subject via the at least one area of the one video frame image displayed as a still image, and causes the system to refrain from performing the frame determination processing with respect to any video frame image subsequent to the one video frame image, and

when the image processing device transitions to the view mode from the control mode, the image processor;

each time a video frame is input, performs the image processing with respect to an image of the subject in the video frame image based on the specified operations of the image processing; and

presents a result of the image processing for the user to view.

21. A non-transitory computer-readable recording medium having recorded thereon a program usable in an image processing system performing image processing with respect to a video input thereto and displaying a result of the image processing, the program causing a processor to execute the image processing comprising:

determining whether an operation mode of the system is a control mode or a view mode, the system performing still image display in the control mode and performing video display in the view mode; and

performing processing in accordance with the operation mode, wherein

when the system is in the control mode, the processing in accordance with the operation mode comprises;

each time a video frame image is input, performing frame determination processing of determining whether the video frame image includes a face image of a subject of the image processing; and

when the frame determination processing determines that the video frame image includes a face image of the subject,

performing reception processing of setting the one video frame image as an image to be displayed for receiving control related to the image processing, displaying the one video frame image as a still image, receiving a specification of at least one area of the one video frame image displayed as a still image and receiving user control for specifying operations of the image processing to be performed on the subject via the at least one area of the one video frame image displayed as a still image, and causing the system to refrain from performing the frame determination processing with respect to any video frame image subsequent to the one video frame image, and

when the system transitions to the view mode from the control mode, the processing in accordance with the operation mode comprises;

each time a video frame image is input, performing the image processing with respect to an image of the subject in the video frame image based on the specified operations of the image processing; and

displaying the result of the image processing for the user to view.