Facial Pose Improvement with Perspective Distortion Correction

• US 20110090303A1
• Filed: 10/16/2009
• Published: 04/21/2011
• Est. Priority Date: 10/16/2009
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Shi, Xiaojin, Chung, Chris Yoochang, Wu, Hsi-Jung, Normile, James
• Granted Patent: US 8,599,238 B2
• Time in Patent Office: Days
• US Class Current: 348/14.16
• CPC Class Codes: G06T 2207/20104 Interactive definition of region of interest [ROI]
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 3/00 Geometric image transformations in the plane of the image
• CPC Class Codes: G06T 5/80 Geometric correction
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 23/81 for suppressing or minimising disturbance in the image signal generation

Abstract

Methods, systems, and apparatus are presented for reducing distortion in an image, such as a video image. A video image can be captured by an image capture device, e.g. during a video conferencing session. Distortion correction processing, such as the application of one or more warping techniques, can be applied to the captured image to produce a distortion corrected image, which can be transmitted to one or more participants. The warping techniques can be performed in accordance with one or more warp parameters specifying a transformation of the captured image. Further, the warp parameters can be generated in accordance with an orientation of the image capture device, which can be determined based on sensor data or can be a fixed value. Additionally or alternatively, the warp parameters can be determined in accordance with a reference image or model to which the captured image should be warped.

Claims

1. A method of reducing distortion in an image, the method comprising:

receiving a captured image;

determining an orientation of a camera used to generate the captured image;

computing a warp parameter based on the determined orientation of the camera; and

applying, based on the computed warp parameter, one or more warp techniques to the captured image to generate a distortion corrected image.

2. The method of claim 1, wherein determining an orientation further comprises:

determining the orientation of the camera in accordance with a default orientation value.

3. The method of claim 1, wherein determining an orientation further comprises:

determining the orientation of the camera based on sensor data.

4. The method of claim 3, further comprising:

receiving the sensor data from a gyroscope associated with the camera.

5. The method of claim 3, wherein the captured image corresponds to a video sequence.

6. The method of claim 5, further comprising:

filtering either or both of the sensor data and the computed warp parameter.

7. The method of claim 3, further comprising:

receiving sensor data associated with another image corresponding to the video sequence; and

updating the computed warp parameter, in accordance with the received sensor data, for use with the another image.

8. The method of claim 1, further comprising:

receiving input identifying a face location in the captured image.

9. The method of claim 8, further comprising:

applying the one or more warp techniques to the captured image in accordance with the identified face location.

10. The method of claim 8, wherein receiving input identifying a face location further comprises:

receiving input from a user defining a boundary corresponding to the face location.

11. A computer program product, encoded on a computer-readable medium, operable to cause data processing apparatus to perform operations comprising:

receiving a captured image depicting a face, wherein the captured image is generated by a camera;

analyzing sensor data to determine an orientation of the camera;

computing one or more warp parameters based on the determined orientation of the camera; and

warping at least a portion of the captured image, based on the computed one or more warp parameters, to generate a distortion corrected image.

12. The computer program product of claim 11, wherein:

the orientation of the camera comprises an angle of tilt relative to a reference plane.

13. The computer program product of claim 11, further operable to cause data processing apparatus to perform operations comprising:

receiving the sensor data from an accelerometer.

14. The computer program product of claim 11, further operable to cause data processing apparatus to perform operations comprising:

receiving input from a face detection algorithm identifying a location of the depicted face.

15. The computer program product of claim 14, further operable to cause data processing apparatus to perform operations comprising:

warping the at least a portion of the captured image in accordance with the identified location of the depicted face.

16. The computer program product of claim 11, further operable to cause data processing apparatus to perform operations comprising:

analyzing sensor data associated with a headset worn by a user to determine an orientation of the camera relative to the user'"'"'s face; and

computing one or more warp parameters in accordance with the determined orientation of the camera relative to the user'"'"'s face.

17. A system comprising:

a camera configured to capture video images;

a position sensor; and

a computing system coupled to the camera and the position sensor, the computing system including one or more processors configured to perform operations comprising;

receiving, from the camera, a captured video image;

receiving sensor data from the position sensor;

analyzing the received sensor data to determine an orientation of the camera;

computing a warp parameter based on the determined orientation of the camera; and

applying, based on the computed warp parameter, one or more warp techniques to the captured video image to generate a distortion corrected image.

18. The system of claim 17, wherein the one or more processors are further configured to perform operations comprising:

generating the distortion corrected image in real-time.

19. The system of claim 17, wherein the position sensor comprises one or more position sensors comprising a gyroscope, an accelerometer, or a combination thereof.

20. The system of claim 17, wherein the computing system further includes:

a touch screen display configured to receive touch input from a user.

21. The system of claim 20, wherein the one or more processors are further configured to perform operations comprising:

receiving, through the touch screen display, input from a user identifying one or more locations on a face depicted in the captured video image.

22. The system of claim 21, wherein the one or more processors are further configured to perform operations comprising:

applying the one or more warp techniques to the captured video image in accordance with the identified one or more locations on the face.

23. The system of claim 21, wherein one or more sensor data offsets associated with the position sensor can be manually configured by a user to allow the user to adjust the position sensor in different positions.

24. A method of reducing distortion in an image, the method comprising:

receiving a captured image;

identifying one or more registration points associated with the captured image;

comparing, based on the identified one or more registration points, the captured image with a reference image;

determining, in response to the comparing, one or more warp parameters; and

applying, based on the determined one or more warp parameters, one or more warp techniques to the captured image to generate a distortion corrected image.

25. The method of claim 24, further comprising:

receiving sensor data from a position sensor associated with a camera used to generate the captured image.

26. The method of claim 25, further comprising:

adjusting the determined one or more warp parameters based on the received sensor data.

27. The method of claim 24, further comprising:

selecting the reference image based on a resolution of the captured image.

28. The method of claim 24, further comprising:

transmitting the distortion corrected image to a remote computing system.

29. The method of claim 24, wherein identifying one or more registration points further comprises:

automatically identifying the one or more registration points based on facial detection processing.

30. The method of claim 24, wherein identifying one or more registration points further comprises:

receiving user input identifying the one or more registration points.

31. The method of claim 24, wherein the reference image further comprises:

a three-dimensional (3-D) reference model.

32. A computer program product, encoded on a computer-readable medium, operable to cause data processing apparatus to perform operations comprising:

receiving a captured image depicting a face;

identifying one or more registration points associated with the captured image;

comparing, based on the identified one or more registration points, the captured image with the reference image;

determining, in response to the comparing, one or more warp parameters; and

applying, based on the determined one or more warp parameters, one or more warp techniques to the captured image to generate a distortion corrected image.

33. The computer program product of claim 32, wherein:

the reference image includes a face different from the depicted face corresponding to the captured image.

34. The computer program product of claim 32, further operable to cause data processing apparatus to perform operations comprising:

presenting the distortion corrected image to a user as a preview image.

35. The computer program product of claim 32, further operable to cause data processing apparatus to perform operations comprising:

generating the distortion corrected image in real-time.