OBJECT MODELING AND REPLACEMENT IN A VIDEO STREAM

• US 20180005026A1
• Filed: 06/30/2016
• Published: 01/04/2018
• Est. Priority Date: 06/30/2016
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Snap, Inc.
• Inventors: Shaburov, Victor, Kotsur, Sergey, Monastyrshyn, Yurii, Pischenko, Alexander
• Granted Patent: US 10,102,423 B2
• Time in Patent Office: Days
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/22 Setup operations, e.g. calibration, key configuration or button assignment
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/52 involving aspects of the displayed game scene
• CPC Class Codes: A63F 13/655 by importing photos, e.g. of the player
• CPC Class Codes: A63F 13/837 Shooting of targets
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20072 Graph-based image processing
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/77 using statistical methods
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: G06V 40/11 Hand-related biometrics; Hand pose recognition

Abstract

Systems, devices, and methods are presented for segmenting an image of a video stream with a client device by receiving one or more images depicting an object of interest and determining pixels within the one or more images corresponding to the object of interest. The systems, devices, and methods identify a position of a portion of the object of interest and determine a direction for the portion of the object of interest. Based on the direction of the portion of the object of interest, a histogram threshold is dynamically modified for identifying pixels as corresponding to the portion of the object of interest. The portion of the object of interest is replaced with a graphical interface element aligned with the direction of the portion of the object of interest.

Claims

1. A method, comprising:

receiving, by one or more processors, one or more images depicting at least a portion of a hand;

determining pixels within the one or more images corresponding to the portion of the hand in a predetermined portion of a field of view of an image capture device, the portion of the hand having a finger;

based on the pixels corresponding to the portion of the hand, identifying a finger position of the finger;

determining a direction of the finger based on the finger position;

based on the direction of the finger, dynamically modifying a histogram threshold for identifying pixels as corresponding to the portion of the hand; and

replacing the portion of the hand and the finger with a graphical interface element aligned with the direction of the finger.

2. The method of claim 1, wherein identifying the finger position further comprises:

forming a convex polygon encompassing at least a part of the portion f the hand; and

identifying one or more defects within the convex polygon, a defect indicating a space between two fingers located on the portion of the hand.

3. The method of claim 1, wherein determining the direction of the finger further comprises:

identifying a tip of the finger, the tip representing a first point for the finger;

identifying a second point on the finger, the second point spaced a distance from the first point along the finger;

generating a direction line extending between the first point and the second point; and

determining a direction of the direction line within the one or more images.

4. The method of claim 1, wherein determining the pixels corresponding to the portion of the hand further comprises:

sampling one or more color values from one or more pixels within the portion of the field of view of the image capture device; and

determining a color profile for the hand based on the one or more color values sampled from the one or more pixels, the color profile representing a median color value of the hand.

5. The method of claim 4, wherein sampling the one or more color values further comprises:

selecting a first pixel within the portion of the field of view;

determining that the first pixel includes a first color value within a predetermined range of color values;

selecting a second pixel within the portion of the field of view, the second pixel being spaced a distance away from the first pixel and within the portion of the field of view;

determining that the second pixel includes a second color value within the predetermined range of color values;

comparing the first color value and the second color value to determine that the second color value is within a predetermined threshold of the first color value; and

including the first color value and the second color value in the color profile for the hand.

6. The method of claim 4, wherein determining the color profile further comprises:

generating a histogram from the one or more color values sampled from within the portion of the field of view;

removing one or more bins of the histogram associated with a number of pixels below a predetermined pixel threshold; and

including one or more bins of the histogram within the color profile associated with a number of pixels above the predetermined pixel threshold.

7. The method of claim 1, further comprising:

generating one or more binary images by isolating the pixels corresponding to the portion of the hand by converting the pixels corresponding to the portion of the hand to a first value and remaining pixels within the field of view to a second value.

8. The method of claim 1, wherein determining the direction of the finger further comprises:

determining a direction of the finger for each frame of a set of previous frames of the one or more images; and

combining the direction of the finger for the set of previous frames to identify an aggregate direction of the finger.

9. The method of claim 1, wherein determining the direction of the finger further comprises:

determining a current direction of the finger from a specified corner of the field of view;

identifying a combined direction of the finger for the one or more images to indicate a previous combined direction;

determining that a change in position between the current direction and the previous combined direction exceeds a position threshold; and

selecting from a set of first direction identification operations and a set of second direction identification operations based on the change in position exceeding the position threshold.

10. The method of claim 1, wherein the direction of the finger is represented by a vector extending between two pixels corresponding to the portion of the hand and dynamically modifying the histogram threshold further comprises:

identifying a first performance characteristic of the vector;

determining that the first performance characteristic exceeds a characteristic threshold;

based on the first performance characteristic, modifying the histogram threshold; and

based on the modification of the histogram threshold, identifying that a second performance characteristic of the vector is within the characteristic threshold.

11. The method of claim 1, wherein determining the pixels within the one or more images corresponding to the portion of the hand identifies a first set of pixels and further comprises:

determining that a set of artifacts within the first set of pixels exceeds an artifact threshold; and

based on modifying the histogram threshold, determining a second set of pixels within the one or more images corresponding to the portion of the hand, the second set of pixels including at least a portion of the first set of pixels.

12. A system, comprising:

one or more processors; and

a processor-readable storage device storing processor-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising;

receiving, by the one or more processors, one or more images depicting at least a portion of a hand;

determining pixels within the one or more images corresponding to the portion of the hand in a predetermined portion of a field of view of an image capture device, the portion of the hand having a finger;

based on the pixels corresponding to the portion of the hand, identifying a finger position of the finger;

determining a direction of the finger based on the finger position;

based on the direction of the finger, dynamically modifying a histogram threshold for identifying pixels as corresponding to the portion of the hand; and

replacing the portion of the hand and the finger with a graphical interface element aligned with the direction of the finger.

13. The system of claim 12, wherein determining the direction of the finger further comprises:

identifying a tip of the finger, the tip representing a first point for the finger;

identifying a second point on the finger, the second point spaced a distance from the first point along the finger;

generating a direction line extending between the first point and the second point; and

determining a direction of the direction line within the one or more images.

14. The system of claim 12, wherein determining the pixels corresponding to the portion of the hand further comprises:

sampling one or more color values from one or more pixels within the portion of the field of view of the image capture device; and

determining a color profile for the hand based on the one or more color values sampled from the one or more pixels, the color profile representing a median color value of the hand.

15. The system of claim 14, wherein determining the color profile further comprises:

generating a histogram from the one or more color values sampled from within the portion of the field of view;

removing one or more bins of the histogram associated with a number of pixels below a predetermined pixel threshold; and

including one or more bins of the histogram within the color profile associated with a number of pixels above the predetermined pixel threshold.

16. The system of claim 12, wherein determining the direction of the finger further comprises:

determining a current direction of the finger from a specified corner of the field of view;

identifying a combined direction of the finger for e one or more images to indicate a previous combined direction;

determining that a change in position between the current direction and the previous combined direction exceeds a position threshold; and

selecting from a set of first direction identification operations and a set of second direction identification operations based on the change in position exceeding the position threshold.

17. The system of claim 12, wherein the direction of the finger is represented by a vector extending between two pixels corresponding to the portion of the hand and dynamically modifying the histogram threshold further comprises:

identifying a first performance characteristic of the vector;

determining that the first performance characteristic exceeds a characteristic threshold;

based on the first performance characteristic, modifying the histogram threshold; and

based on the modification of the histogram threshold, identifying that a second performance characteristic of the vector is within the characteristic threshold.

18. A processor-readable storage device storing processor-executable instructions that, when executed by a processor of a machine, cause the machine to perform operations comprising:

receiving one or more images depicting at least a portion of a hand;

determining pixels within the one or more images corresponding to the portion of the hand in a predetermined portion of a field of view of an image capture device, the portion of the hand having a finger;

based on the pixels corresponding to the portion of the hand, identifying a finger position of the finger;

determining a direction of the finger based on the finger position;

based on the direction of the finger, dynamically modifying a histogram threshold for identifying pixels as corresponding to the portion of the hand; and

replacing the portion of the hand and the finger with a graphical interface element aligned with the direction of the finger.

19. The processor-readable storage device of claim 18, wherein determining the direction of the finger further comprises:

identifying a tip of the finger, the tip representing a first point for the finger;

identifying a second point on the finger, the second point spaced a distance from the first point along the finger;

generating a direction line extending between the first point and the second point; and

determining a direction of the direction line within the one or more images.

20. The processor-readable storage device of claim 18, wherein determining the direction of the finger further comprises:

determining a current direction of the finger from a specified corner of the field of view;

identifying a combined direction of the finger for the one or more images to indicate a previous combined direction;

determining that a change in position between the current direction and the previous combined direction exceeds a position threshold; and

selecting from a set of first direction identification operations and a set of second direction identification operations based on the change in position exceeding the position threshold.