Adapting Quantization

• US 20160100166A1
• Filed: 12/04/2014
• Published: 04/07/2016
• Est. Priority Date: 10/03/2014
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Dragne, Lucian, Hess, Hans Peter
• Application Number:
  US14/560,669
• Publication Number:
  US 20160100166A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: H04N 19/124 Quantisation
• CPC Class Codes: H04N 19/136 Incoming video signal characteristics or properties
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/47 End-user applications
• CPC Class Codes: H04N 21/4728 for selecting a Region Of Interest [ROI], e.g. for requesting a higher resolution version of a selected region
• CPC Class Codes: H04N 21/4781 Games
• CPC Class Codes: H04N 21/4788 communicating with other users, e.g. chatting arrangements for providing for computer conferences, e.g. chat rooms, to substation in data switching networks H04L12/1813; distributed application using peer-to-peer [P2P] networks H04L67/104
• CPC Class Codes: H04N 5/44 Receiver circuitry for the reception of television signals according to analogue transmission standards H04N5/14 takes precedence

Abstract

A device comprising: an encoder for encoding a video signal representing a video image of a scene captured by a camera, and a controller. The encoder comprises a quantizer for performing a quantization on the video signal as part of said encoding. The controller is configured to receive skeletal tracking information from a skeletal tracking algorithm relating to one or more skeletal features of a user present in the scene, and based thereon to define one or more regions-of-interest within the video image corresponding to one or more bodily areas of the user, and to adapt the quantization to use a finer quantization granularity within the one or more regions-of-interest than outside the one or more regions-of interest.

Claims

1. A device comprising:

an encoder for encoding a video signal representing a video image of a scene captured by a camera, the encoder comprising a quantizer for performing a quantization on said video signal as part of said encoding; and

a controller configured to receive skeletal tracking information from a skeletal tracking algorithm relating to one or more skeletal features of a user present in said scene, and based thereon to define one or more regions-of-interest within the video image corresponding to one or more bodily areas of the user, and to adapt the quantization to use a finer quantization granularity within the one or more regions-of-interest than outside the one or more regions-of interest.

2. The device of claim 1, wherein the controller is configured to define a plurality of different regions-of-interest each corresponding to a respective bodily area of the user, and to adapt the quantization to use a finer quantization granularity within each of said plurality of regions-of-interest than outside the plurality of regions-of-interest.

3. The device of claim 2, wherein one or more of the different regions-of-interest are quantized with the finer quantization granularity only at some times and not others.

4. The device of claim 3, wherein the controller is configured to adaptively select which of the different regions-of-interest is currently quantized with the finer quantization granularity in dependence on a current bitrate constraint.

5. The device of claim 4, wherein the bodily areas are assigned an order of priority, and the controller is configured to perform said selection according to the order of priority of the bodily areas to which the different regions-of-interest correspond.

6. The device of claim 2, wherein the controller is configured to adapt the quantization to use different levels of quantization granularity within different ones of said plurality of regions-of interest, each being finer than outside the plurality of regions-of-interest.

7. The device of claim 6, wherein said bodily areas are assigned an order of priority, and the controller is configured to set the different levels according to the order of priority of the bodily areas to which the different regions-of-interest correspond.

8. The device of claim 1, wherein each of the bodily areas is one of:

(a) the user'"'"'s whole body;

(b) the user'"'"'s head, torso and arms;

(c) the user'"'"'s head, thorax and arms;

(d) the user'"'"'s head and shoulders;

(e) the user'"'"'s head;

(f) the user'"'"'s torso

(g) the user'"'"'s thorax;

(h) the user'"'"'s abdomen;

(i) the user'"'"'s arms and hands;

(j) the user'"'"'s shoulders;

or

(k) the user'"'"'s hands.

9. The device of claim 5, wherein the order of priority is:

(i) the user'"'"'s head;

(ii) the user'"'"'s head and shoulders;

or head, thorax and arms;

or head, torso and arms;

(iii) the user'"'"'s whole body;

such that (iii) is quantized with the finer quantization if the bitrate constraint allows, and if not only (ii) is quantized with the finer quantization if the bitrate constraint allows, and if not only (i) is quantized with the finer quantization.

10. The device of claim 7, wherein the order of priority is:

(i) the user'"'"'s head;

(ii) the user'"'"'s hands, arm, shoulders, thorax and/or torso;

(iii) the rest of the user'"'"'s whole body;

such that is (i) quantized with a first level of quantization granularity, (ii) is quantized with one or more second levels of quantization granularity, and (iii) is quantized with a third level of quantization granularity, the first level being finer than each of the one or more second levels, each of the second levels being finer than the third level, and the third level being finer than outside the regions-of-interest.

11. The device of claim 1, comprising a transmitter configured to transmit the encoded video signal over a channel to at least one other device.

12. The device of claim 4, comprising a transmitter configured to transmit the encoded video signal over a channel to at least one other device, wherein the controller is configured to determine an available bandwidth of said channel, and said bitrate constraint is equal to or otherwise limited by the available bandwidth.

13. The device of claim 1, wherein the controller is configured to apply a successive increase in the coarseness of the quantization granularity from at least one of the one or more regions-of-interest toward the outside.

14. The device of claim 1, wherein the controller is configured to apply a spring model to smooth a motion of the one or more regions-of-interest as they follow the one or more corresponding bodily areas based on the skeletal tracking information.

15. The device of claim 1, comprising a transmitter for transmitting the encoded video signal over a network.

16. The device of claim 1, wherein the skeletal tracking algorithm is implemented on said device and is configured to determine said skeletal tracking information based on one or more separate sensors other than said camera.

17. The device of claim 1, comprising dedicated graphics processing resources and general purpose processing resources, wherein the skeletal tracking algorithm is implemented in the dedicated graphics processing resources and the encoder is implemented in the general purpose processing resources.

18. The device of claim 17, wherein the general purpose processing resources comprise a general purpose processor and the dedicated graphics processing resources comprise a separate graphics processor, the encoder being implemented in the form of code arranged to run on the general purpose processor and the skeletal tracking algorithm being implemented in the form of code arranged to run on the graphics processor.

19. A computer program product comprising code embodied on a computer-readable storage medium and configured so as when run on one or more processors to perform operations of:

encoding a video signal representing a video image of a scene captured by a camera, the encoding comprising performing a quantization on said video signal;

receiving skeletal tracking information from a skeletal tracking algorithm, relating to one or more skeletal features of a user present in said scene;

based on the skeletal tracking information, defining one or more regions-of-interest within the video image corresponding to one or more bodily areas of the user; and

adapting the quantization to use a finer quantization granularity within the one or more regions-of-interest than outside the one or more regions-of interest.

20. A method comprising:

encoding a video signal representing a video image of a scene captured by a camera, the encoding comprising performing a quantization on said video signal;

receiving skeletal tracking information from a skeletal tracking algorithm, relating to one or more skeletal features of a user present in said scene;

based on the skeletal tracking information, defining one or more regions-of-interest within the video image corresponding to one or more bodily areas of the user; and

adapting the quantization to use a finer quantization granularity within the one or more regions-of-interest than outside the one or more regions-of interest.