Region-of-interest processing for video telephony

• US 20060215753A1
• Filed: 07/15/2005
• Published: 09/28/2006
• Est. Priority Date: 03/09/2005
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: El-Maleh, Khaled Helmi, Lee, Yen-Chi, Tsai, Ming-Chang
• Granted Patent: US 8,019,175 B2
• Time in Patent Office: Days
• US Class Current: 375/240.80
• CPC Class Codes: H04N 19/124 Quantisation
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object
• CPC Class Codes: H04N 21/234345 the reformatting operation being performed only on part of the stream, e.g. a region of the image or a time segment
• CPC Class Codes: H04N 21/4621 Controlling the complexity of the content stream or additional data, e.g. lowering the resolution or bit-rate of the video stream for a mobile client with a small screen arrangements for using the results of monitoring on user's side in broadcast systems H04H60/65; flow control in packet networks H04L47/10
• CPC Class Codes: H04N 21/4728 for selecting a Region Of Interest [ROI], e.g. for requesting a higher resolution version of a selected region
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q

Abstract

The disclosure is directed to techniques for region-of-interest (ROI) processing for video telephone (VT) applications. According to the disclosed techniques, a recipient device defines ROI information for video information transmitted by a sender device, i.e., far-end video information. The recipient device transmits the ROI information to the sender device. Using the ROI information transmitted by the recipient device, the sender device applies preferential encoding to an ROI within a video scene. In this manner, the recipient device is able to remotely control ROI encoding of far-end video information by the sender device.

Claims

1. A method comprising:

receiving information from a remote device specifying a region of interest (ROI) within near-end video encoded by a local device and received by the remote device; and

encoding the near-end video based on the ROI to enhance image quality of the ROI relative to non-ROI areas of the video.

2. The method of claim 1, further comprising transmitting the encoded near-end video to the remote device, and receiving, in the local device, far-end video encoded by the remote device.

3. The method of claim 1, further comprising receiving the information specifying the ROI with encoded far-end video received from the remote device, wherein the information specifying the ROI is embedded within the encoded far-end video.

4. The method of claim 1, further comprising receiving the information specifying the ROI by out-of-band signaling from the remote device.

5. The method of claim 1, further comprising:

receiving, in the local device, far-end video encoded by the remote device;

generating information specifying an ROI within the encoded far-end video; and

transmitting the ROI information with the encoded near-end video to the remote device.

6. The method of claim 1, further comprising:

receiving, in the local device, far-end video encoded by the remote device; and

decoding the encoded far-end video received from the remote device to enhance the image quality of the ROI in the far-end video relative to non-ROI areas of the far-end video.

7. The method of claim 6, wherein decoding the encoded far-end video includes applying higher quality post-processing or error concealment techniques to the ROI in the far-end video relative to non-ROI areas of the far-end video.

8. The method of claim 1, further comprising generating a macroblock (MB) map identifying MBs that are within the ROI based on the information specifying the ROI.

9. The method of claim 1, wherein encoding the near-end video includes applying higher quality encoding or error protection techniques to the ROI in the near-end video relative to non-ROI areas of the near-end video.

10. The method of claim 1, further comprising authenticating a remote user associated with the remote device before encoding the near-end video based on the ROI.

11. The method of claim 10, wherein authenticating includes determining whether the remote user is authorized to control encoding of the near-end video based on the ROI.

12. The method of claim 10, wherein authenticating includes seeking, from a local user associated with the local device, authorization for the remote user to control encoding of the near-end video based on the ROI.

13. The method of claim 1, wherein receiving information from a remote device includes receiving information from a plurality of remote devices specifying multiple ROIs within the near-end video, the method further comprising authenticating remote users associated with the remote devices to select one of the remote users to control encoding of the near-end video based on the respective ROI.

14. The method of claim 1, further comprising:

monitoring motion information associated with the encoded near-end video;

adjusting the ROI based on the motion information; and

encoding the near-end video based on the adjusted ROI.

15. The method of claim 14, further comprising generating a macroblock (MB) map identifying MBs that are within the ROI based on the information specifying the ROI, and wherein adjusting the ROI includes modifying status of MBs as being included in or excluded from the ROI based on the motion information.

16. The method of claim 1, wherein the information specifying an ROI includes textual or verbal information, the method further comprising defining the ROI based on the textual or verbal information.

17. The method of claim 16, wherein defining the ROI includes defining the ROI at an intermediate server in communication with at least one of the local device and the remote device.

18. A video encoding device comprising:

a region of interest (ROI) engine that receives information from a remote video communication device specifying a region of interest (ROI) within near-end video transmitted to the remote device; and

a video encoder that encodes the near-end video to enhance image quality of the ROI relative to non-ROI areas of the video.

19. The device of claim 18, wherein the video encoder transmits the encoded near-end video to the remote device, the device further comprising a video decoder that receives far-end video encoded by the remote device.

20. The device of claim 19, wherein the video decoder receives the information specifying the ROI with encoded far-end video received from the remote device, and wherein the information specifying the ROI is embedded within the encoded far-end video received from the remote device.

21. The device of claim 18, further comprising a video decoder that receives far-end video encoded by the remote device, wherein the video decoder receives the information specifying the ROI by out-of-band signaling from the remote device.

22. The device of claim 21, wherein the ROI engine generates information specifying an ROI within the encoded far-end video, and the video encoder transmits the ROI information with the encoded near-end video to the remote device.

23. The device of claim 18, further comprising a video decoder that decodes the encoded far-end video received from the remote device to enhance the image quality of the ROI in the far-end video relative to non-ROI areas of the far-end video.

24. The device of claim 23, wherein the video decoder applies higher quality post-processing or error concealment techniques to the ROI in the far-end video relative to non-ROI areas of the far-end video.

25. The device of claim 18, further comprising an ROI mapper module that generates a macroblock (MB) map identifying MBs that are within the ROI based on the information specifying the ROI, and an ROI controller that processes the information specifying the ROI for application to the ROI mapper module.

26. The device of claim 18, wherein the video encoder applies higher quality encoding or error protection techniques to the ROI in the near-end video relative to non-ROI areas of the near-end video.

27. The device of claim 18, further comprising an authentication module that authenticates a remote user associated with the remote device before the near-end video is encoded based on the ROI, wherein the authentication module determines whether the remote user is authorized to control encoding of the near-end video based on the ROI.

28. The device of claim 27, wherein the authentication module seeks, from a local user associated with the device, authorization for the remote user to control encoding of the near-end video based on the ROI.

29. The device of claim 18, wherein the information received from a remote device includes information from a plurality of remote devices specifying multiple ROIs within the near-end video, the system further comprising an authentication module that authenticates remote users associated with the remote devices to select one of the remote users to control encoding of the near-end video based on the respective ROI.

30. The device of claim 18, further comprising a tracking module that monitors motion information associated with the near-end video, and adjusts the ROI based on the motion information, wherein the encoder encodes the near-end video based on the adjusted ROI.

31. The device of claim 30, further comprising an ROI mapper module that generates a macroblock (MB) map identifying MBs that are within the ROI based on the information specifying the ROI, wherein adjustment of the ROI by the tracking module includes modification of status of MBs as being included in or excluded from the ROI based on the motion information.

32. The device of claim 18, wherein the information specifying an ROI includes textual or verbal information, the system further comprising an extraction module that defines the ROI based on the textual or verbal information.

33. The device of claim 18, wherein the information specifying an ROI includes textual or verbal information, the system further comprising an intermediate extraction server, situated remotely from the video communication device and the remote video communication device, that defines the ROI based on the textual or verbal information.

34. A computer-readable medium comprising instructions to cause a processor to:

receive information from a remote device specifying a region of interest (ROI) within near-end video encoded by a local device and received by the remote device; and

encode the near-end video to enhance image quality of the ROI relative to non-ROI areas of the video.

35. The computer-readable medium of claim 34, wherein the instructions cause the processor to transmit the encoded near-end video to the remote device, and receive, in the local device, far-end video encoded by the remote device.

36. The computer-readable medium of claim 34, wherein the instructions cause the processor to receive the information specifying the ROI with encoded far-end video received from the remote device, wherein the information specifying the ROI is embedded within the encoded far-end video.

37. The computer-readable medium of claim 34, wherein the instructions cause the processor to receive the information specifying the ROI by out-of-band signaling from the remote device.

38. The computer-readable medium of claim 34, wherein the instructions cause the processor to generate information specifying an ROI within encoded far-end video received from the remote device, and transmit the ROI information with the encoded near-end video to the remote device.

39. The computer-readable medium of claim 34, wherein the instructions cause the processor to decode the encoded far-end video received from the remote device to enhance the image quality of the ROI in the far-end video relative to non-ROI areas of the far-end video.

40. The computer-readable medium of claim 39, wherein the instructions cause the processor to decode the encoded far-end video by applying higher quality post-processing or error concealment techniques to the ROI in the far-end video relative to non-ROI areas of the far-end video.

41. The computer-readable medium of claim 34, wherein the information specifying the ROI includes a macroblock (MB) map identifying MBs that are within the ROI.

42. The computer-readable medium of claim 34, wherein the instructions cause the processor to encode the near-end video by applying higher quality encoding or error protection techniques to the ROI in the near-end video relative to non-ROI areas of the near-end video.

43. The computer-readable medium of claim 34, wherein the instructions cause the processor to determine whether the remote user is authorized to control encoding of the near-end video based on the ROI before encoding the near-end video based on the ROI, and wherein the instructions cause the processor to seek, from a local user associated with the local device, authorization for the remote user to control encoding of the near-end video based on the ROI.

44. The computer-readable medium of claim 34, wherein the information is received from a plurality of remote devices specifying multiple ROIs within the near-end video, and the instructions cause the processor to authenticate remote users associated with the remote devices to select one of the remote users to control encoding of the near-end video based on the respective ROI.

45. The computer-readable medium of claim 34, wherein the instructions cause the processor to:

monitor motion information associated with the encoded near-end video;

adjust the ROI based on the motion information; and

encode the near-end video based on the adjusted ROI.

46. The computer-readable medium of claim 45, wherein the information specifying the ROI includes a macroblock (MB) map identifying MBs that are within the ROI, and the instructions cause the processor to adjust the ROI includes modifying status of MBs as being included in or excluded from the ROI based on the motion information.

47. A method comprising:

generating information specifying a region of interest (ROI) within far-end video transmitted by a remote device and received by a local device; and

transmitting the information to the remote device for use in encoding the far-end video based on the ROI to enhance image quality of the ROI relative to non-ROI areas of the video.

48. The method of claim 47, wherein the information specifying the ROI is embedded within near-end video encoded by the local device and transmitted to the remote device.

49. The method of claim 47, wherein the information specifying the ROI is received by out-of-band signaling from the remote device.

50. The method of claim 47, wherein the information specifying the ROI includes a macroblock (MB) map identifying MBs that are within the ROI.

51. A video encoding device comprising:

a region of interest (ROI) engine that generates information specifying a region of interest (ROI) within far-end video received from a remote device; and

a video encoder that encodes near-end video and transmits information specifying the ROI with the encoded near-end video for use by the remote device in encoding the far-end video based on the ROI to enhance image quality of the ROI relative to non-ROI areas of the far-end video.

52. The device of claim 51, wherein the information specifying the ROI is embedded within the near-end video transmitted to the remote device.

53. The device of claim 51, wherein the information specifying the ROI is transmitted by out-of-band signaling to the remote device.

54. The device of claim 51, wherein the information specifying the ROI includes a macroblock (MB) map identifying MBs that are within the ROI.