Region of interest-based video transfer

• US 8,755,441 B2
• Filed: 05/10/2010
• Issued: 06/17/2014
• Est. Priority Date: 05/10/2010
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Mukherjee, Uddipan, Matsubayashi, Don Hideyasu
• Primary Examiner(s): Nguyen, Phuoc
• Assistant Examiner(s): John, Clarence
• Application Number:
  US12/777,187
• Publication Number:
  US 20110276652A1
• Time in Patent Office: 1,499 Days
• Field of Search: 709/201, 709/203, 709217-222, 709231-233, 709/236, 709/237, 709238-241, 375/240, 375/240.2
• US Class Current: 375/240.2
• CPC Class Codes: H04N 19/124 Quantisation
• CPC Class Codes: H04N 19/129 Scanning of coding units, e.g. zig-zag scan of transform coefficients or flexible macroblock ordering [FMO]
• CPC Class Codes: H04N 19/162 User input
• CPC Class Codes: H04N 19/164 Feedback from the receiver or from the transmission channel
• CPC Class Codes: H04N 19/167 Position within a video image, e.g. region of interest [ROI]
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/186 the unit being a colour or a chrominance component
• CPC Class Codes: H04N 19/192 the adaptation method, adaptation tool or adaptation type being iterative or recursive
• CPC Class Codes: H04N 19/37 with arrangements for assigning different transmission priorities to video input data or to video coded data
• CPC Class Codes: H04N 19/46 Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/61 in combination with predictive coding

Abstract

Image data for frames of a moving image is transmitted from a first computer to a second computer. A region of interest for all of the frames in the moving image is identified by calculations using the image data for at least some of the frames. Coordinates for the region of interest are transmitted. Image data for a first frame and second frame for the region of interest is encoded and transmitted. A region surrounding the region of interest is identified. Coordinates for the region surrounding the region of interest are transmitted. Image data for the first frame for the region surrounding the region of interest is encoded and transmitted, wherein transmission of the encoded image data for the first frame for the region surrounding the region of interest is effected after transmission of encoded image data for both of the first and second frames for the region of interest.

Claims

1. A method for transmitting image data for multiple frames of a moving image from a first computer to a second computer over a network, wherein the method comprises:

identifying a region of interest for all of the multiple frames in the moving image by the first computer, wherein the first computer identifies the region of interest by calculations using the image data for at least some of the multiple frames;

encoding image data for a first frame for the region of interest;

transmitting coordinates for the region of interest from the first computer to the second computer;

transmitting the encoded image data for the first frame for the region of interest from the first computer to the second computer for decoding and display by the second computer at a position corresponding to the transmitted coordinates for the region of interest;

encoding image data for a second frame subsequent in time to the first frame for the region of interest;

transmitting the encoded image data for the second frame for the region of interest from the first computer to the second computer for decoding and display by the second computer at a position corresponding to the transmitted coordinates for the region of interest;

identifying a region surrounding the region of interest by the first computer;

encoding image data for the first frame for the region surrounding the region of interest;

transmitting coordinates for the region surrounding the region of interest from the first computer to the second computer; and

transmitting the encoded image data for the first frame for the region surrounding the region of interest from the first computer to the second computer for decoding and display by the second computer at a position corresponding to the transmitted coordinates for the region surrounding the region of interest, wherein transmission of the encoded image data for the first frame for the region surrounding the region of interest is effected after transmission of encoded image data for both of the first frame for the region of interest and the second frame for the region of interest.

2. A method according to claim 1, further comprising a repetition of the steps of identifying a region of interest;

encoding image data for a first frame for the region of interest;

transmitting coordinates for the region of interest;

transmitting the encoded image data for the first frame for the region of interest;

encoding image data for a second frame subsequent in time to the first frame for the region of interest;

transmitting the encoded image data for the second frame for the region of interest, for each subsequent frame until encoded image data for the region of interest is transmitted for an entirety of the moving image, and further comprising a repetition of the steps of identifying a region surrounding the region of interest by the first computer;

encoding image data for the first frame for the region surrounding the region of interest;

transmitting coordinates for the region surrounding the region of interest; and

transmitting the encoded image data for the first frame for the region surrounding the region of interest, for each subsequent frame until encoded image data for the region surrounding the region of interest is transmitted, for an entirety of the moving image.

3. A method according to claim 1, wherein the encoding of the region of interest and the region surrounding the region of interest is adjusted in quality depending on limitations imposed by a user or the network.

4. A method according to claim 1, wherein the image quality of the image data is scaled for each region surrounding the region of interest based on the distance between the region and the region of interest.

5. A method according to claim 1, wherein the image data is encoded to produce a lossless transmission.

6. A method according to claim 1, wherein the image data is encoded to produce a lossy transmission.

7. A method according to claim 1, further comprising packaging of the encoded image data and the coordinates corresponding to the encoded image data into a data packet, one after the other, wherein the steps of transmitting coordinates and transmitting the encoded image data are comprised by transmitting the data packet.

8. A method according to claim 1, wherein the identification of the region of interest uses edge detection calculations on the image data for the some of the multiple frames.

9. A method according to claim 1, wherein the identification of the region of interest uses spatial frequency calculations on the image data for the some of the multiple frames.

10. A method according to claim 1, wherein the identification of the region of interest uses face detection calculations on the image data for the some of the multiple frames.

11. A method according to claim 1, wherein the identification of the region of interest incorporates machine learning techniques, wherein the machine learning techniques analyze samples of image data for the some of the multiple frames, and wherein the region of interest is identified based on patterns uncovered from analyzing the samples.

12. A method according to claim 1, wherein the identification of the region of interest incorporates data mining techniques, wherein the data mining techniques analyze samples of image data for the some of the multiple frames, and wherein the region of interest is identified based on patterns uncovered from analyzing the samples.

13. A method according to claim 1, wherein the identification of the region of interest uses metadata for the image data for the some of the multiple frames, wherein the metadata for the image data provides image content information for the image data, and wherein the region of interest is identified based on the image content information.

14. A method according to claim 1, wherein the identification of the region of interest uses block recognition techniques on the image data for the some of the multiple frames.

15. A method according to claim 1, wherein the encoding of image data for one or both of the region of interest and the regions other than the region of interest is performed in successive passes of increasing visual quality.

16. A method according to claim 1, wherein the region of interest and the region surrounding the region of interest are comprised of blocks of pixels, and wherein the steps of encoding and transmitting are performed on the blocks of pixels.

17. A method according to claim 16, wherein the blocks of pixels are transmitted in an order starting from the center of the region of interest and spiraling outwardly toward the edges of the region of interest.

18. A method according to claim 1, wherein, after transmission of encoded image data for the first frame for the region surrounding the region of interest, the first computer encodes and transmits image data for a second frame subsequent in time to the first frame for the region surrounding the region of interest to the second computer for decoding and display by the second computer at a position corresponding to the transmitted coordinates for the region surrounding the region of interest.

19. An image processing apparatus, comprising:

a computer-readable memory constructed to store computer-executable process steps; and

a processor constructed to execute the computer-executable process steps stored in the memory;

wherein the process steps stored in the memory cause the processor to transmit image data for multiple frames of a moving image from a first computer to a second computer over a network, and wherein the process steps include computer-executable process steps to;

identify a region of interest for all of the multiple frames in the moving image by the first computer, wherein the first computer identifies the region of interest by calculations using the image data for at least some of the multiple frames;

transmit coordinates for the region of interest from the first computer to the second computer;

encode image data for a first frame for the region of interest;

transmit the encoded image data for the first frame for the region of interest from the first computer to the second computer for decoding and display by the second computer at a position corresponding to the transmitted coordinates for the region of interest;

encode image data for a second frame subsequent in time to the first frame for the region of interest;

transmit the encoded image data for the second frame for the region of interest from the first computer to the second computer for decoding and display by the second computer at a position corresponding to the transmitted coordinates for the region of interest;

identify a region surrounding the region of interest by the first computer;

transmit coordinates for the region surrounding the region of interest from the first computer to the second computer;

encode image data for the first frame for the region surrounding the region of interest; and

transmit the encoded image data for the first frame for the region surrounding the region of interest from the first computer to the second computer for decoding and display by the second computer at a position corresponding to the transmitted coordinates for the region surrounding the region of interest, wherein transmission of the encoded image data for the first frame for the region surrounding the region of interest is effected after transmission of encoded image data for both of the first frame for the region of interest and the second frame for the region of interest.

20. An apparatus according to claim 19, further comprising a repetition of the steps of identifying a region of interest;

encoding image data for a first frame for the region of interest;

transmitting coordinates for the region of interest;

transmitting the encoded image data for the first frame for the region of interest;

encoding image data for a second frame subsequent in time to the first frame for the region of interest;

transmitting the encoded image data for the second frame for the region of interest, for each subsequent frame until encoded image data for the region of interest is transmitted for an entirety of the moving image, and further comprising a repetition of the steps of identifying a region surrounding the region of interest by the first computer;

encoding image data for the first frame for the region surrounding the region of interest;

transmitting coordinates for the region surrounding the region of interest; and

transmitting the encoded image data for the first frame for the region surrounding the region of interest, for each subsequent frame until encoded image data for the region surrounding the region of interest is transmitted, for an entirety of the moving image.

21. An apparatus according to claim 19, wherein the encoding of the region of interest and the region surrounding the region of interest is adjusted in quality depending on limitations imposed by a user or the network.

22. An apparatus according to claim 19, wherein the image quality of the image data is scaled for each region surrounding the region of interest based on the distance between the region and the region of interest.

23. An apparatus according to claim 19, wherein the image data is encoded to produce a lossless transmission.

24. An apparatus according to claim 19, wherein the image data is encoded to produce a lossy transmission.

25. An apparatus according to claim 19, further comprising packaging of the encoded image data and the coordinates corresponding to the encoded image data into a data packet, one after the other, wherein the steps of transmitting coordinates and transmitting the encoded image data are comprised by transmitting the data packet.

26. An apparatus according to claim 19, wherein the identification of the region of interest uses edge detection calculations on the image data for the some of the multiple frames.

27. An apparatus according to claim 19, wherein the identification of the region of interest uses spatial frequency calculations on the image data for the some of the multiple frames.

28. An apparatus according to claim 19, wherein the identification of the region of interest uses face detection calculations on the image data for the some of the multiple frames.

29. An apparatus according to claim 19, wherein the identification of the region of interest incorporates machine learning techniques, wherein the machine learning techniques analyze samples of image data for the some of the multiple frames, and wherein the region of interest is identified based on patterns uncovered from analyzing the samples.

30. An apparatus according to claim 19, wherein the identification of the region of interest incorporates data mining techniques, wherein the data mining techniques analyze samples of image data for the some of the multiple frames, and wherein the region of interest is identified based on patterns uncovered from analyzing the samples.

31. An apparatus according to claim 19, wherein the identification of the region of interest uses metadata for the image data for the some of the multiple frames, wherein the metadata for the image data provides image content information for the image data, and wherein the region of interest is identified based on the image content information.

32. An apparatus according to claim 19, wherein the identification of the region of interest uses block recognition techniques on the image data for the some of the multiple frames.

33. An apparatus according to claim 19, wherein the encoding of image data for one or both of the region of interest and the regions other than the region of interest is performed in successive passes of increasing visual quality.

34. An apparatus according to claim 19, wherein the region of interest and the region surrounding the region of interest are comprised of blocks of pixels, and wherein the steps of encoding and transmitting are performed on the blocks of pixels.

35. An apparatus according to claim 34, wherein the blocks of pixels are transmitted in an order starting from the center of the region of interest and spiraling outwardly toward the edges of the region of interest.

36. An apparatus according to claim 19, wherein, after transmission of encoded image data for the first frame for the region surrounding the region of interest, the first computer encodes and transmits image data for a second frame subsequent in time to the first frame for the region surrounding the region of interest to the second computer for decoding and display by the second computer at a position corresponding to the transmitted coordinates for the region surrounding the region of interest.

37. A non-transitory computer-readable storage medium storing a computer-executable program for causing a computer to perform a method for transmitting image data for multiple frames of a moving image from a first computer to a second computer over a network, the method comprising:

identifying a region of interest for all of the multiple frames in the moving image by the first computer, wherein the first computer identifies the region of interest by calculations using the image data for at least some of the multiple frames;

transmitting coordinates for the region of interest from the first computer to the second computer;

encoding image data for a first frame for the region of interest;

transmitting the encoded image data for the first frame for the region of interest from the first computer to the second computer for decoding and display by the second computer at a position corresponding to the transmitted coordinates for the region of interest;

encoding image data for a second frame subsequent in time to the first frame for the region of interest;

transmitting the encoded image data for the second frame for the region of interest from the first computer to the second computer for decoding and display by the second computer at a position corresponding to the transmitted coordinates for the region of interest;

identifying a region surrounding the region of interest by the first computer;

transmitting coordinates for the region surrounding the region of interest from the first computer to the second computer;

encoding image data for the first frame for the region surrounding the region of interest; and

transmitting the encoded image data for the first frame for the region surrounding the region of interest from the first computer to the second computer for decoding and display by the second computer at a position corresponding to the transmitted coordinates for the region surrounding the region of interest, wherein transmission of the encoded image data for the first frame for the region surrounding the region of interest is effected after transmission of encoded image data for both of the first frame for the region of interest and the second frame for the region of interest.

38. An image processing module for transmitting image data for multiple frames of a moving image from a first computer to a second computer over a network, the image processing module being non-transitory and comprising:

an identification module constructed to identify a region of interest for all of the multiple frames in the moving image by the first computer, wherein the first computer identifies the region of interest by calculations using the image data for at least some of the multiple frames, and wherein the identification module is further constructed to identify a region surrounding the region of interest by the first computer;

a data transmitting module constructed to transmit coordinates for the region of interest from the first computer to the second computer, and wherein the data transmitting module is further constructed to transmit coordinates for the region surrounding the region of interest from the first computer to the second computer;

an encoding module constructed to encode image data for a first frame for the region of interest, wherein the encoding module is further constructed to encode image data for a second frame subsequent in time to the first frame for the region of interest, and to encode image data for a first frame for the region surrounding the region of interest;

wherein the data transmitting module is further constructed to transmit the encoded image data for the first frame for the region of interest from the first computer to the second computer for decoding and display by the second computer at a position corresponding to the transmitted coordinates for the region of interest, to transmit the encoded image data for the second frame for the region of interest from the first computer to the second computer for decoding and display by the second computer at a position corresponding to the transmitted coordinates for the region of interest, and to transmit the encoded image data for the first frame for the region surrounding the region of interest from the first computer to the second computer for decoding and display by the second computer at a position corresponding to the transmitted coordinates for the region surrounding the region of interest, wherein transmission of the encoded image data for the first frame for the region surrounding the region of interest is effected after transmission of encoded image data for both of the first frame for the region of interest and the second frame for the region of interest.

39. A method for receiving image data for multiple frames of a moving image from a first computer at a second computer over a network, wherein the method comprises:

receiving coordinates for a region of interest from the first computer at the second computer;

receiving encoded image data for a first frame for the region of interest;

decoding the encoded image data for the first frame for the region of interest;

displaying the image data for the first frame for the region of interest at a position corresponding to the received coordinates for the region of interest;

receiving encoded image data for a second frame subsequent in time to the first frame for the region of interest;

decoding the encoded image data for the second frame for the region of interest;

displaying the image data for the second frame for the region of interest at a position corresponding to the received coordinates for the region of interest;

receiving coordinates for a region surrounding the region of interest from the first computer at the second computer;

receiving encoded image data for a first frame for the region surrounding the region of interest, wherein receipt of the encoded image data for the first frame for the region surrounding the region of interest is effected after receipt of encoded image data for both of the first frame for the region of interest and the second frame for the region of interest;

decoding the encoded image data for the first frame for the region surrounding the region of interest; and

displaying the image data for the first frame for the region surrounding the region of interest at a position corresponding to the received coordinates for the region surrounding the region of interest.

40. A method according to claim 39, further comprising a repetition of the steps of receiving coordinates for a region of interest;

receiving encoded image data for a first frame for the region of interest;

decoding the encoded image data for the first frame for the region of interest;

displaying the image data for the first frame for the region of interest at a position corresponding to the received coordinates for the region of interest;

receiving encoded image data for a second frame subsequent in time to the first frame for the region of interest;

decoding the encoded image data for the second frame for the region of interest;

displaying the image data for the second frame for the region of interest at a position corresponding to the received coordinates for the region of interest, for each subsequent frame until encoded image data for the region of interest is received for an entirety of the moving image, and further comprising a repetition of the steps of receiving coordinates for a region surrounding the region of interest;

receiving encoded image data for a first frame for the region surrounding the region of interest;

decoding the encoded image data for the first frame for the region surrounding the region of interest; and

displaying the image data for the first frame for the region surrounding the region of interest at a position corresponding to the received coordinates for the region surrounding the region of interest, for each subsequent frame until encoded image data for the region surrounding the region of interest is received, for an entirety of the moving image.

41. A method according to claim 39, further comprising receiving the encoded image data and the coordinates corresponding to the encoded image data as a data packet and unpackaging the data packet to retrieve the encoded image data and the coordinates corresponding to the encoded image data, wherein the steps of receiving the encoded image data and the coordinates corresponding to the encoded image data are comprised by receiving the data packet.

42. A method according to claim 39, wherein the region of interest and the region surrounding the region of interest are comprised of blocks of pixels, and wherein the steps of receiving, decoding, and displaying are performed on the blocks of pixels.

43. A method according to claim 42, wherein the blocks of pixels are received in an order starting from the center of the region of interest and spiraling outwardly toward the edges of the region of interest.

44. A method according to claim 39, wherein the region of interest is identified by the first computer by calculations using the image data.

45. A method according to claim 39, wherein, after receipt of encoded image data for the first frame for the region surrounding the region of interest, the second computer receives, decodes, and displays image data for a second frame subsequent in time to the first frame for the region surrounding the region of interest at a position corresponding to the received coordinates for the region surrounding the region of interest.

46. An image processing apparatus, comprising:

a computer-readable memory constructed to store computer-executable process steps; and

a processor constructed to execute the computer-executable process steps stored in the memory;

wherein the process steps stored in the memory cause the processor to receive image data for multiple frames of a moving image from a first computer at a second computer over a network, and wherein the process steps include computer-executable process steps to;

receive coordinates for a region of interest from the first computer at the second computer;

receive encoded image data for a first frame for the region of interest at the second computer for decoding and display by the second computer at a position corresponding to the received coordinates for the region of interest;

decode the image data for the first frame for the region of interest;

display the image data for the first frame for the region of interest at a position corresponding to the received coordinates for the region of interest;

receive encoded image data for a second frame subsequent in time to the first frame for the region of interest;

decode the image data for the second frame for the region of interest;

display the image data for the second frame for the region of interest at a position corresponding to the received coordinates for the region of interest;

receive coordinates for a region surrounding the region of interest from the first computer at the second computer;

receive the image data for a first frame for the region surrounding the region of interest at the second computer for decoding and display by the second computer at a position corresponding to the received coordinates for the region of interest, wherein receipt of the encoded image data for the first frame for the region surrounding the region of interest is effected after receipt of encoded image data for both of the first frame for the region of interest and the second frame for the region of interest;

decode the image data for the first frame for the region surrounding the region of interest; and

display the image data for the first frame for the region surrounding the region of interest at a position corresponding to the received coordinates for the region surrounding the region of interest.

47. An apparatus according to claim 46, further comprising a repetition of the steps of receiving coordinates for a region of interest;

receiving encoded image data for a first frame for the region of interest;

decoding the encoded image data for the first frame for the region of interest;

displaying the image data for the first frame for the region of interest at a position corresponding to the received coordinates for the region of interest;

receiving encoded image data for a second frame subsequent in time to the first frame for the region of interest;

decoding the encoded image data for the second frame for the region of interest;

displaying the image data for the second frame for the region of interest at a position corresponding to the received coordinates for the region of interest, for each subsequent frame until encoded image data for the region of interest is received for an entirety of the moving image, and further comprising a repetition of the steps of receiving coordinates for a region surrounding the region of interest;

receiving encoded image data for a first frame for the region surrounding the region of interest;

decoding the encoded image data for the first frame for the region surrounding the region of interest; and

displaying the image data for the first frame for the region surrounding the region of interest at a position corresponding to the received coordinates for the region surrounding the region of interest, for each subsequent frame until encoded image data for the region surrounding the region of interest is received, for an entirety of the moving image.

48. An apparatus according to claim 46, further comprising receiving the encoded image data and the coordinates corresponding to the encoded image data as a data packet and unpackaging the data packet to retrieve the encoded image data and the coordinates corresponding to the encoded image data, wherein the steps of receiving the encoded image data and the coordinates corresponding to the encoded image data are comprised by receiving the data packet.

49. An apparatus according to claim 46, wherein the region of interest and the region surrounding the region of interest are comprised of blocks of pixels, and wherein the steps of receiving, decoding, and displaying are performed on the blocks of pixels.

50. An apparatus according to claim 49, wherein the blocks of pixels are received in an order starting from the center of the region of interest and spiraling outwardly toward the edges of the region of interest.

51. An apparatus according to claim 46, wherein the region of interest is identified by the first computer by calculations using the image data.

52. An apparatus according to claim 46, wherein, after receipt of encoded image data for the first frame for the region surrounding the region of interest, the second computer receives, decodes, and displays image data for a second frame subsequent in time to the first frame for the region surrounding the region of interest at a position corresponding to the received coordinates for the region surrounding the region of interest.

53. A non-transitory computer-readable storage medium storing a computer-executable program for causing a computer to perform a method for receiving image data of an image from a first computer at a second computer over a network, the method comprising:

receiving coordinates for a region of interest from the first computer at the second computer;

receiving encoded image data for a first frame for the region of interest at the second computer for display by the second computer at a position corresponding to the received coordinates for the region of interest;

decoding the encoded image data for the first frame for the region of interest;

displaying the image data for the first frame for the region of interest at a position corresponding to the received coordinates for the region of interest;

receiving encoded image data for a second frame subsequent in time to the first frame for the region of interest;

decoding the encoded image data for the second frame for the region of interest;

displaying the image data for the second frame for the region of interest at a position corresponding to the received coordinates for the region of interest;

receiving coordinates for a region surrounding the region of interest from the first computer at the second computer;

receiving encoded image data for a first frame for the region surrounding the region of interest at the second computer for display by the second computer at a position corresponding to the transmitted coordinates for the region of interest, wherein receipt of the encoded image data for the first frame for the region surrounding the region of interest is effected after receipt of encoded image data for both of the first frame for the region of interest and the second frame for the region of interest;

decoding the encoded image data for the first frame for the region surrounding the region of interest; and

displaying the image data for the first frame for the region surrounding the region of interest at a position corresponding to the received coordinates for the region surrounding the region of interest.

54. An image processing module for receiving image data for multiple frames of a moving image from a first computer at a second computer over a network, the image processing module being non-transitory and comprising:

a data receiving module that is constructed to receive coordinates for a region of interest from the first computer at the second computer, wherein the data receiving module is further constructed to receive encoded image data for a first frame for the region of interest at the second computer for display by the second computer at a position corresponding to the received coordinates for the region of interest, to receive encoded image data for a second frame subsequent in time to the first frame for the region of interest, to receive coordinates for a region surrounding the region of interest from the first computer at the second computer, and to receive encoded image data for a first frame for the region surrounding the region of interest at the second computer for display by the second computer at a position corresponding to the transmitted coordinates for the region of interest, wherein receipt of the encoded image data for the first frame for the region surrounding the region of interest is effected after receipt of encoded image data for both of the first frame for the region of interest and the second frame for the region of interest;

a decoding module for decoding the encoded image data for the first frame for the region of interest, wherein the decoding module is further constructed to decode the encoded image data for the second frame for the region of interest, and to decode the encoded image data for the first frame for the region surrounding the region of interest;

a display module for displaying the image data for the first frame for the region of interest at a position corresponding to the received coordinates for the region of interest, wherein the display module is further constructed to display image data for the second frame for the region of interest at a position corresponding to the received coordinates for the region of interest, and to display image data for the first frame for the region surrounding the region of interest at a position corresponding to the received coordinates for the region surrounding the region of interest.