Multi-resolution predictive foveation for bandwidth reduction of moving images

• US 7,499,594 B2
• Filed: 10/23/2002
• Issued: 03/03/2009
• Est. Priority Date: 04/15/2002
• Status: Expired due to Fees

• Current Assignee: Gamehancement LLC (Jeffrey M. Gross)
• Original Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Inventors: Sullivan, Marc A., Kortum, Philip T.
• Primary Examiner(s): Bella; Matthew C
• Assistant Examiner(s): Ge; Yuzhen
• Application Number:
  US10/279,778
• Publication Number:
  US 20030194142A1
• Time in Patent Office: 2,323 Days
• Field of Search: 382/239
• US Class Current: 382/239
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/47 Detecting features for summarising video content
• CPC Class Codes: H04N 19/162 User input
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object
• CPC Class Codes: H04N 19/59 involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution

Abstract

A scene of an image sequence is compressed using a number of foveation zones, each foveation zone being weighted based on a probability of a viewer looking at a corresponding portion of the first scene. Each foveation zone in the scene has a corresponding resolution to ensure that at least one object therein is viewer-recognizable. Another scene of the image sequence is compressed using a different number of foveation zones, each foveation zone being weighted based on a probability of a viewer looking at a corresponding portion of the second scene. Each foveation zone in the other scene has a corresponding resolution to ensure that at least one object therein is viewer-recognizable.

Claims

1. A method comprising:

compressing using a processor a first scene of an image sequence relating to a first view location based on a first plurality of foveation zones, each of the first plurality of foveation zones being weighted based on a probability of a viewer looking at a corresponding portion of the first scene, each of the first plurality of foveation zones having a corresponding resolution so that at least one object therein is viewer-recognizable; and

compressing using the processor a second scene of the image sequence relating to a second view location based on a second plurality of foveation zones, each of the second plurality of foveation zones being weighted based on a probability of a viewer looking at a corresponding portion of the second scene, each of the second plurality of foveation zones having a corresponding resolution so that at least one object therein is viewer-recognizable;

wherein at least a portion of the first scene and the second scene of the image sequence occur in the image sequence at the same time such that at least a portion of the image sequence is compressed based on both the first scene and the second scene; and

wherein during the portion of the image sequence that is compressed based on both the first scene and the second scene, the image sequence is compressed based on a probability that a viewer is looking at the first view location vs. the second view location.

2. The method of claim 1 wherein the corresponding resolution of each of the first plurality of foveation zones ensures that all objects in at least half of the zone'"'"'s area are viewer-recognizable.

3. The method of claim 1 wherein the corresponding resolution of each of the first plurality of foveation zones ensures that substantially all of the objects therein are viewer-recognizable.

4. The method of claim 1 further comprising:

wherein the first plurality of foveation zones consists of a different number of foveation zones than the second plurality of foveation zones.

5. The method of claim 1 further comprising:

determining view locations within the first scene for a sample of viewers;

determining the first plurality of foveation zones based on the view locations;

estimating the probability for each of the foveation zones based on the view locations; and

weighting each of the foveation zones based on the probability.

6. The method of claim 5 further comprising:

generating a compressed version of the image sequence based on said compressing; and

transmitting the compressed version to viewers whose view locations within the image sequence are not monitored.

7. The method of claim 1 further comprising:

determining the first plurality of foveation zones based on a scene type for the first scene;

estimating the probability for each of the foveation zones based on the scene type for the first scene; and

weighting each of the foveation zones based on the probability and a minimum acceptable resolution.

8. The method of claim 1 wherein each of the first plurality of foveation zones is definable by a continuous falloff function.

9. The method of claim 1 wherein each of the first plurality of foveation zones has a corresponding half-resolution constant based on the probability of a viewer looking at the corresponding portion of the first scene.

10. The method of claim 9 wherein the first plurality of foveation zones comprises a first foveation zone and a second foveation zone, wherein the first foveation zone has a greater probability of a viewer looking at its corresponding portion than the second foveation zone, and wherein the half-resolution constant of the first foveation zone is greater than the half-resolution constant of the second foveation zone.

11. The method of claim 1 wherein each of the first plurality of foveation zones is definable by a discrete function.

12. The method of claim 1 wherein each of the first plurality of foveation zones has a corresponding fixed resolution based on the probability of a viewer looking at the corresponding portion of the first scene and a minimum acceptable resolution.

13. The method of claim 12 wherein the first plurality of foveation zones comprises a first foveation zone and a second foveation zone, wherein the first foveation zone has a greater probability of a viewer looking at its corresponding portion than the second foveation zone, and wherein the resolution of the first foveation zone is greater than the resolution of the second foveation zone.

14. A computer-readable medium having a stored computer-readable program to direct a computer to perform an act of:

compressing a first scene of an image sequence relating to a first view location based on a first plurality of foveation zones, each of the first plurality of foveation zones being weighted based on a probability of a viewer looking at a corresponding portion of the first scene, each of the first plurality of foveation zones having a corresponding resolution so that at least one object therein is viewer-recognizable; and

compressing a second scene of the image sequence relating to a second view location based on a second plurality of foveation zones, each of the second plurality of foveation zones being weighted based on a probability of a viewer looking at a corresponding portion of the second scene, each of the second plurality of foveation zones having a corresponding resolution so that at least one object therein is viewer-recognizable;

wherein at least a portion of the first scene and the second scene of the image sequence occur in the image sequence at the same time such that at least a portion of the image sequence is compressed based on both the first scene and the second scene; and

wherein during the portion of the image sequence that is compressed based on both the first scene and the second scene, the image sequence is compressed based on a probability that a viewer is looking at the first view location vs. the second view location.

15. The computer-readable medium of claim 14 wherein the corresponding resolution of each of the first plurality of foveation zones ensures that all objects in at least half of the zone'"'"'s area are viewer-recognizable.

16. The computer-readable medium of claim 14 wherein the corresponding resolution of each of the first plurality of foveation zones ensures that substantially all of the objects therein are viewer-recognizable.

17. The computer-readable medium of claim 14 wherein the computer-readable program further directs the computer to perform an act of:

wherein the first plurality of foveation zones consists of a different number of foveation zones than the second plurality of foveation zones.

18. The computer-readable medium of claim 14 wherein the computer-readable program further directs the computer to perform acts of:

determining view locations within the first scene for a sample of viewers;

determining the first plurality of foveation zones based on the view locations;

estimating the probability for each of the foveation zones based on the view locations; and

weighting each of the foveation zones based on the probability and a minimum acceptable resolution.

19. The computer-readable medium of claim 16 wherein the computer-readable program further directs the computer to perform acts of:

generating a compressed version of the image sequence based on said compressing; and

transmitting the compressed version to viewers whose view locations within the image sequence are not monitored.

20. The computer-readable medium of claim 14 wherein the computer-readable program further directs the computer to perform acts of:

determining the first plurality of foveation zones based on a scene type for the first scene;

estimating the probability for each of the foveation zones based on the scene type for the first scene; and

weighting each of the foveation zones based on the probability and a minimum acceptable resolution.

21. The computer-readable medium of claim 14 wherein each of the first plurality of foveation zones is definable by a continuous falloff function.

22. The computer-readable medium of claim 14 wherein each of the first plurality of foveation zones has a corresponding half-resolution constant based on the probability of a viewer looking at the corresponding portion of the first scene.

23. The computer-readable medium of claim 22 wherein the first plurality of foveation zones comprises a first foveation zone and a second foveation zone, wherein the first foveation zone has a greater probability of a viewer looking at its corresponding portion than the second foveation zone, and wherein the half-resolution constant of the first foveation zone is greater than the half-resolution constant of the second foveation zone.

24. The computer-readable medium of claim 14 wherein each of the first plurality of foveation zones is definable by a discrete function.

25. The computer-readable medium of claim 14 wherein each of the first plurality of foveation zones has a corresponding fixed resolution based on the probability of a viewer looking at the corresponding portion of the first scene and a minimum acceptable resolution.

26. The computer-readable medium of claim 25 wherein the first plurality of foveation zones comprises a first foveation zone and a second foveation zone, wherein the first foveation zone has a greater probability of a viewer looking at its corresponding portion than the second foveation zone, and wherein the resolution of the first foveation zone is greater than the resolution of the second foveation zone.

27. An apparatus comprising:

a processor, an image compressor to compress a first scene of an image sequence relating to a first view location based on a first plurality of foveation zones, each of the first plurality of foveation zones being weighted based on a probability of a viewer looking at a corresponding portion of the first scene, each of the first plurality of foveation zones having a corresponding resolution so that at least one object therein is viewer-recognizable and to compress a second scene of the image sequence relating to a second view location based on a second plurality of foveation zones, each of the second plurality of foveation zones being weighted based on a probability of a viewer looking at a corresponding portion of the second scene, each of the second plurality of foveation zones having a corresponding resolution so that at least one object therein is viewer-recognizable;

wherein at least a portion of the first scene and the second scene of the image sequence occur in the image sequence at the same time such that at least a portion of the image sequence is compressed based on both the first scene and the second scene; and

wherein during the portion of the image sequence that is compressed based on both the first scene and the second scene, the image sequence is compressed based on a probability that a viewer is looking at the first view location vs. the second view location.

28. The apparatus of claim 27 wherein the corresponding resolution of each of the first plurality of foveation zones ensures that all objects in at least half of the zone'"'"'s area are viewer-recognizable.

29. The apparatus of claim 27 wherein the corresponding resolution of each of the first plurality of foveation zones ensures that substantially all of the objects therein are viewer-recognizable.

30. The apparatus of claim 27 wherein the first plurality of foveation zones consists of a different number of foveation zones than the second plurality of foveation zones.

31. The apparatus of claim 27 further comprising:

at least one eye tracking device to determine view locations within the first scene for a sample of viewers; and

a processor responsive to the at least one eye tracking device to determine the first plurality of foveation zones based on the view locations, to estimate the probability for each of the foveation zones based on the view locations, and to weight each of the foveation zones based on the probability and a minimum acceptable resolution.

32. The apparatus of claim 31 wherein the image processor generates a compressed version of the image sequence, the apparatus further comprising:

a transmitter to transmit the compressed version to viewers whose view locations within the image sequence are not monitored.

33. The apparatus of claim 27 further comprising:

a processor to determine the first plurality of foveation zones based on a scene type for the first scene, to estimate the probability for each of the foveation zones based on the scene type for the first scene, and to weight each of the foveation zones based on the probability and a minimum acceptable resolution.

34. The apparatus of claim 27 wherein each of the first plurality of foveation zones is definable by a continuous falloff function.

35. The apparatus of claim 27 wherein each of the first plurality of foveation zones has a corresponding half-resolution constant based on the probability of a viewer looking at the corresponding portion of the first scene.

36. The apparatus of claim 35 wherein the first plurality of foveation zones comprises a first foveation zone and a second foveation zone, wherein the first foveation zone has a greater probability of a viewer looking at its corresponding portion than the second foveation zone, and wherein the half-resolution constant of the first foveation zone is greater than the half-resolution constant of the second foveation zone.

37. The apparatus of claim 27 wherein each of the first plurality of foveation zones is definable by a discrete function.

38. The apparatus of claim 27 wherein each of the first plurality of foveation zones has a corresponding fixed resolution based on the probability of a viewer looking at the corresponding portion of the first scene and a minimum acceptable resolution.

39. The apparatus of claim 27 wherein the first plurality of foveation zones comprises a first foveation zone and a second foveation zone, wherein the first foveation zone has a greater probability of a viewer looking at its corresponding portion than the second foveation zone, and wherein the resolution of the first foveation zone is greater than the resolution of the second foveation zone.

40. A method comprising:

compressing using a processor a first scene of an image sequence based on a first plurality of foveation zones relating to a probability of a viewer looking at a first view location; and

compressing using the processor a second scene of the image sequence based on a second plurality of foveation zones relation to a probability of a viewer looking at a second view location;

wherein at least a portion of the first scene and the second scene of the image sequence occur concurrently;

wherein during the portion of the first and second scenes of the image sequence that occur concurrently, the image sequence is compressed based on both the first and second view locations and based on a probability that a viewer is looking at the first view location vs. the second view location.