Multi-point predictive foveation for bandwidth reduction of moving images

• US 7,251,373 B2
• Filed: 12/21/2005
• Issued: 07/31/2007
• Est. Priority Date: 04/15/2002
• Status: Expired due to Term

• Current Assignee: Gamehancement LLC (Jeffrey M. Gross)
• Original Assignee: SBC Technology Resources, Inc. (AT&T, Inc.)
• Inventors: Sullivan, Marc Andrew, Kortum, Philip Ted
• Primary Examiner(s): Couso; Jose L.
• Application Number:
  US11/313,335
• Publication Number:
  US 20060098882A1
• Time in Patent Office: 587 Days
• Field of Search: 382232-233, 382238-240, 382/236, 382/243, 382244-252, 348/384.1, 348/387.1, 348/390.1, 375/240, 375/240.01
• US Class Current: 382/239
• CPC Class Codes: H04N 19/162 User input
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object
• CPC Class Codes: H04N 19/59 involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution

Abstract

A scene of an image sequence is compressed using a number of foveation zones, each foveation zone being weighted based on a probability of a viewer looking at a corresponding portion of the first scene. Another scene of the image sequence is compressed using a different number of foveation zones, each foveation zone being weighted based on a probability of a viewer looking at a corresponding portion of the second scene.

Claims

1. A method comprising:

compressing an image sequence based on a plurality of foveation zones, wherein a first foveation zone of the plurality of foveation zones corresponds to a first view location and a second foveation zone of the plurality of foveation zones corresponds to a second view location, and wherein during a portion of the image sequence, the first and second view locations occur concurrently.

2. The method of claim 1 further comprising:

determining view locations within the plurality of foveation zones for a sample of viewers.

3. The method of claim 2 further comprising:

generating a compressed version of the image sequence based on said compressing; and

transmitting the compressed version to viewers whose view locations within the image sequence are not monitored.

4. The method of claim 1 further comprising:

estimating a probability of a viewer looking at the first view location and the second view location based at least in part on a scene type occurring during the portion of the image sequence where the first and second view locations occur concurrently; and

weighting at least the first and second foveation zones based on the probability.

5. The method of claim 1 wherein each of the plurality of foveation zones is definable by a continuous falloff function.

6. The method of claim 1 wherein each of the plurality of foveation zones has a corresponding half-resolution constant based on a probability of a viewer looking at the view location corresponding the foveation zone.

7. The method of claim 6 wherein the probability of a viewer looking at the first view location corresponding to the first foveation zone is greater than the probability of a viewer looking at the second view location corresponding to the second foveation zone, and wherein the half-resolution constant of the first foveation zone is greater than the half-resolution constant of the second foveation zone.

8. The method of claim 1 wherein each of the plurality of foveation zones is definable by a discrete function.

9. The method of claim 1 wherein each of the plurality of foveation zones has a corresponding fixed resolution based on a probability of a viewer looking at a view location corresponding to the foveation zone.

10. A computer-readable medium having computer-readable instructions to direct a computer to perform an act of:

compressing an image sequence based on a plurality of foveation zones, wherein a first foveation zone of the plurality of foveation zones corresponds to a first view location and a second foveation zone of the plurality of foveation zones corresponds to a second view location, and wherein during a portion of the image sequence, the first and second view locations occur concurrently.

11. The computer-readable medium of claim 10 wherein the computer-readable instructions further direct the computer to perform acts of:

determining view locations within the plurality of foveation zones for a sample of viewers.

12. The computer-readable medium of claim 11 wherein the computer-readable instructions further direct the computer to perform acts of:

generating a compressed version of the image sequence based on said compressing; and

transmitting the compressed version to viewers whose view locations within the image sequence are not monitored.

13. The computer-readable medium of claim 10 wherein the computer-readable instructions further direct the computer to perform acts of:

estimating a probability of a viewer looking at the first view location and the second view location based at least in part on a scene type occurring during the portion of the image sequence where the first and second view locations occur concurrently; and

weighting at least the first and second foveation zones based on the probability.

14. The computer-readable medium of claim 10 wherein each of the plurality of foveation zones is definable by a continuous falloff function.

15. The computer-readable medium of claim 10 wherein each of the plurality of foveation zones has a corresponding half-resolution constant based on a probability of a viewer looking at a view location corresponding to the foveation zone.

16. The computer-readable medium of claim 15 wherein the probability of a viewer looking at the first view location corresponding to the first foveation zone is greater than the probability of the viewer looking at the second view location corresponding to the second foveation zone, and wherein the half-resolution constant of the first foveation zone is greater than the half-resolution constant of the second foveation zone.

17. The computer-readable medium of claim 10 wherein each of the plurality of foveation zones is definable by a discrete function.

18. The computer-readable medium of claim 10 wherein each of the plurality of foveation zones has a corresponding fixed resolution based on a probability of a viewer looking at a view location corresponding to the foveation zone.

19. An apparatus comprising:

an image compressor to compress an image sequence based on a plurality of foveation zones, wherein a first foveation zone of the plurality of foveation zones corresponds to a first view location and a second foveation zone of the plurality of foveation zones corresponds to a second view location, and wherein during at least a portion of the image sequence, the first and second view locations occur concurrently.

20. The apparatus of claim 19 further comprising:

at least one eye tracking device to determine view locations for a sample of viewers; and

a processor responsive to the at least one eye tracking device to determine the plurality of foveation zones based on the view locations, to estimate a probability of a viewer looking at the view locations, and to weight each of the foveation zones based on the probability of the viewer looking at the view locations.

21. The apparatus of claim 20 wherein the image processor generates a compressed version of the image sequence, the apparatus further comprising:

a transmitter to transmit the compressed version to viewers whose view locations within the image sequence are not monitored.

22. The apparatus of claim 19 further comprising:

a processor to determine the plurality of foveation zones based on scene types within the image sequence, to estimate a probability of a view looking at each of the foveation zones based on the scene types within the image sequence, and to weight each of the foveation zones based on the probability.

23. The apparatus of claim 19 wherein each of the plurality of foveation zones is definable by a continuous falloff function.

24. The apparatus of claim 19 wherein each of the plurality of foveation zones has a corresponding half-resolution constant based on a probability of a viewer looking at a view location corresponding to each of the foveation zone.

25. The apparatus of claim 24 wherein a probability of a viewer looking at the view first location corresponding to the first foveation zone is greater than a probability of the viewer looking at the second view location corresponding to the second foveation zone, and wherein the half-resolution constant of the first foveation zone is greater than the half-resolution constant of the second foveation zone.

26. The apparatus of claim 19 wherein each of the plurality of foveation zones is definable by a discrete function.

27. The apparatus of claim 19 wherein each of the plurality of foveation zones has a corresponding fixed resolution based on a probability of a viewer looking at a view location corresponding to the foveation zone.

28. The apparatus of claim 27 wherein the probability of a viewer looking at the first view location corresponding to the first foveation zone is greater than the probability of the viewer looking at the second view location corresponding to the second foveation zone, and wherein the resolution of the first foveation zone is greater than the resolution of the second foveation zone.