Arbitrary object tracking augmented reality applications

• US 7,050,078 B2
• Filed: 12/19/2002
• Issued: 05/23/2006
• Est. Priority Date: 12/19/2002
• Status: Active Grant

• Current Assignee: Accenture Global Services Limited (Accenture PLC)
• Original Assignee: Accenture Global Services GmbH (Accenture PLC)
• Inventors: Dempski, Kelly L.
• Primary Examiner(s): Shankar, Vijay
• Assistant Examiner(s): DHARIA, PRABODH M
• Application Number:
  US10/326,158
• Publication Number:
  US 20040119662A1
• Time in Patent Office: 1,251 Days
• Field of Search: 345 7- 9, 345/420, 345/435, 345/473, 345/700, 382/103, 382/154, 382/284, 364/514
• US Class Current: 715/700
• CPC Class Codes: G01S 3/7864 T.V. type tracking systems
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 2027/0198 System for aligning or maintaining alignment of an image in a predetermined direction
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042

Abstract

Video images of objects in a real-world environment are taken from the perspective of a viewer. The user'"'"'s field of view may be captured in the video images that are processed to select a segment of the video image or an object depicted in the video image. An image such as a computer-generated annotation or graphic overlay way be registered to the segment, and therefore track the segment from the user'"'"'s field of view of the segment, without prior knowledge of the spatial relationship of the segment to the real-world environment according to a centroid for an interframe difference of the video image associated with the selected object. The image may be displayed in the user'"'"'s field of view or in the video image. The computer-generated image tracks the movement of the segment with respect to the video image.

Claims

1. A computer readable medium having stored thereon computer executable instructions to cause a computer programmed thereby to perform a method for tracking an image to an arbitrary object captured in a video representation of a real-world environment, wherein the method of performing comprises:

(a) capturing the video representation of the real-world environment, wherein the video representation includes a sequence of video frames, each having a plurality of pixels arranged according to a rectilinear coordinate grid where each pixel is defined by a pixel value;

(b) registering the image to the object according to movement of a centroid of an interframe difference between consecutive interframe differences of sequential video frames associated with the object, the interframe difference being determined according to differences between pixel values of corresponding pixels associated with object for consecutive video frames; and

(c) generating a signal to display the image in a humanly visualized form at a location in apparent registration to the arbitrary object.

2. The computer readable medium of claim 1 further comprising executable instructions for displaying the image embedded in the video representation.

3. The computer readable medium of claim 1 further comprising executable instructions for displaying the image in a viewer display.

5. The device of claim 1, wherein the display comprises a head-mounted display.

6. The device of claim 1, wherein the display comprises a see-through head-mounted display.

7. The device of claim 1, wherein the display comprises a video terminal.

8. The device of claim 1, wherein the display comprises a remotely located video terminal in communication with the processor.

9. The device of claim 1, wherein each of the plurality of sequential video frames comprises a plurality of pixels arranged according to rectilinear coordinate system wherein each of the plurality of pixels is associated with a pixel coordinate and a pixel value.

10. The device of claim 9, wherein the processor is further configured to define a segment of the plurality of video frames associated with the arbitrary object according to pixels associated with the arbitrary object and to calculate an interframe difference for consecutive video frames based on pixel value for the pixels associated with the segment.

11. The device of claim 10, wherein the processor is further configured to determine a coordinate location of the centroid for the interframe difference of consecutive video frames.

12. The device of claim 11, wherein the processor is further configured to associate a location of the virtual image according to the coordinate location of the centroid.

13. The device of claim 12, wherein the processor is further configured to determine a coordinate location of the centroid based on a subset for the interframe difference of consecutive video frames.

14. The device of claim 12, wherein the processor comprises a wearable processor.

15. The device of claim 12, wherein the processor comprises a remote processor.

4. An augmented reality device, comprising:

(a) a video capturing device configured to acquire video images of objects in a real-world environment of a human viewer, to video image having a plurality of sequential video frames;

(b) a processor configured to register a virtual image to an arbitrary object depicted in the acquired video image according to a centroid of an interframe video image difference between successive sequential video frames of the acquired video image; and

(c) a display configured to render the virtual image in a field of view of the viewer.

16. A method for tracking an arbitrary object in an augmented reality application, the method comprising:

(a) capturing a video image of a user'"'"'s field of view, the video image having a stream of sequential video frames;

(b) determining an interframe difference between consecutive video frames of the stream of sequential video frames, the interframe difference being associated with a selected object depicted in the video image; and

(c) registering a computer-generated image to the selected object based on a movement of a coordinate location of a centroid of the interframe difference.

17. The method of claim 16, further comprising:

(d) generating a signal to display the computer-generated image in a humanly visualized form.

18. The method of claim 16, wherein step (a) comprises the step of generating a graphic representation of the field of view for each video frame, the graphic representation including a rectilinear grid of pixels, wherein each pixel is associated with a pixel coordinate and a pixel value.

19. The method of claim 18, wherein step (b) comprises the steps of:

(i) defining pixels associated with the selected object; and

(ii) for a pair of sequential video frames, calculating a difference between pixels associated with the selected object based on pixel coordinate and pixel value.

20. The method of claim 19, wherein step (c) comprises the step of:

(i) calculating the centroid for an interframe difference of consecutive video frames, the centroid being located at a coordinate location of a center of mass of the interframe difference; and

(ii) associating a location of the computer-generated image with respect to video image based on the location of the centroid.

21. The method of claim 20 further comprising:

(iii) for each sequential video frame, calculating the centroid for an interframe difference of consecutive video frames; and

(iv) for each sequential video frame, associating the location of the computer-generated image based on the calculated location of the centroid.

22. The method of claim 21 further comprising the step of displaying a visual representation of the virtual image in a see-through display.

23. The method of claim 21 wherein step (c) is performed by a wearable processor.

24. The method of claim 21 wherein step (c) is performed by a remote processor.

25. The method of claim 21 further comprising the step of displaying a visual representation of the virtual image embedded in the video image.

26. The method of claim 21 wherein the step (iii) comprises calculating the centroid for an interframe difference of consecutive video frames based on a threshold pixel value.

27. The method of claim 21 further comprising the step of displaying a visual representation of the virtual image and the video image at a remote terminal.

28. The method of claim 27 wherein the step of displaying a visual representation of the virtual image and the video image at a remote terminal comprises delivering a video signal to the remote terminal over a digital networking infrastructure.

29. The method of claim 28 wherein the step of displaying a visual representation of the virtual image and the video image at a remote terminal comprises delivering a video signal to the remote terminal via an analog video transmission.

31. The method of claim 29 further comprising the step of generating a signal to display the computer-generated image in a humanly visualized form.

32. The method of claim 31 wherein the pixel representation includes a plurality of pixels, each pixel being associated with a respective grid coordinate and a respective pixel value.

33. The method of claim 32 wherein the step (c) comprises the steps of:

(i) defining a pixel group associated with the selected object;

(ii) for each sequential pixel representation, calculating a difference between the pixel group according to pixel value and grid coordinate;

(iii) calculating the centroid of the difference between the pixel group; and

(iv) for each pixel representation, rendering the computer-generated image according to the centroid of the difference of pixel representations.

34. The method of claim 33 further comprising the step of rendering the computer-generated image in a human user'"'"'s field of view of the real-world environment in a see-through head mounted display.

35. The method of claim 33 further comprising the step of rendering the computer-generated image in a visual representation of the video image in a head mounted display.

36. The method of claim 33 further comprising the step of rendering the computer-generated image in a visual representation of the video image in a remote terminal.

30. A method for augmenting a field of view of a real-world environment comprising the steps of:

(a) generating a video image associated with the field of view, the video image defined by a sequence of video frames each including a pixel representation of the field of view;

(b) selecting an object in the video image, the selected object associated with pixel representations in the sequence of picture frames; and

(c) registering a computer-generated image to the pixel representations of the selected object according to a centroid of a difference between each sequential pixel representation associated with the selected object.

37. A method for tracking an image to an object captured in a video representation of a real-world environment, comprising the step of:

(a) registering the image to the object according to movement of a centroid of an interframe difference between consecutive interframe differences of sequential video frames associated with object the interframe difference based on differences between corresponding pixel representations for the object in the sequential video frames;

(b) displaying the image embedded in the video representation at a remote terminal.

38. The method of claim 37, further comprising the step of:

(c) displaying the image in a viewer display.