Arbitrary object tracking in augmented reality applications

• US 20040119662A1
• Filed: 12/19/2002
• Published: 06/24/2004
• Est. Priority Date: 12/19/2002
• Status: Active Grant

• Current Assignee: Accenture Global Services Limited (Accenture PLC)
• Original Assignee: Accenture Global Services GmbH (Accenture PLC)
• Inventors: Dempski, Kelly L.
• Granted Patent: US 7,050,078 B2
• Time in Patent Office: Days
• US Class Current: 345/8
• CPC Class Codes: G01S 3/7864 T.V. type tracking systems
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 2027/0198 System for aligning or maintaining alignment of an image in a predetermined direction
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042

Abstract

The arbitrary object tracking relates to the field of augmented reality applications. Video images of objects in a real-world environment are taken from the perspective of a viewer in the real-world environment. The user'"'"'s field of view of the real-world environment may be captured in the video images, which may be processed to select a segment of the video image or an object depicted in the video image. An image such as a computer-generated annotation or graphic overlay may be registered to the segment, and therefore track the segment from the user'"'"'s field of view of the segment, without prior knowledge of the spatial relationship of the segment to the real-world environment. The computer-generated image tracks the segment according to a centroid for an interframe difference of the video image associated with the selected object and may be displayed in the user'"'"'s field of view or in the video image. As the segment moves or the viewer'"'"'s perspective of the real-environment changes, the computer-generated image will track the movement of the segment with respect to the video image.

Claims

1. An augmented reality device, comprising:

(a) a video capturing device configured to acquire video images of objects in a real-world environment of a human viewer, the video image having a plurality of sequential video frames;

(b) a processor configured to register a virtual image to an arbitrary object depicted in the acquired video image according to a centroid of an interframe video image difference; and

(c) a display configured to render the virtual image in a field of view of the viewer.

2. The device of claim 1, wherein the display comprises a head-mounted display.

3. The device of claim 1, wherein the display comprises a see-through head-mounted display.

4. The device of claim 1, wherein the display comprises a video terminal.

5. The device of claim 1, wherein the display comprises a remotely located video terminal in communication with the processor.

6. The device of claim 1, wherein each of the plurality of sequential video frames comprises a plurality of pixels arranged according to rectilinear coordinate system wherein each of the plurality of pixels is associated with a pixel coordinate and a pixel value.

7. The device of claim 6, wherein the processor is further configured to define a segment of the plurality of video frames associated with the arbitrary object according to pixels associated with the arbitrary object and to calculate an interframe difference for consecutive video frames based on pixel value for the pixels associated with the segment.

8. The device of claim 7, wherein the processor is further configured to determine a coordinate location of the centroid for the interframe difference of consecutive video frames.

9. The device of claim 8, wherein the processor is further configured to associate a location of the virtual image according to the coordinate location of the centroid.

10. The device of claim 9, wherein the processor is further configured to determine a coordinate location of the centroid based on a subset for the interframe difference of consecutive video frames.

11. The device of claim 9, wherein the processor comprises a wearable processor.

12. The device of claim 9, wherein the processor comprises a remote processor.

13. A method for tracking an arbitrary object in an augmented reality application, the method comprising:

(a) capturing a video image of a user'"'"'s field of view, the video image having a stream of sequential video frames;

(b) determining an interframe difference between sequential video frames, the interframe difference being associated with a selected object depicted in the video image; and

(c) registering a computer-generated image to the selected object based on a movement of a coordinate location of a centroid of the interframe difference.

14. The method of claim 13, further comprising:

(d) generating a signal to display the computer-generated image in a humanly visualized form.

15. The method of claim 13, wherein step (a) comprises the step of generating a graphic representation of the field of view for each video frame, the graphic representation including a rectilinear grid of pixels, wherein each pixel is associated with a pixel coordinate and a pixel value.

16. The method of claim 15, wherein step (b) comprises the steps of:

(i) defining pixels associated with the selected object; and

(ii) for a pair of sequential video frames, calculating a difference between pixels associated with the selected object based on pixel coordinate and pixel value.

17. The method of claim 16, wherein step (c) comprises the step of:

(i) calculating the centroid for an interframe difference of consecutive video frames, the centroid being located at a coordinate location of a center of mass of the interframe difference; and

(ii) associating a location of the computer-generated image with respect to video image based on the location of the centroid.

18. The method of claim 17 further comprising:

(iii) for each sequential video frame, calculating the centroid for an interframe difference of consecutive video frames, and

(iv) for each sequential video frame, associating the location of the computer-generated image based on the calculated location of the centroid.

19. The method of claim 18 further comprising the step of displaying a visual representation of the virtual image in a see-through display.

20. The method of claim 18 wherein step (c) is performed by a wearable processor.

21. The method of claim 18 wherein step (c) is performed by a remote processor.

22. The method of claim 18 further comprising the step of displaying a visual representation of the virtual image embedded in the video image.

23. The method of claim 18 wherein the step (iii) comprises calculating the centroid for an interframe difference of consecutive video frames based on a threshold pixel value.

24. The method of claim 18 further comprising the step of displaying a visual representation of the virtual image and the video image at a remote terminal.

25. The method of claim 24 wherein the step of displaying a visual representation of the virtual image and the video image at a remote terminal comprises delivering a video signal to the remote terminal over a digital networking infrastructure.

26. The method of claim 25 wherein the step of displaying a visual representation of the virtual image and the video image at a remote terminal comprises delivering a video signal to the remote terminal via an analog video transmission.

28. The method of claim 26 further comprising the step of generating a signal to display the computer-generated image in a humanly visualized form.

29. The method of claim 28 wherein the pixel representation includes a plurality of pixels, each pixel being associated with a respective grid coordinate and a respective pixel value.

30. The method of claim 29 wherein the step (c) comprises the steps of:

(i) defining a pixel group associated with the selected object;

(ii) for each sequential pixel representation, calculating a difference between the pixel group according to pixel value and grid coordinate;

(iii) calculating the centroid of the difference between the pixel group; and

(iv) for each pixel representation, rendering the computer-generated image according to the centroid of the difference of pixel representations.

31. The method of claim 30 further comprising the step of rendering the computer-generated image in a human user'"'"'s field of view of the real-world environment in a see-through head mounted display.

32. The method of claim 30 further comprising the step of rendering the computer-generated image in a visual representation of the video image in a head mounted display.

33. The method of claim 30 further comprising the step of rendering the computer-generated image in a visual representation of the video image in a remote terminal.

27. A method for augmenting a field of view of a real-world environment comprising the steps of:

(a) generating a video image associated with the field of view, the video image defined by a sequence of pixel representations of the field of view;

(b) selecting an object in the video image; and

(c) registering a computer-generated image to the selected object according to a centroid of a difference between each sequential pixel representation associated with the selected object.

34. A method for tracking an image to an object captured in a video representation of a real-world environment, comprising the step of:

(a) registering the image to the object according to movement of a centroid of an interframe difference between sequential video frames associated with the object.

35. The method of claim 34, further comprising the step of:

(b) displaying the image embedded in the video representation at a remote terminal.

36. The method of claim 35, further comprising the step of:

(c) displaying the image in a viewer display.

37. A computer readable medium having stored thereon computer executable instructions to cause a computer programmed thereby to perform a method for tracking an image to an arbitrary object captured in a video representation of a real-world environment, wherein the method of performing comprises:

(a) capturing the video representation of the real-world environment, wherein the video representation includes a sequence of video frames having a plurality of pixels arranged according to a rectilinear coordinate grid and having a pixel value;

(b) registering the image to the object according to movement of a centroid of an interframe difference between sequential video frames associated with the object, the interframe difference being determined according to pixel coordinate and pixel value; and

(c) generating a signal to display the image in a humanly visualized form at a location in apparent registration to the arbitrary object.

38. The computer readable medium of claim 37 further comprising executable instructions for displaying the image embedded in the video representation.

39. The computer readable medium of claim 37 further comprising executable instructions for displaying the image in a viewer display.