Object-tracking systems and methods

• US 8,811,660 B2
• Filed: 03/27/2009
• Issued: 08/19/2014
• Est. Priority Date: 09/29/2008
• Status: Active Grant

• Current Assignee: Venus Concept, Inc.
• Original Assignee: Restoration Robotics, Inc. (Venus Concept, Inc.)
• Inventors: Qureshi, Shehrzad A., Breton, Kyle R., Tenney, John A.
• Primary Examiner(s): Akhavannik, Hadi
• Assistant Examiner(s): RASHIDIAN, MOHAMMAD M
• Application Number:
  US12/412,753
• Publication Number:
  US 20100080417A1
• Time in Patent Office: 1,971 Days
• Field of Search: 382/100, 382103-104, 382/107, 382128-132, 382/285, 382/276, 382/218, 382/171, 345/169, 345/630, 345/156, 345/420, 345/474, 600/424, 600/426, 600/427, 600/429, 600/431, 128/905, 128/920, 128/925, 606/130, 606/187
• US Class Current: 382/103
• CPC Class Codes: A61B 2017/00752 Hair removal or transplantation
• CPC Class Codes: A61B 2017/00969 used for transplantation
• CPC Class Codes: A61B 2034/105 Modelling of the patient, e.g. for ligaments or bones
• CPC Class Codes: A61B 2034/2057 Details of tracking cameras
• CPC Class Codes: A61B 2034/2065 Tracking using image or pattern recognition
• CPC Class Codes: A61B 2090/363 Use of fiducial points
• CPC Class Codes: A61B 2090/3937 Visible markers
• CPC Class Codes: A61B 34/10 Computer-aided planning, simulation or modelling of surgical operations
• CPC Class Codes: A61B 34/20 Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• CPC Class Codes: A61B 34/30 Surgical robots
• CPC Class Codes: A61B 34/32 operating autonomously
• CPC Class Codes: A61B 90/17 for soft tissue, e.g. breast-holding devices
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/30088 Skin; Dermal
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 2207/30242 Counting objects in image
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/246 : using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/248 : involving reference images or patches
• CPC Class Codes: G06T 7/40 : Analysis of texture depth or shape recovery from texture G06T7/529
• CPC Class Codes: G06T 7/74 : involving reference images or patches
• CPC Class Codes: G16H 30/20 : for handling medical images, e.g. DICOM, HL7 or PACS
• CPC Class Codes: G16H 30/40 : for processing medical images, e.g. editing
• CPC Class Codes: G16H 50/20 : for computer-aided diagnosis, e.g. based on medical expert systems
• CPC Class Codes: Y10S 901/47 : Optical

Abstract

A system and method for tracking, identifying, and labeling objects or features of interest is provided. In some embodiments, tracking is accomplished using unique signature of the feature of interest and image stabilization techniques. According to some aspects a frame of reference using predetermined markers is defined and updated based on a change in location of the markers and/or specific signature information. Individual objects or features within the frame may also be tracked and identified. Objects may be tracked by comparing two still images, determining a change in position of an object between the still images, calculating a movement vector of the object, and using the movement vector to update the location of an image device.

Claims

1. A system for tracking a feature of interest on a body, the system comprising:

an interface adapted to receive a plurality of images from an image capture device, at least a first image comprising a feature of interest;

a signature identification component, wherein the signature identification component identifies signature information of the feature of interest and detects at least one marker in the first image that is distinct from and has a location different from a location of the feature of interest, wherein the identified signature information is unique to the feature of interest and comprises a plurality of characteristics of the feature of interest;

a marker referencing system, wherein the marker referencing system defines a frame of reference corresponding to the at least one marker, determines whether the at least one marker is in a second image taken at a different point in time relative to the first image, adjusts the frame of reference corresponding to a change in position of the at least one marker from the first image to the second image, and searches in a direction and within a predetermined distance from a motion vector corresponding to the adjusted frame of reference to locate the feature of interest in the second image using the identified signature information of the feature of interest.

2. The system of claim 1, wherein the signature identification component and the marker referencing system are part of a single computer program.

3. The system of claim 1, wherein the signature identification component comprises a processor connected to memory, and wherein the processor associates an electronic identifier with the at least one marker and stores the electronic identifier in the memory.

4. The system of claim 1, wherein the marker referencing system comprises a processor connected to memory, and wherein the processor compares a location of each of the at least one marker in the first image to a location of each of the at least one marker in a second image, calculates a corresponding change in a frame of reference, and stores a result of the calculation in the memory.

5. The system of claim 1, further comprising an image capture device and wherein the image-capture device is a camera.

6. The system of claim 5, wherein the system is a robotic system further comprising a robotic arm and wherein the image capture device is operatively connected to the robotic arm.

7. The system of claim 1, wherein the feature of interest is a hair follicle and wherein the at least one marker is an anatomical landmarks on the body, an external fiducial, or a combination of the above.

8. The system of claim 1, wherein the at least one marker is a follicular unit and detecting the at least one marker further comprises identifying at least one of the emergence angle of the follicular unit from a body surface or the caliber of the hair.

9. The system of claim 1, wherein the marker referencing system defines a search region comprising an area within a predetermined angle from the direction of the motion vector.

10. The system of claim 1, further comprising an image capture device, and wherein the image capture device is moved according to an average shift of a point cloud.

11. The system of claim 1, wherein the marker referencing system is configured to adjust the frame of reference if one or more of the at least one marker is visible in the second image even if the feature of interest is not visible in the second image.

12. The system of claim 1, wherein the feature of interest comprises a bald area on the body surface, and the at least one marker is any readily identifiable object.

13. The system of claim 1, wherein the at least one marker comprises hair, a mole, texturing, a scar, a freckle, a wrinkle, a bump, a depression of a body surface, or blood.

14. The system of claim 1, wherein adjusting the frame of reference comprises adjusting both translational and rotational components of the frame of reference.

15. The system of claim 1, wherein after locating the feature of interest in the second image, the system is further configured to perform a procedure with respect to the feature of interest.

16. The system of claim 1, wherein determining whether the at least one marker is in the second image comprises comparing a series of pixels or regions-of-interest between the first and the second image.

17. A system for tracking a feature of interest on a body surface, the system comprising:

an imaging device, the imaging device configured for capturing still images; and

an image processor comprising one or more modules for executing operations on the captured images, the one or more modules comprising instructions for;

identifying at least one marker in a first still image, the first still image containing a feature of interest other than the at least one marker;

determining a signature of the feature of interest in the first image, the signature is unique to the feature of interest and comprises a plurality of characteristics of the feature of interest;

defining a frame of reference corresponding to the at least one marker;

locating the at least one marker in a second still image taken at a different point in time than the first image; and

updating a translational component of the frame of reference based on a motion vector corresponding to a change in position of the at least one marker in the second still image with respect to the first still image; and

searching for the feature of interest in the second still image in a search region using the determined signature, the search region is defined in a direction and within a predetermined distance from the motion vector.

18. The system of claim 17, wherein the one or more modules comprise instructions for updating a rotational component of the frame of reference.

19. The system of claim 17, wherein the system further comprises a robotic arm operatively associated with the imaging device and the processor.

20. A method for tracking a feature of interest on a body, the method comprising:

identifying at least one marker in a first still image taken by an imaging device, the first still image containing a feature of interest other than the at least one marker;

determining a signature of the feature of interest in the first still image;

defining a frame of reference corresponding to the at least one marker;

locating the at least one marker in a second still image taken by the imaging device at a different point in time relative to the first image; and

updating a translational component of the frame of reference based on a motion vector corresponding to a change in a position of the at least one marker in the second still image, with respect to the first still image; and

searching for the feature of interest in the second still image in a search region using the determined signature, the search region is defined in a direction and within a predetermined distance from the motion vector.

21. The method of claim 20, wherein the feature of interest is a follicular unit and determining the signature of the follicular unit in the first still image comprises identifying two or more of a length, type, caliber, emergence angle, area, shape, or color of the follicular unit.

22. The method of claim 20, wherein identifying the at least one marker in a first still image comprises identifying at least three markers, and

wherein the method further comprises updating both translational and rotational components of the frame of reference.

23. The method of claim 20, wherein identifying the at least one marker in the first still image comprises analyzing at least one of a length, an area, a shape, a type, or a color of the at least one marker.

24. The method of claim 23, wherein the at least one marker is a follicular unit and identifying the at least one marker further comprises identifying at least one of the emergence angle of the follicular unit from a body surface or the caliber of the hair.

25. The method of claim 20, wherein the signature of the feature of interest is determined by transforming the first still image to extract one or more characteristics that constitutes the signature of the feature of interest.

26. The method of claim 20, wherein the search region comprises an area within a predetermined angle from the direction of the motion vector.

27. The method of claim 20, wherein defining the frame of reference is accomplished using a point cloud approach and the method further comprising moving an imaging device according to an average shift of a point cloud.

28. The method of claim 20, wherein the feature of interest is a bald spot and the method further comprises defining one or more hair implantation sites.

29. The method of claim 20, wherein the at least one marker comprises at least three markers and the method further comprises:

tracking at least one object in the first still image;

determining whether the at least three markers are in the second still image; and

in response to at least one of the at least three markers not being in the second still image, detecting a position of the feature of interest in the second still image using the determined signature of the feature of interest and updating the frame of reference based on a position of any of the at least three markers that are in the second still image and the at least one object in the second image,

wherein a sum of a combination of markers and the at least one objects used to detect the position of the feature of interest is at least three.

30. The method of claim 29, wherein the at least one object comprises a follicular unit, a mole, a scar, a freckle, a wrinkle, a bump, or a depression of a body surface.

31. The method of claim 20, wherein the motion vector is computed using one or more of a point cloud approach, Gray-Coded Bit-Plane, optical flow, and block search image stabilization approach.

32. The method of claim 20, further comprising removing the feature of interest or transplanting the feature of interest from a donor area to a recipient area.

33. The method of claim 20, wherein the at least one marker comprises a follicular unit, a mole, texturing, blood, a scar, a freckle, a wrinkle, a bump, or a depression of a body surface.

34. The method of claim 20, wherein the frame of reference is defined using a point cloud based only on one or more objects present in both the first still image and the second still image.