Object creation using body gestures

• US 9,928,634 B2
• Filed: 03/01/2013
• Issued: 03/27/2018
• Est. Priority Date: 03/01/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Cao, Xiang, Liu, Yang, Han, Teng, Shiratori, Takaaki, Umetani, Nobuyuki, Zhang, Yupeng, Tong, Xin, Ren, Zhimin
• Primary Examiner(s): Nguyen, Vu
• Application Number:
  US14/770,800
• Publication Number:
  US 20160027199A1
• Time in Patent Office: 1,852 Days
• Field of Search: None
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2219/20 Indexing scheme for editing of 3D models

Abstract

An intuitive interface may allow users of a computing device (e.g., children, etc.) to create imaginary three dimensional (3D) objects of any shape using body gestures performed by the users as a primary or only input. A user may make motions while in front of an imaging device that senses movement of the user. The interface may allow first-person and/or third person interaction during creation of objects, which may map a body of a user to a body of an object presented by a display. In an example process, the user may start by scanning an arbitrary body gesture into an initial shape of an object. Next, the user may perform various gestures using his body, which may result in various edits to the object. After the object is completed, the object may be animated, possibly based on movements of the user.

Claims

1. A method comprising:

creating, using an imaging device, a three dimensional (3D) object having an initial shape based on an outline of an initial pose of a user;

causing presentation of the 3D object having the initial shape on a display to provide feedback to the user;

editing the 3D object by;

determine a reference mode of a plurality of reference modes for editing the 3D object, the plurality of reference modes including a spatial mapping mode and a body reference mapping mode, the body reference mapping mode allowing the user to edit the 3D object based on at least one of the user being inside of the 3D object and the user being the 3D object, and the spatial mapping mode allowing the user to edit the 3D object based on at least one of the user stepped out of the 3D object and the user being next to the 3D object;

determining an edit operation to be performed on the 3D object;

performing the edit operation on the 3D object based on a body gesture of the user that is sensed by the imaging device and the reference mode for editing the 3D object; and

causing presentation of the 3D object to the user; and

animating the 3D object based on additional body gestures of the user.

2. The method as recited in claim 1, wherein the performing the edit operation includes:

selecting at least a portion of a surface of the 3D object based on a first part of the body gesture of the user; and

modifying the at least the portion of the surface of the 3D object based on a second part of the body gesture of the user.

3. The method as recited in claim 1, wherein the editing the 3D object is performed by using a mapping between the user and the 3D object, the mapping being the spatial mapping mode where the user is mapped to a location outside of the 3D object to allow the user to interact with the 3D object in a third-person editing perspective.

4. The method as recited in claim 3, further comprising determining the mapping prior to performing the edit operation.

5. The method as recited in claim 1, wherein the 3D object is an avatar.

6. The method as recited in claim 1, wherein the edit operation is at least one of a push operation, a drag operation, a sketch operation, a sculpt operation, a cut operation, or a paint operation.

7. The method as recited in claim 1, further comprising:

mapping a plurality of features of the 3D object to a corresponding plurality of body parts of the user sensed by the imaging device; and

allowing the user to modify the mapping of one or more features of the plurality of features of the 3D object to one or more body parts of the corresponding plurality of body parts of the user,

wherein the animating the 3D object includes moving the plurality of features of the 3D object based on movement of the corresponding plurality of body parts of the user.

8. The method as recited in claim 1, further comprising storing the 3D object for access by another computing device.

9. The method as recited in claim 1, further comprising:

receiving a selection of an existing 3D object, and

adding the existing 3D object to the 3D object.

10. The method as recited in claim 9, further comprising obtaining the existing 3D object by scanning a real world object using the imaging device.

11. The method as recited in claim 1, wherein the determining the edit operation includes identifying a tool based on object recognition on an object held by the user or based on the body gesture;

and modifying the 3D object with the tool based at least in part on the body gesture.

12. The method as recited in claim 1, further comprising receiving a voice command using a microphone, and wherein the determining the edit operation or the performing the edit operation is based at least partly on the voice command.

13. A system comprising:

an imaging device to detect a body gestures of a user;

one or more processors; and

memory to store computer readable instructions that, when executed on the one or more processors, cause the one or more processors to perform acts comprising;

creating, using the imaging device, a three dimensional (3D) object having an initial shape based on an outline of an initial pose of the user;

causing presentation of the 3D object having the initial shape on a display to provide feedback to the user;

identifying a body gesture of the user based on information received from the imaging device;

determining an edit operation to be performed on the 3D object based on the body gesture;

modifying the 3D object using the edit operation;

identifying a tool based on the body gesture;

determining a tool operation to be performed by the tool;

determine a reference mode of a plurality of reference modes for editing the 3D object, the plurality of reference modes including a spatial mapping mode and a body reference mapping mode, the body reference mapping mode allowing the user to edit the 3D object based on at least one of the user being inside of the 3D object and the user being the 3D object, and the spatial mapping mode allowing the user to edit the 3D object based on at least one of the user stepped out of the 3D object and the user being next to the 3D object; and

modifying the 3D object, wherein modifying the 3D object includes causing the tool to interact with the 3D object based on the body gesture and the reference mode for editing the 3D object.

14. The system as recited in claim 13, wherein the edit operation modifies a location of at least a portion of a surface of the 3D object based on movement of the user during the body gesture.

15. The system as recited in claim 13,

wherein the determining the tool operation is further based at least in part on identification of the tool held by the user and detected by the imaging device.

16. The system as recited in claim 13, further comprising creating the 3D object using at least one of voxel modeling or implicit surface modeling.

17. The system as recited in claim 13, wherein the 3D object is an avatar, and wherein the acts further comprise animating the avatar based on movements of the user.

18. One or more memory storing computer-executable instructions that, when executed on one or more processors, causes the one or more processors to perform acts comprising:

obtaining a three dimensional (3D) object having an initial shape based on an outline of an initial pose of a user;

causing presentation of the 3D object having the initial shape on a display to provide feedback to the user;

determine a reference mode of a plurality of reference modes for editing the 3D object, the plurality of reference modes including a spatial mapping mode and a body reference mapping mode, the body reference mapping mode allowing the user to edit the 3D object based on at least one of the user being inside of the 3D object and the user being the 3D object, and the spatial mapping mode allowing the user to edit the 3D object based on at least one of the user stepped out of the 3D object and the user being next to the 3D object;

determining an edit operation to be performed on the 3D object;

performing the edit operation on the 3D object based on a body gesture of the user that is sensed by a imaging device and the reference mode for editing the 3D object;

causing presentation of the 3D object;

allowing the user to map a first feature of the 3D object to a first body part of the user; and

animating the 3D object by moving the first feature in response to a first movement of the first body part of the user.

19. The one or more memory as recited in claim 18, wherein the edit operation modifies a location of at least a portion of a surface of the 3D object based on movement of the user during the body gesture.

20. The one or more memory as recited in claim 18, further comprising:

allowing the user to map a second feature and a third feature of the 3D object to a second body part of the user;

wherein animating the 3D object further comprises moving the second feature and the third feature of the 3D object based on a second movement of the second body part of the user.