How-to-do
Data Extraction
Motion Data Domain (Mixamo FBX)
The applications of machine learning and deep learning models are emerging every day and a paramount question arises for a beginner: “From where to start?” As a newcomer in Data Science field, mind juggles between choices such as NLP, Computer Vision or anything else. Yet application of any ML/DL algorithm follow the same pipeline: Data extraction, cleaning, model training, evaluation, model selection, deployment. This post ain’t different from others, yet it is curated towards deriving data for Animation or Health care informatics field.
Motion data is complex in nature as it is a hierarchically linked structure just like a graph where each data point or node is a joint and links or edges to other joints are bones. The orientation and location of bones and data points describes the pose which might vary as the number of frames increases and the captured or simulated subject changes it’s position.
By not wasting any more time, I’ll start with script itself. It’s broken down in several steps, but if you want, you can directly jump to the script code given in GitHub link below.
Data Acquisition
Before moving on to further steps, I suggest you download a sample Mixamo(.fbx) file from the link provided below.Put this file into “regular” folder.
I have also provided a sample file inside the folder to test given script.
Data processing
First we import the following libraries. Remember, “bpy” is a Blender library which can only be accessed by Blender work bench. Hence, write the script only in Blender.
#Library Imports
import bpy
import os
import time
import sys
import json
from mathutils import Vector
import numpy as npNext, we set the following setting variables for I/O directory path.
#Settings#This is the main file which loads with clear 'Scene' setting HOME_FILE_PATH = os.path.abspath('homefile.blend')
MIN_NR_FRAMES = 64
RESOLUTION = (512, 512) #Crucial joints sufficient for visualisation #FIX ME - Add more #joints if desirable for MixamRig
BASE_JOINT_NAMES = ['Head', 'Neck','RightArm', 'RightForeArm', 'RightHand', 'LeftArm', 'LeftForeArm', 'LeftHand', 'Hips', 'RightUpLeg', 'RightLeg', 'RightFoot', 'LeftUpLeg', 'LeftLeg', 'LeftFoot', ]
#Source directory where .fbx exist
SRC_DATA_DIR ='regular' #Ouput directory where .fbx to JSON dict will be stored
OUT_DATA_DIR ='fbx2json' #Final directory where NPY files will ve stored
FINAL_DIR_PATH ='json2npy'
In the above code snippet, we have set the RESOLUTION as well as Number of Joints that will be used for data extraction. You can limit the number of frames as well to have uniform number of frames for each animation file. The final processed data will be stored in “\\json2npy” file.
Once the BASE_JOINT_NAMES are selected, we will create a dictionary to access each element of the rig which is by default named as “MixamoRig”.
#Number of joints to be used from MixamoRig
joint_names = ['mixamorig:' + x for x in BASE_JOINT_NAMES]First, the .fbx files are converted to JSON object dictionaries with each dictionary containing per frame joint location information. This is done by using function given below.
def fbx2jointDict():
#Remove 'Cube' object if exists in the scene
if bpy.data.objects.get('Cube') is not None:
cube = bpy.data.objects['Cube']
bpy.data.objects.remove(cube)
#Intensify Light Point in the scene
if bpy.data.objects.get('Light') is not None:
bpy.data.objects['Light'].data.energy = 2
bpy.data.objects['Light'].data.type = 'POINT'
#Set resolution and it's rendering percentage
bpy.data.scenes['Scene'].render.resolution_x = RESOLUTION[0]
bpy.data.scenes['Scene'].render.resolution_y = RESOLUTION[1]
bpy.data.scenes['Scene'].render.resolution_percentage = 100
#Base file for blender
bpy.ops.wm.save_as_mainfile(filepath=HOME_FILE_PATH)
#Get animation(.fbx) file paths
anims_path = os.listdir(SRC_DATA_DIR)
#Make OUT_DATA_DIR
if not os.path.exists(OUT_DATA_DIR):
os.makedirs(OUT_DATA_DIR)
for anim_name in anims_path:
anim_file_path = os.path.join(SRC_DATA_DIR,anim_name)
save_dir = os.path.join(OUT_DATA_DIR,anim_name.split('.')[0],'JointDict')
#Make save_dir
if not os.path.exists(save_dir):
os.makedirs(save_dir)
#Load HOME_FILE and .fbx file
bpy.ops.wm.read_homefile(filepath=HOME_FILE_PATH)
bpy.ops.import_scene.fbx(filepath=anim_file_path)
#End Frame Index for .fbx file
frame_end = bpy.data.actions[0].frame_range[1]
for i in range(int(frame_end)+1):
bpy.context.scene.frame_set(i)
bone_struct = bpy.data.objects['Armature'].pose.bones
armature = bpy.data.objects['Armature']out_dict = {'pose_keypoints_3d': []}
for name in joint_names:
global_location = armature.matrix_world @ bone_struct[name].matrix @ Vector((0, 0, 0))
l = [global_location[0], global_location[1], global_location[2]]
out_dict['pose_keypoints_3d'].extend(l)
save_path = os.path.join(save_dir,'%04d_keypoints.json'%i)
with open(save_path,'w') as f:
json.dump(out_dict, f)--EXPLANATION--
In the function above, first we remove pre-rendered objects like “Cube”, which is set as default when Blender is opened. Then we set the “Light” object settings to increase radiance of energy as well as type as “Point”. These objects can be access using “bpy.data.objects[name of object]” to manipulate the data related to it. Also, we have set the resolution settings by using “object.data.scenes[name of scene]” to manipulate scene rendering setting.Once it is done, we save the file as “main_file()” blend file. All the files in the directory ending with “.fbx” is listed and loaded in loop where each loop extracts the global location of Armature and it’s bones. Armature object can be used to get matrix of location in the video file and it’s pose structure for each frame and save it as a dictionary of bones and it’s location in JSON object file.
Finally, jointDict2npy() method is used to collect each of the JSON object file per animation with all the animation concatenated to represent it as a matrix of frames and locations, just like images.
def jointDict2npy():
json_dir = OUT_DATA_DIR
npy_dir = FINAL_DIR_PATH
if not os.path.exists(npy_dir):
os.makedirs(npy_dir)
anim_names = os.listdir(json_dir)
for anim_name in anim_names:
files_path = os.path.join(json_dir,anim_name,'jointDict')
frame_files = os.listdir(files_path)
motion = []
for frame_file in frame_files:
file_path = os.path.join(files_path,frame_file)
with open(file_path) as f:
info = json.load(f)
joint = np.array(info['pose_keypoints_3d']).reshape((-1, 3))
motion.append(joint[:15,:])
motion = np.stack(motion,axis=2)
save_path = os.path.join(npy_dir,anim_name)
if not os.path.exists(save_path):
os.makedirs(save_path)
print(save_path)
np.save(save_path+'\\'+'{i}.npy'.format(i=anim_name),motion)--EXPLANATION--
Above function saves each of the animation in the form of .npy file which consist of 3D-Data array like Tensor formatted as (NumberOfJoints, NumberOfAxes, NumberOfFrames). “.npy” is an efficient structure which saves the data encoded in binary representation. Another alternative is to save it as a compressed file using “savez_compressed()” to “.npz” files.
Execute the script using Command Prompt running as Administrator. Type the following command to execute script:
CMD COMMAND : blender --background -P fbx2npy.py'if __name__ == '__main__':
#Convert .fbx files to JSON dict
fbx2jointDict()
#Convert JSON dict to NPY
jointDict2npy()
Final results of processing will look like the given images below. These are some 5 frames selected from the video motion file. You can use the “visualise_frame.py” file to visualise the results given below on the sample data.
Conclusion
In this post we learnt about how we can process the animation data in .fbx format to get the data about the locations of each joint on per frame basis. There are many applications in the wild that require the motion data to learn tasks like Pose Estimation, Motion Retargeting etc. In the next post, I’ll share how this type of data can be normalised so that it can be used by deep learning models. Till then, see ya amigos!!!
