[Maya C++ API] Per vertex color update in MPxDeformerNode

Code snippet in C++ to be able to update the color of a mesh with Maya API on a per vertex basis.

Enable  color in Maya

To display per vertex color in Maya's viewport, select the mesh and enable in Maya's menu:

• Maya 2016 < Color ➞ Toggle Per Vertex Colors Attribute

• Maya 2017 >= Mesh Display ➞ Toggle Display Colors Attribute


Or call the MEL equivalent: toggleShadeMode();.

You can check wether or not the shade mode is enabled/disabled by checking the mesh attribute displayColors with the MEL command: getAttr "mesh_name.displayColors";

Color set

Maya allows you to define several color sets associated to a single mesh. You can manage the sets of colors through the UI with:Mesh Display ➞ Color Set Editor

If your mesh node does not receive any input mesh in his attribute inMesh then color sets and color values will be stored directly into the mesh node. But most likely your mesh is going to be the end result of another node (skin cluster, subdivision, poly node generation etc.) and in this case an intermediate node of type polyColorPerVertex will be placed right before inMesh.

Set per vertex color with MEL

 To set per vertex color with MEL use:

# polyColorPerVertex -rgb 0.8 0.0 0. someMeshShape.vtx[0:1381];

This will change the per vertex color of the currently active color set, or create a new color set along with necessary dependency nodes (e.g. polyColorPerVertex) if none exists.

You can manage color sets (creation, deletion etc.) with polyColorSet. For instance change the active color set to "colorSet1" using:

# polyColorSet -currentColorSet -colorSet "colorSet1";


Although using the command line is the easiest way to change colors programatically it is quite slow. Here I explain how to directly access the colors through attributes. This knowledge will be needed to implement faster access through C++ and MPlug.

So, the structure of the node polyColorPerVertex attributes is as follows:

// vertexColor[$i] contains list every vertex indices [0:nb_verts]
$i = $vertex_index
polyColorPerVertex1.colorPerVertex.vertexColor[$i].vertexColorRGB 0.0 1.0 0.0;
polyColorPerVertex1.colorPerVertex.vertexColor[$i].vertexAlpha 1.0;

// vertexFaceColor[$j] is a sparse list of the "face-vertex" indices
// This list size equals the number of faces adjacent to $i
// $j indices are not consecutive. 
// Example:
polyColorPerVertex1.colorPerVertex.vertexColor[$i].vertexFaceColor[20].vertexFaceColorRGB 0.0 1.0 0.0;
polyColorPerVertex1.colorPerVertex.vertexColor[$i].vertexFaceColor[20].vertexFaceAlpha 1.0;
polyColorPerVertex1.colorPerVertex.vertexColor[$i].vertexFaceColor[1080].vertexFaceColorRGB 0.0 1.0 0.0;
polyColorPerVertex1.colorPerVertex.vertexColor[$i].vertexFaceColor[1080].vertexFaceAlpha 1.0;
...

You must set both vertexColor[] and vertexFaceColor[] to the same color to change a vertex' color:

int $nb_verts[] = `polyEvaluate -vertex pCylinder1`;
for($i = 0; $i < $nb_verts[0]; ++$i)
{     
     setAttr polyColorPerVertex1.colorPerVertex.vertexColor[$i].vertexColorRGB 0.0 1.0 0.0;
     setAttr polyColorPerVertex1.colorPerVertex.vertexColor[$i].vertexAlpha 1.0;

     // vertexFaceColor also need to be set!     
     int $indices[] = `getAttr -multiIndices polyColorPerVertex1.colorPerVertex.vertexColor[$i].vertexFaceColor`;
     for($j in $indices){
         setAttr polyColorPerVertex1.colorPerVertex.vertexColor[$i].vertexFaceColor[$j].vertexFaceColorRGB 0.0 1.0 0.0;
         setAttr polyColorPerVertex1.colorPerVertex.vertexColor[$i].vertexFaceColor[$j].vertexFaceAlpha 1.0;
    }
}

 This MEL code is highly ineficient but helps understand the process.

Set per vertex color within a command

If color do not need to be updated in real-time, and color update happens only in a handfull of cases then setting colors through a C++/Python command present a good tradeoff between speed and implementation complexity. I recommend using a deformer node in other cases.

MFnMesh::setVertexColors()

Outside a node it's quite simple to update the mesh colors through the class MFnMesh of the Maya API: 

#define mayaCheck(code) do{ const g_status_(code); const int l = __LINE__;\
    if(g_status_ != MS::kSuccess) \
        throw My_exception(g_status_, __TO_STR(__FILE__), l); \
    }while(false)

    //--------------------
{  
    MObject mesh_shape;
    MFnMesh mesh_fn( mesh_shape, &status);
    mayaCheck(status);
    int num_verts = mesh_fn.numVertices(&status);
    mayaCheck(status);
    MColorArray colors;
    MIntArray vertex_idx_list;
    if( colors.length() != num_verts){
        mayaCheck(colors.setLength(num_verts));
        mayaCheck(vertex_idx_list.setLength(num_verts));
    }

    MGlobal::displayInfo(MString("num verts: ") + num_verts);
    for(int i = 0; i < num_verts; ++i)
    {
        colors.set(MColor(1.0f, 0.0f, 0.0f), i); // Set to red
        vertex_idx_list[i] = i;
    }

    mayaCheck( mesh_fn.setVertexColors(colors, vertex_idx_list) );
}

Although way faster than MEL commands, setVertexColors()can be slow. There seem to be a large overhead with large meshes regardless of the number of vertex color you set. I measured 70ms on my Intel I7 3.60Ghz for 80 000 vertices. Therefore it is not suitable for interactive operations like paint brush for large meshes.

MFnMesh::setVertexColors() & MPlugs

What I found is the best tradeoff between ease of implementation and speed is to use MPlug when you know you only need to set a few subset of vertex colors.

MObject mesh_shape; // The mesh we are working on.
    
MPlug in_mesh_plug = get_plug(mesh_shape, "inMesh");
MObject poly_color_node;
if( !find_node_of_type(in_mesh_plug, MItDependencyGraph::kUpstream, MFn::kPolyColorPerVertex, poly_color_node) ){
    mayaWarning("Can't find polyColorPerVertexNode for "+ get_name(mesh_shape) );
    return;
}
    
MPlug colorPerVertex = get_plug(poly_color_node, "colorPerVertex");
MPlug vertexColor = get_child(colorPerVertex, 0);
    
// Ideally this needs to loop only over a subset of vertices
// Otherwise setVertexColors() is likely to be faster
for(unsigned idx = 0; idx < nb_vertices; ++idx)
{
    //MPlug element = find_element(vertexColor, idx);
    MPlug element = element_at(vertexColor, idx);
        
    // 0: polyColorNode.colorPerVertex.vertexColor[idx].vertexColorRGB 0.0 1.0 0.0;
    // 1: polyColorNode.colorPerVertex.vertexColor[idx].vertexAlpha 1.0;
    // 2: polyColorNode.colorPerVertex.vertexColor[idx].vertexFaceColor[];
    MPlug vertexColorRGB   = get_child(element, 0);
    MPlug vertexColorAlpha = get_child(element, 1);
    MPlug vertexFaceColor  = get_child(element, 2);
        
    Vec3 color = compute_color(..);
    set_as(vertexColorRGB, color);
    set_as(vertexColorAlpha, 1.0f);
        
    unsigned nb_faces = num_elements(vertexFaceColor);
    for(unsigned j = 0; j < nb_faces; ++j)
    {
        MPlug element = element_at(vertexFaceColor, j);        
        // 0: polyColorNode.colorPerVertex.vertexColor[idx].vertexFaceColor[j_logicalIdx].vertexFaceColorRGB 0.0 1.0 0.0;
        // 1: polyColorNode.colorPerVertex.vertexColor[idx].vertexFaceColor[j_logicalIdx].vertexFaceAlpha 1.0;
        MPlug vertexFaceColorRGB = get_child(element, 0);
        MPlug vertexFaceAlpha    = get_child(element, 1);
            
        set_as(vertexFaceColorRGB, color );
        set_as(vertexFaceAlpha, 1.0f);
    }
}

Here are the function utilities used in the above snippet:

MPlug element_at(const MPlug& plug, unsigned physical_index)
{
    MStatus status;
    MPlug elt = plug.elementByPhysicalIndex(physical_index, &status);
    mayaCheck( status );
    return elt;
}

unsigned num_elements( const MPlug& plug)
{
    MStatus status;
    unsigned num = plug.numElements(&status);
    mayaCheck(status);
    return num;
}

unsigned get_nb_children(const MPlug& plug)
{
    MStatus status;
    unsigned num = plug.numChildren(&status);
    mayaCheck(status);
    return num;
}

MPlug get_child(const MPlug& plug, unsigned index)
{
    mayaAssert( index < get_nb_children(plug)  );
    MStatus status;
    MPlug child = plug.child(index, &status);
    mayaCheck(status);
    return child;
}

MPlug get_plug(const MObject& node, const MString& attribute)
{
    MStatus status;
    MFnDependencyNode dg_fn ( node );
    MPlug plug = dg_fn.findPlug ( attribute, true, &status );
    mayaCheck(status);
    return plug;
}

MString get_name(MObject obj)
{
    MStatus status;
    MFnDependencyNode dep(obj);
    MString name = dep.name(&status);
    mayaCheck(status);
    return name;
}

void set_plug_as_3Float(MPlug& plug, const Vec3& value)
{
    MStatus status;
    MFnNumericData data;
    MObject obj = data.create(MFnNumericData::k3Float, &status);
    mayaCheck(status);
    mayaCheck(data.setData3Float(value.x, value.y, value.z));
    mayaCheck(plug.setMObject(obj));
}

template
void set_as(MPlug& plug, float v ){
    static_assert( std::is_same<T, float>::value, "Not float" );
    mayaCheck( plug.setFloat(v) );
}

template
void set_as(MPlug& plug, const Vec3& v){
    static_assert( std::is_same<T, Vec3>::value,"not Vec3" );
    set_plug_as_3Float(plug, v);
}


bool find_node_of_type(MPlug source_plug,
                       MItDependencyGraph::Direction type,
                       MFn::Type node_type,
                       MObject& result )
{
    MStatus status;
    if( is_connected(source_plug) )
    {        
        MItDependencyGraph dg_it(source_plug,
                                 MFn::kInvalid,
                                 type,
                                 MItDependencyGraph::kDepthFirst,
                                 MItDependencyGraph::kPlugLevel,
                                 &status);
        
        mayaCheck(status);
        dg_it.disablePruningOnFilter();
        for ( ; ! dg_it.isDone(); dg_it.next() )
        {
            MObject curr_node = dg_it.currentItem(&status);
            mayaCheck(status);
            if (curr_node.apiType() == node_type)
            {
                result = curr_node;
                return true;
            }
        }
    }
    result = MObject::kNullObj;
    return false;
}

Set per vertex color within a Node

Most suitable when you need real-time update, and you want the update to occur automatically without having to track down every single changes. Using a node allows to detect easily any change through the node's attributes.

It's also useful in the following cases:

• debug a node deformer by visualizing the algorithm's result.
• per vertex color may get overwritten by some upstream nodes like a skin cluster. In this case you need a node to ensure the color stays the same even when the dependency graph is re-evaluated.

Next I show how to change vertex color with Maya API inside a MPxDeformerNode deformer node. The main trick is to not forget to call MPxDeformerNode::setDeformationDetails() note that setVertexColor() sometimes won't work for some strange reasons... Instead use setVertexColors() it's faster anyway.

MStatus Inherits_mpxdeformer::deform(MDataBlock& block,
                                     MItGeometry& geom_it,
                                     const MMatrix& object_matrix,
                                     unsigned int multi_index)
{
    
    try
    {
        MStatus status;
        MArrayDataHandle out_array = block.outputArrayValue(MPxDeformerNode::outputGeom, &status);
        mayaCheck(status);
        MDataHandle houtput = out_array.inputValue(&status);
        mayaCheck(status);
        MFnMesh mesh_fn ( houtput.asMesh(), &status);
        mayaCheck(status);
        MColorArray colors;
        MIntArray vertex_idx;
        for (geom_it.reset(); !geom_it.isDone(); geom_it.next()) {
            colors.append(1, 0, 0, 1);
            vertex_idx.append(geom_it.index());
        }

        mesh_fn.setVertexColors(colors, vertex_idx);
        houtput.setClean();
    }
    catch (std::exception& e)
    {
        maya_print_error( e );
        return MS::kFailure;
    }
    return MStatus::kSuccess;
    
}

// -----------------------------------------------------------------------------

void Inherits_mpxdeformer::postConstructor()
{
    mayaCheck( MPxDeformerNode::setDeformationDetails(MPxDeformerNode::kDeformsColors) );
}

Set per vertex color within a MPxNode

I would advise against this method for the following reasons:

• more complex to implement
• prone to application crash when used in conjunction with other Maya features such as "mirror skin weights" or "copy skin weights".
• huge lag in some scene when used in conjunction with PaintScriptTool command.

Overall too many problems that I could not solve. It seems the way input/output mesh are connected and copied might have been the cause of all the issues but I never was able to solve them.

Here is how to do it for your reference. You need to create your own input and output mesh attribute, connect your node in between the source and destination mesh. Then copy the data and modify the color using MFnMesh:

MColorArray Node_color::_current_colors;
MIntArray Node_color::_vertex_idx_list;
MObject Node_color::_s_in_mesh;
MObject Node_color::_s_out_mesh;

// -----------------------------------------------------------------------------  

MStatus Node_color::initialize()
{
    try
    {
        MStatus status;
        MFnTypedAttribute typed_attr;
        _s_in_mesh = typed_attr.create("in_mesh", "im", MFnMeshData::kMesh, MObject::kNullObj, &status);
        mayaCheck( status );
        mayaCheck( typed_attr.setStorable(false) );
        mayaCheck( typed_attr.setConnectable(true) );
        mayaCheck( addAttribute(_s_in_mesh) );

        _s_out_mesh = typed_attr.create("out_mesh", "om", MFnMeshData::kMesh, MObject::kNullObj, &status);
        mayaCheck( status );
        mayaCheck( typed_attr.setStorable(false) );
        mayaCheck( typed_attr.setConnectable(true) );
        mayaCheck( addAttribute(_s_out_mesh) );
        attributeAffects( _s_in_mesh, _s_out_mesh );
        
        return MS::kSuccess;
    }
    catch (std::exception& e)
    {
        maya_print_error( e );
        return MS::kFailure;
    }
    return MStatus::kSuccess;
}

// -----------------------------------------------------------------------------        

MStatus Node_color::compute(const MPlug& plug, MDataBlock& data_block)
{
    try
    {
        MStatus status;

        if (plug.attribute() != _s_out_mesh) {
            return MS::kSuccess;
        }

        // Copy in mesh as is in the out mesh
        mayaCheck(status);
        MDataHandle in_mesh_data  = data_block.inputValue(_s_in_mesh, &status);
        mayaCheck(status);
        MDataHandle out_mesh_data = data_block.outputValue(_s_out_mesh, &status);
        mayaCheck(status);
        mayaCheck(out_mesh_data.copy(in_mesh_data));
        mayaCheck(out_mesh_data.set(in_mesh_data.asMesh()));

        bool display_color = true;
        if(display_color)
        {
            MObject mesh = out_mesh_data.asMesh();
            MFnMesh mesh_fn( mesh, &status);
            mayaCheck(status);

            int num_verts = mesh_fn.numVertices(&status);
            mayaCheck(status);
            if( _current_colors.length() != num_verts)
            {                               
                mayaCheck(_current_colors.setLength(num_verts));
                mayaCheck(_vertex_idx_list.setLength(num_verts));
                for(int i = 0; i < num_verts; ++i) {
                    _current_colors.set(MColor(0.5f, 0.5f, 0.5f), i);
                    _vertex_idx_list[i] = i;
                }
            }
            
            mayaAssert(_vertex_idx_list.length() == num_verts);
            mayaAssert(_current_colors.length() == num_verts);
            
            for(int i = 0; i < num_verts; ++i) {                
                MColor c = compute_color(i);
                _current_colors.set(c, i);
            }
            
            mayaCheck( mesh_fn.setVertexColors(_current_colors, _vertex_idx_list) );
        }
        data_block.setClean(plug);

    }
    catch (std::exception& e)
    {
        maya_print_error( e );
        return MS::kFailure;
    }
    return MStatus::kSuccess;
}

Original post here

No comments

Name

Email (optional field, I won't disclose or spam but it's necessary to notify you if I respond to your comment)

All html tags except <b> and <i> will be removed from your comment. You can make links by just typing the url or mail-address.

Anti-spam question:
√4 + 40 = ?

Only those who dare to fail greatly can ever achieve greatly.
-- Robert F. Kennedy