Scripting Maya

0. Thoughts on Maya

It's easy to find learning a piece of software like Maya daunting. At most schools, it takes two full semester-long classes in 3D animation before someone really understands how to create 3D animation from beginning to end, and that doesn't even cover the advanced techniques and features buried inside of the program. Scripting something like Maya can seem an even more difficult task, as most of the time, we know Maya from its user interface, and not from the way it stores data internally.

What Maya does do for you is echo the API commands in its own scripting language, MEL. Maya's UI is written in MEL, at the same level I was working when I started writing my scripts. That means that the source code to the Maya tools written in MEL would provide a valuable learning tool in my attempt to learn the ins and outs of the program. Autodesk don't, however, provide step-by-step tutorials or even an explanation of its API by feature, rather, Maya developers are expected to learn the program through a combination of its command reference and the UI source code.

In addition to MEL, Autodesk provide a binding into Maya for the popular programming language Python. If you've already begun scripting for Maya in Python, you probably already know the weaknesses in the provided Python layer can make for some ugly code. The good news is that translating MEL into Python is fairly obvious. The bad news is that it results in code like this:

curQual = cmds.modelEditor (currentPanel, q=True, rnm = True)

if curQual == 'base_OpenGL_Renderer':

cmds.modelEditor (currentPanel, e=True,displayAppearance = "smoothShaded", displayLights = "default", displayTextures = True, rnm = "hwRender_OpenGL_Renderer")

else:

cmds.modelEditor (currentPanel, e=True,displayAppearance = "smoothShaded", displayLights = "default", displayTextures = True, rnm = "base_OpenGL_Renderer")

Calling one function for everything relating to, say, 3D Paint or a model view can get confusing quickly, especially since you're either setting or getting flags that are explained only briefly in the documentation. In order to see how these functions are actually used, you need to dive into the Maya source code and pick out the relevant bits separate from the code that creates the user interface. In practice, this means turning on “Echo All Commands” inside of Maya, doing something you want to automate inside of Maya, and then grabbing the name of the UI call from the script editor and grepping through the hundreds of thousands of lines of MEL lurking in Maya's /scripts folder. I think most Python hackers would agree that this development process and way of laying out the API is decidedly Unpythonic (http://en.wikipedia.org/wiki/Pythonic#Programming_philosophy). Some projects exist, such as PyMel (http://code.google.com/p/pymel/), that make Python syntax and MEL objects make more sense with each other, but unfortunately we didn't have time to look into those for this class.

With all of that said, once you get the hang of development in Maya, you quickly realize that you have a powerful 3D package at your control that allows you to take advantage of some very advanced features without having to code a renderer for them yourself. In terms of time taken, using Maya's particle engine or a 3D paint tool is undoubtably faster than writing your own, and this class is all about speed, so learning how to manipulate these tools with code was necessary to the mission of the class.

When we first decided talking about the kinds of tools a programmer might contribute to the class, a lot of my initial ideas centered around creating art programmatically. I thought about writing some code that laid out a virtual world based on a few basic objects that populate it, so that a program could be fed a few models of carnival booths and rides and then spit out an intelligently laid out scene. Another idea we talked about was taking a rigged model and generating walking animation for it based on the layout of its limbs, similar to what happens in the game Spore.

We eventually decided to pursue a different path, and focus on improving the interaction between the animator and Maya. We talked a lot about how the animator can get the ideas in his or her mind into Maya as quickly and efficiently as possible, while still having creative control over what results.

1. 3d Paint Tool

We decided that one group would start exploring the possibility of 3D matte painting, which would greatly reduct the amount of world building and modeling necessary to create the scene for our movie, without a sacrifice in visual quality (provided the camera stayed put). I was tasked with creating a 3D paint tool to streamline the painting process in Maya.

This tool would include a “Prime” button that would let an artist prepare a mesh for 3d painting a single click. Naturally, this requires a lot of steps to complete, and I ultimately decided to give the artist the option of turning off each step as needed. This saves the artist time in the beginning, but we realized that the real time saver would be minimizing the number of times the artist needs to go back to the Maya 3d Paint panel and change settings. For that reason, I added some hotkey management code that would swap out common single-button hotkeys for 3d Paint specific functions (e.g. C for choose color, G for grab color from screen). The animators really liked this, and said that specifically saved them a lot of time. Another feature we were interested in was moving the current 3d paint texture map to Photoshop. That eventually became the next tool in itself.

One final thing I should mention on the 3d paint tool is that there were some troubles getting it working at first. One peculiar thing about Maya 2008 and 8.5 is that it requires all named commands to be linked to MEL code, even in Python. This has apparently been fixed in 2009 with the sourceType flag on the nameCommand command, but most of the computers in Hampshire's lab weren't running 2009, so I used a quick fix using the MEL command python(), to effectively jump from Python to MEL and back to Python. A little unusual, but it worked...

Another problem with the 3d paint tool was its tendency to not restore the hotkeys once you were done using it. This was a nuisance that the animators had to go into the hotkey editor and fix themselves. The problem is that even though my script could restore the hotkeys, Maya didn't seem to provide a way to trap the event of the script ending. Strangely enough, it does provide events for when your tool window is minimized and restored (minimizeCommand/restoreCommand flags on the window command), but even these weren't called at the appropriate time. I ended up adding a restore all hotkeys button to the window for the artist to use before they finished with the tool.

2. Getting Renders out of Maya and Into Photoshop

This posed an interesting problem at first, as we weren't sure how to divide up the painting duty between Photoshop and Maya's 3D paint, and in which order they would be used. It seems like the painters were mostly outlining in Maya and finishing up all of the detail painting in Photoshop, so it seemed necessary to write a tool that could facilitate an export to Photoshop as quickly as my previous tool made it possible to manage a palette inside of Maya.

Since our artists were already dividing up the textures for 3D paint into categories and render layers in Maya, it seemed to make sense to develop a tool that could organize a camera, its materials, and its child render layers and textures all in one, while pushing renders of each layer out to Photoshop. I found the render command in Maya script, which I preferred over Maya's external renderer because it let us have a scene file open as we rendered it and had built in flags for rendering specific render layers at a time. By default, it rendered to .iff with bad transparency. Maya provides a function called convertIffToPsd which I started using in the first version of my code, but Maya's generated .psd files didn't retain transparency as alpha (instead painting the empty void black), so we tried several different file formats. Maya's export to .png preserved transparency perfectly, so I devised a system to include such a file in a .psd by means of a smart object (that's Photoshop for an updatable reference to an image stored elsewhere) and a Photoshop actions file (.atn) to update it.

Getting Maya to spit out a .png as opposed to a .iff required some hunting through the MEL code as well. The trick is to call setAttr ('defaultRenderGlobals.outf',32), with 32 being Maya's special identifier for .png files. You may want to setAttr the value back to 7 (.iff, the default) after your call to render.

I built an early version of this tool and demoed it for the class, but we ultimately decided we wanted to lose the organizational overhead and let the artists push out a single render reference image themselves that they could paint over, and “spray” the new paint over the old texture back in Maya.

Getting that single image out was surprisingly easy, and the script ended up being deployed as a windowless button that you could press and generate a quick render of the selected camera on your desktop.

3.Creating a Hypergraph Shader

Having created a series of paintings on top of a quick render, it became necessary to plan how to move the new paint back into Maya. Basically, we moved the organization task of fitting the painting images in place from the “before” Photoshop stage to the “after”.

The two animators, Taryn and Tatiana, who were painting in the class and provided commentary on the tools took two different approaches. One chose to use a quad shading switch in Maya, which is a special shadingNode that acts as kind of a multiplexer for pixel values within a shader. Because we had so many paintings that would have to fill a single surface, we thought it would be a good idea to have the shader make an intelligent decision as to which psdNode to source the image from, instead of building shaders for each image.

We did run out of time at this point in the semester, so I was never able to build a final version of this script, but I did have an early version to show off on the last day. In this final tool, a painter would first select a camera and lock it into place, and then create some materials associated with that camera's shot. The painter could then assign each material (which already had a psdNode created for it in hypershade). Then, the painter would select a meshes they wanted to texture, and assign it to a material (it's important to realize that the relationship between meshes and materials is many to one), which would then update the hypergraph shader. Finally, when they were done, they could mash a final button which would connect the whole mess together through the quad shading switch. Seeing what this looked like created manually made it seem like a process in dire need of speeding up.

The animators decided later in that class to move to multiple shaders for other reasons, which does eliminate the problem of creating the quad shading switch, although a tool to create multiple shaders could certainly have been just as handy, and that most likely is what I would have started on next if I had more time.

4.Conclusion

I hope that the code I've written as part of this class becomes helpful to CG students here at Hampshire and animators anywhere that might be reading this blog. Improving the interaction between technically minded artists and necessarily complicated software is a rich area to explore, and I wouldn't be surprised if there were big gains in efficiency to be had in the process of streamlining the interface of a massive piece of software like Maya. As for me, I'm satisfied to have learned a lot more about the techniques and terminology surrounding computer graphics, and to have picked up Maya scripting skills along the way. I'm still impressed by the program's size and capability, but no longer intimidated, and plan on continuing to play around with Maya this summer.