Use SVN to get only changed files in JSFL (javascript)
Another quick snippet of code! This time, some functions in Javascript/JSFL on fetching lists of changed/diff’d files from SVN to process. Javascript is quickly growing on me, despite it’s often preached shortfalls.
// This is probably the wrong way to create a javascript object. lol.
function fileListGen() {
}
fileListGen.prototype.runSvn = function (cmd) {
// super secret undocumented command!
FLfile.runCommandLine(cmd);
}
// pull out just the filenames from the very verbose SVN return.
fileListGen.prototype.parseSvnOut = function(fileLoc) {
var rawString = FLfile.read(FLfile.platformPathToURI(fileLoc));
var lines = rawString.split("\n");
var svnDiffedFiles = [];
// ignore lines with no filenames. trim otherwise.
for (s = 0; s < lines.length; s += 1) {
var str = lines[s];
if (str.indexOf("Index:") == -1) continue;
str = str.replace(/(\r\n|\n|\r)/gm,"");
svnDiffedFiles.push(str.split(": ")[1]);
}
return svnDiffedFiles;
}
// call SVN. Pipe output to temp file. Parse.
fileListGen.prototype.getChangedFiles = function(baseDir) {
var outFileURL = baseDir + "\\svnOut.txt";
var cmd = "svn diff \"" + baseDir + "\" > \"" + outFileURL + "\"";
// delete our old output file if it exists.
if (FLfile.exists(FLfile.platformPathToURI(outFileURL))) {
FLfile.remove(FLfile.platformPathToURI(outFileURL)); // delete our file if it already exists.
}
// execute the SVN command (strange behavior from flash here..
//not entirely sure when it considers the CMD execution complete?)
this.runSvn(cmd);
// parse the txt and return the output.
return this.parseSvnOut(outFileURL);
}
And once again I return to work!
Fix missing bitmaps in just a few lines of maxscript
It seems everyone has tools to retarget missing bitmaps in maxscript, but nobody seems to share how it’s done.
Here’s a really simple solution to the problem using a dab of dotnet and zero error checking, which wouldn’t be very hard to add.
fn findSameNamedAsset asset newDir = ( local filenameOfAsset = (DotNetClass "System.IO.Path").GetFileName asset local di = DotNetObject "System.IO.DirectoryInfo" newDir local files = di.GetFiles filenameOfAsset (DotNetClass "System.IO.SearchOption").AllDirectories files[1].directoryName ) fn repathMissingBitmaps newDir = ( ATSOps.Visible = true local arr = #() ATSOps.GetFilesByFileSystemStatus #Missing &arr for asset in arr do ( AtsOps.clearSelection() AtsOps.selectFiles #(asset) local newFileName = findSameNamedAsset asset newDir AtsOps.SetPathOnSelection newFileName ATSOps.refresh() ) OK )
And back to work I go!
Maxscript Heap Tool
I’ve been working with windows forms. I was a little annoyed when I kept crashing 3dsMax via outOfMemroy exceptions after loading hundreds of bitmaps. So, I hastily created this little UI to check my current memory standing. It even has a little log. How quaint.
If the rollout is open, you can call HeapTool.registerEvent
If you want to record the memory change from a certain point, you can call heapTool.setMemState() to register the current heap size as the value we wish to compare to. for example:
heapTool.setMemState() if doesFileExist thumbnailLoc then thumbImg = loadNetBitmap thumbnailLoc else thumbImg = loadNetBitmap defaultThumb heapTool.registerEvent "Add dotNet Bitmap" quiet:true
Here’s the entire, copy-pasta’d, raw, unclean, uncaring tool code. It should be noted that the “large number” types that are returned from “heapSize” and “heapFree” are very nastily hacked into integers. It’s not meant to be accurate. moreso, it gives a picture of what actions are costing lots of memory.
Anyway, use at your own risk. (on a side note, man, the formatter for maxscript looks terribad!)
rollout heapTool "Heap Tool" width:320 height:380
(
label lblUsed "Used Memory: " pos:[8,8] width:104 height:16
progressBar pbUsed "" pos:[8,24] width:304 height:16 color:(color 30 10 190)
label lblFree "Free Memory: " pos:[8,48] width:104 height:16
progressBar pbFree "Free Memory" pos:[8,64] width:304 height:16 color:(color 30 10 190)
button btnUpdate "Update" pos:[8,88] width:80 height:24
button btnGc "gc()" pos:[96,88] width:40 height:24
button btnFree "free bmps" pos:[144,88] width:72 height:24
button btnUndo "clear undo" pos:[224,88] width:64 height:24
listBox lbxMemLog "" pos:[8,120] width:304 height:19
local lastMemState = 0
local eventBuffer = #()
fn getFormattedHeapVal =
(
local h1 = heapSize as string
h1 = (substring h1 1 (h1.count - 1)) as integer
local h2 = heapFree as string
h2 = (substring h2 1 (h2.count - 1)) as integer
#(h1, h2)
)
fn setMemState =
(
h = getFormattedHeapVal()
lastMemState = h[1] - h[2]
)
fn updateDialog =
(
-- cheat the sheeit out of the values.
h = getFormattedHeapVal()
lbxMemLog.items += eventBuffer
eventbuffer = #()
local used = (h[1] as float - h[2] as float) / h[1] as float * 100.0f
local free = (h[2] as float / h[1] as float) * 100.0f
lastMemState = h[1] - h[2]
pbUsed.value = used
pbFree.value = free
)
fn registerEvent eventName quiet:false=
(
h = getFormattedHeapVal()
local delta = ((h[1] - h[2]) - lastMemState)
local changeWord = " allocated "
if delta < 0.0f do changeword = " released "
local element = ("\""+ eventName+ "\"" + changeWord + (delta as string) + "\n")
if quiet then
(
setMemState()
eventBuffer += #(element)
)
else
(
lbxMemLog.items += #(element)
updateDialog()
)
)
on btnUpdate pressed do updateDialog()
on btnGc pressed do (gc(); registerEvent "garbge collect")
on btnFree pressed do (freeSceneBitmaps(); registerEvent "Free Bmps")
on btnUndo pressed do (clearUndoBuffer(); registerEvent "Clear Undo")
on heapTool open do (lastMemState = 0; updateDialog())
)
createDialog heapTool
Git Exceptions.
I’ll admit, I work in a small office and don’t really have access to a large amount of resources, but we make do. What I wanted to try out was using our networked drive as a server for a file system. I didn’t want to sacrifice computers for a server. More importantly, I needed a way to share our assets between the few people who work on characters. And, I didn’t want to rely on a tedious copy/paste system via windows.
So, I thought i’d give Git a go. Git initially presents itself in a very good way. It works over networked drives, is decentralized, and feature-rich enough to allow for mostly-easy manipulation of data back and forth between our central ‘bare’ repository and client machines. Git has a large amount of support from various groups of opinionated programmers. I even saw some people mention using it on TAO. All of these are good signs, so thumbs up, right?
I had read multiple times while doing preliminary research that git was quite difficult to really Grok and this proved itself true very quickly. I constantly found it a struggle to push and pull binary assets between remote repositories without something going wrong: Files missing, files being overwritten, unexplained errors when submitting or syncing assets. The list continues. The net product of me using git was an extra hour or two at the end of the day insuring my intern and I were sync’d up with mostly identical assets.
So, last week we decided to abandon git and sacrifice a computer to act as our SVN server. Linus Torvalds would think i’m stupid.
Git’s framework is both individualist and democratic. It expresses itself as a trust-driven system of dynamically moving branches of a single software expression. Changes and ideas are promoted naturally via developers selecting what’s best in a tree of branches and evolving the software to a better state. Game development simply does not work this way. Game development is dictated, mostly rigid, with goals that we’d like to think are solid and achievable. Movement other than forward towards our imaginary goalposts is advertised as unwise. Independence is not a primary goal in game development, and from my experience, democracy within a game company is mostly ineffective.
But, lets not get too meta. I think git’s problems mainly stem from the lackluster GUIs that layer on top of the git shell experience. Both GitGUI and Git Extensions lack a clear purpose when performing actions that have tangible results in SVN. In Git extensions, where I spent most of my time, everything felt so broad. Changes were submitted as huge chunks of files that had to be merged in their entirety, and any conflicts were simply overwritten instead of delicately sorted out. This hardly does anything to help understand the software.
In hindsight, it probably would have been better to work on asset changes within their own branches that were then moved around the network, and then merged for a final result. This idea would have fit better within the git paradigm, But, is so contradictory to the idea of working with assets as an artist that it’s hard to want to push for a change like this. Considering i’m going to be on the same project for the next year or so, It’s going to be a while before I am ready to give git another attempt.
-Nick
Generic read/write methods for generic data in maxscript
I’ve spent a lot of time experimenting with multiple ways of saving data from 3dsmax: INI Files, raw text files written and read via maxscript (ew), and some pretty long-winded C# save/load classes. Each had its own pluses and minuses, but it was a chore to make any adjustment to the read/write code when anything about the structure of the data changed because it was all so very static in its functionality. The ideal solution is to allow our data to define our exports in a dynamic and flexible way that’s reusable.
The key to understanding how this works within max is to take a look at the maxscript command “getPropNames”, which allows us to query the names of all properties within a given structure or object. With Maxscript we can call this command using the self-referencing keyword “this” inside of a struct to gather all properties within the current context. Combine this with “getProperty” and “classof” and we have a simple way to query every property name, property value, and property type for most every structure within Maxscript. It is also a possibility to implement this within C# using similar methods, but we’ll focus on maxscript for this post:
struct animalType ( prop1 = "bear", prop2 = "cat", /* lots and lots of properties */ fn exportData = ( local properties = getPropNames this local propValues = for property in properties collect (getProperty this property) local propTypes = for value in propValues collect (classof value) ) )
These three commands give us three parallel arrays of all properties and methods within our structure, and most importantly, enough information to reconstruct the data when we want to re-load it again. This does not however, handle arrays or node references, which complicate things and will be discussed in a further post.
Moving on, we want to be able to write all of this accumulated data to some sort of file to save it. My favorite format this month is XML, so we’ll go ahead and utilize C#’s wonderful XMLWriter class to handle our output. Using our gathered data, We simply initialize our XMLwriter class and then iterate over our arrays, producing a single element for each property within our structure. At less than 40 lines with white space, it’s a pretty small piece of code that handles a lot of scenarios.
/*inside of a struct...*/
fn exportData xmlFile =
(
/* get all of our properties and methods*/
local properties = getPropNames this
local propValues = for property in properties collect (getProperty this property)
local propTypes = for value in propValues collect (classof value)
/*build a format class so we indent out output. makes it readable */
local xmlFormatter = dotnetobject "System.Xml.XmlWriterSettings"
xmlFormatter.Indent = true
/*Instantiate our xml writer class */
local XMLWriter = dotnetclass "System.Xml.XmlWriter"
XMLWriter = XMLWriter.Create xmlFile xmlFormatter
/*write the start of our document*/
XmlWriter.WriteStartDocument()
XmlWriter.WriteStartElement("structDef")
/* Iterate over all of our gathered properties, except methods*/
for i = 1 to properties.count where propTypes[i] != MAXScriptFunction do
(
/* create an element for each property. save it's type and name as an attribute */
XmlWriter.WriteStartElement("property")
XmlWriter.WriteAttributeString "name" (properties[i] as string)
XmlWriter.WriteAttributeString "type" (propTypes[i] as string)
XmlWriter.WriteString (propValues[i] as string)
XmlWriter.WriteEndElement()
)
/* write the tail of the xml document */
XmlWriter.WriteEndElement()
XmlWriter.WriteEndDocument()
/* close the file. finalizes the write and puts the data to the target */
XmlWriter.Close()
),
Running the above code yields xml similar to the example below.
<structDef> <property name="numEyes" type="Integer">6</property> <property name="id" type="String">hypnocat</property> <property name="type" type="String">cat</property> <property name="numLegs" type="Integer">15</property> <property name="averageLegLength" type="Float">70.31</property> </structDef>
When we load this data, are presented with an interesting challenge. Any data that we pull from the XML file is going to be of type string. If we assign all of these raw values straight back to our structure, we’re going to end up with a struct full of strings and lose any previous and possibly important type data regarding our parameters. Luckily, we can use our self-referencing to cast/convert all of our strings to the proper type.
fn loadData xmlFile =
(
/*open our xml file */
local xmlReader = dotNetObject "System.Xml.XmlDocument"
xmlReader.Load(xmlFile)
/* select all xml nodes of type 'property' */
local properties = dotNetClass "System.Xml.XPathNodeList"
properties = xmlReader.SelectNodes("structDef/property")
/*get the enumerator and iterate over all nodes*/
enum = properties.GetEnumerator()
while enum.MoveNext() do
(
/*build correct data type*/
local thisPropertyName = enum.current.Attributes.Item[0].Value
local thisPropertyType = enum.current.Attributes.Item[1].Value
local thisPropertyValue = enum.current.InnerText as (classof (getProperty this thisPropertyName))
/*set the property generically*/
setProperty this thisPropertyName thisPropertyValue
)
)
The key to preserving our property types is (classof (getProperty this thisPropertyName)). We can’t cast our values to their appropriate type using just the name of the Type as a string, so we work around that limitation by relying on the structure our load function exists within. Using the property name we’ve loaded, we call getProperty to fetch the already-existing value within our struct . This returns a “Type” rather than a string, and can be used to cast using “as”. Clever (or at least, I think so.)
This setup should handle most common struct properties except for arrays and node references, which we’ll handle later. This post is already incredibly long!
HTML: Simple. Fun.
I’ve spent all day coding Maxscript and fixing 3d assets at work. Somehow, coming home and spending the evening writing some simple HTML and CSS for my website refresher has been completely satisfying and not tiring whatsoever.
It’s amazing how much different my motivation levels have been now that i’m actually employed and not scrambling for a job.
I think i’ll be condensing my tech art page into a demo reel and some tool demos tomorrow evening. Excited for the unknowns of the future and feeling confident!
On a side note, working with CSS tonight has revealed the lexicon of how CSS is actually used within a website. CSS is actually rather, quite abstractly, SQLish. But most people wont understand this, or i’m just really tired.
-Nick
Python application stuff
I’m working on a new python application at the moment. It’s a timer based productivity software that’s supposed to use the “set egg timer to 25 minutes and WORK” methodology of getting stuff done. Oddly, i’m not actually using this methodology to keep myself motivated – coding is engaging enough as-is to not require such implements.
Anyway, today, the very base of the timing update systems were put into place. I wanted to have a system that watched over multiple objects that kept track of their own elapsed time and updated at specific intervals. This would allow users to setup multiple milestones with unique “work periods” for differing task lengths. To do this, my code uses three specific classes that percolate time data in a semi-downward motion.
At the top of the hierarchy is the overall time-ticking object. The time object sits in looped perpetuity, continually telling the manager class (described later) to update its children at a designated time interval. This small class inherits from the awesome ‘threading’ module, which allows its functionality to be executed in, you guessed it, another thread – this allows the perpetual while loop to not interupt the flow the overall app. The timer can be started by invoking the inherited method threading.start(), which initiates the code within run(). The timer can be then stopped by invoking threading._stop(). Stopping the execution of the thread is a universal break for all of our dependant timers.
using threading
using time
updateInterval = 1.000
class ticker(threading.Thread):
timeManager = None
def run(self):
if self.timeManager == None:
self._stop()
print ("time manager not specified")
return
while True:
time.sleep(updateInterval)
print("ticking")
self.timeManager.tick()
Next is our management function that keeps track of all of our timers. I’m beginning to think that this part of the application is an unnecessary middleman, but at the current state it remains. The task manager keeps a list of timer objects that are then updated at our tick intervals. It iterates over these objects, and calls the assumed-existing function ‘update’, to percolate our time ticks down the chain. the task manager ensures unique names within timerObjectList as to allow searching for specific timers (yet to be implemented).
class taskManager():
totalElapsed = 0.000
timerObjectList = []
def __init__(self):
print("task manager initialized")
def tick(self):
self.totalElapsed += updateInterval
self.updateTimerObjects()
def updateTimerObjects(self):
#print("updating timer objects")
for timerObj in self.timerObjectList:
print("updating timer object " + timerObj.name)
timerObj.update()
def addTimer(self, newTimerObject):
print("adding element")
for timerObj in self.timerObjectList:
if timerObj.name == newTimerObject.name:
print("name not unique")
return False
print("adding timer object")
self.timerObjectList.append(newTimerObject)
return True
def removeTimer(self, timerName):
i = -1
timerToRemove = False
for timerObj in self.timerObjectList:
i += 1
if timerObj.name == timerName:
timerToRemove = i
if timerToRemove:
self.timerObjectList.remove(timerToRemove)
return True
else:
print("object with name " + timerName + "does not exist")
return False
Currently the sloppiest of the bunch is the production timer object. The production timer runs for an allocated amount of time, then pauses (for the user to take a break). The user then resumes the production timer to start a new ‘cycle of production’. Rinse and repeat.
The biggest flaw sofar in the entire system is that the percolation of time downwards from the ticker class an inaccurate representation of real elapsed time. Times collected will vary by micro of milliseconds from the actual time. Eventually the system will collect time by checking time.clock() to gather correct elapsed time.
class productionTimer():
name = "default"
updateMe = True;
prevUpdateTime = 0.000
elapsed = 0.000
cycleLength = 0.000
pauseLength = 0.000
cycles = 0
def __init__(self, timerLength, timerName):
print("placeholder")
self.name = timerName
self.cycleLength = timerLength
def update(self):
if (self.updateMe):
self.elapsed += updateInterval
print("being updated!")
if self.remainingTime() <= 0:
print("cycle complete!")
self.cycles += 1
self.elapsed = 0.000
self.pause()
def remainingTime(self):
return self.cycleLength - self.elapsed
def pause(self):
self.updateMe = False
print("paused!")
def resume(self):
self.updateMe = True
print("resumed!")
Next on my list is cleanup for these few classes, and the beginnings of a pyQT gui. I’ve been playing with the pyQT module in Maya, and am quickly discovering it to be an awesome GUI creation system. Hats off to it! (whatever that means).
-nick
Finished with the lava shader, for now
Today I finished up my work on my lava shader. It turned out pretty nice!
I have a narrated demo video of the shader here: http://www.youtube.com/watch?v=pScLYuWuirw
The shader can be downloaded Here A max 2010 demo scene can be had Here
This project was a fantastic learning experience, and it leaves me wanting to spend more time on shaders. It raised some general questions that I’ve not yet had time to tackle as well:
- How can we tie geometry shaders into materials to make interesting wavy effects? water? bubbly surfaces?
- How can we use more information from the mesh’s world to adjust that object’s visual appearance beyond pre-rendered cube maps?
- It is possible to write a maxscript interface for the hlsl shader. Is this effort even worthwhile?
- Why is 3dsmax mapchannel ordering so wonky?
HLSL wise, I still want to spend more time with screen space effects, and take a look at some fun challenges such as color adjustments using LUTs and writing some sort of mood/environment color adjustment schema.
Meanwhile, it’s time for me to transition into an area that i’ve been neglecting for some time: Python.
Also, some awesome blogs to take a look at :
http://www.jshopf.com/blog/
http://www.rtrowbridge.com/blog/
http://adammechtley.com/
Magma distortion
Today I sat down to convert my messy and convoluted lava shader into a cleaner two-pass variety. Previously, the shader required me to do lots of samples, ifs, lerps and the such. For example, this little snippet of code lerps between the magma and igneous rock layers depending on the value of the alpha. The alpha used vertex paint, and a user-defined float “lavaAmount” to control the amount of magma being shown.
// sample both the rock and the magma diffuse tex
float4 rockDiffuse = tex2D(s_IgneousRockTex, vert_in.UV0);
// build a lerp value between the igneous tex and the magma tex.
//int because we only want 1s and 0s. no floaty values.
int lerpValue = clamp((rockDiffuse.a + lavaAmount + vert_in.vColor.r), 0, 1);
// finally, build the diffuse texture.
float4 diffuseTex = lerp(rockDiffuse, tex2D(s_magmaTex, ( vert_in.UV0 + -(wTime/80)) ), lerpValue);
The normal mapping was equally icky-gross.
// sample and average the normal map and panning map.
float3 magmaNormal = (tex2D(s_MagmaNormalMap_surface, (vert_in.UV0 * 0.2 + -(wTime / 80)) ) * 0.5f ) +
(tex2D(s_MagmaNormalMap_panning, vert_in.UV0 * 1.2 + (wTime/ 100)) * 0.5f );
float3 rockNormal = tex2D(s_IgneousRockNormal, vert_in.UV0);
// If we're on the igneous level, use igneous. else, use magma.
float3 nrml = lerpValue < 1 ? rockNormal : magmaNormal;
// finally, move that vector to world space.
curNormal = mul(nrml * 2 - 1, tanToWorldSpace);
Now with two passes, Igneous rock layer is simply alpha’d out by our previously-defined lerp value. This layer is the second pass, which is drawn over the magma layer.
// sample our igneous rock texture.
float4 rockDiffuse = tex2D(s_IgneousRockTex, vert_in.UV0);
// build our lerp values. The 1 minus is simply to make semantics of the UI match the value.
int lerpValue = (1 - lavaAmount + rockDiffuse.a);
// make our 1 -> 0 and our 0 -> 1
lerpValue = !lerpValue;
// and then, simply set our values.
output_color.rgb = rockDiffuse;
output_color.a = lerpValue;
Here’s a video of my magma layer. It incorperates the averaging of the two normals, which are then used to distort the UV channel. This is a first-pass of this effect.
Here’s the code for the distortion and the normal map. After sampling our normal map the code distorts our UV input and then re-samples the normal map to ensure that the normal map warps with the diffuse map.
// build the texture of our normal maps;
curNormal = tex2D(s_MagmaNormalMap_surface, vert_in.UV0);
curNormal += tex2D(s_MagmaNormalMap_panning, vert_in.UV0 * 1.2 + (wTime/ 100));
//Distort our UVs. the scalar 0.1 is scale value for the size of the distortion.
vert_in.UV0 += curNormal.xz * 0.1;
//Now re-sample the normal map with distorted UVs to get a normal from the new location;
float3 magmaNormal = tex2D(s_MagmaNormalMap_surface, vert_in.UV0);
magmaNormal += tex2D(s_MagmaNormalMap_panning, vert_in.UV0 * 1.2 + (wTime/ 100));
//Because we added two normal maps together, divide by two to get the mean of the normals. (average)
magmaNormal /= 2.0f;
curNormal = mul(magmaNormal * 2 - 1, tanToWorldSpace);
This shader is coming along quite nicely, and the split to two passes makes it quite readable, unlike before.
Ramp shading.
Recently, I’ve gotten my base Blinn/Phong shader to a state where i’m relatively happy with it. It’s posted to my portfolio on the Tech Art Page. I plan on using this shader in the future as the base for any material-related shaders. For example, today I wrote a very fast ramp shader mod for the blinnphong. This took me little under 30 minutes to implement, and the results are pretty nice!
Also, using distance attenuation on your shader causes your light brightness to increase as it gets closer to the surface of your object. As this lambertian term of brightness is also used as lookup on the ramp texture, you get a neat effect if you don’t clamp it between 1 and 0. Look!:
The ramp texture loops around as the “U” value increases beyond 1, and wraps to the front of the texture. This causes the banding pattern shown in the above video. This issue is fixed by setting your sampler_state AddressU and AddressV to ‘clamp’.
I love this stuff!
