It was a crazy day, July 31st on the deadline of the project. Everything that could go wrong went wrong for us. Now we are not sure if the project would be completed in time for Siggraph 2009 demo.3PM EST was the deadline for the animators (Dan & Paula) to turn in their work for Filipe to do the final touches and then set it up for the renderfarm. Paula finished her work at 2PM EST and sent out the email with “Subject: DONE!” and Dan did the same at around the same time. Now Filipe started to work on the finishing touches around 3PM as planned to work our HD version of the Preview. That is when everything went topsy turvey.
Filipe started having problems with SVN permissions. He could not move some files from ZanQdo’s folder on svn to his to link textures for the turkey that he build. His initial SOS messages were through which took sometime with the back and forth messaging when time was off the essence. He was advised to join in on the chatroom which he did. Rexy guided him through the clean up and he was up and running in about 30 minutes. It was 5PM. When he started to do his final touches.
Rexy, Mark, Konstantine were working on Animux Version 2.0 Beta for siggraph in parallel (all of us only connected through the internet on IRC). We created the version 2.0 Beta on thursday night and did our initial testing. However, we missed one most important test, NFS (network file system) mounts, which ofcourse, the renderfarm needs for the slaves to send files to the master. Mark got delayed at his day job and got started 2 hours later to start the renderfarm at around 530PM. Created the Animux images (copy animux) on USB sticks and started booting the diskless slaves using Animux (EASY TO SET UP RENDERFARM FEATURE) by booting into slaves. The slaves would start mount the NFS and stay online for 20 minutes and just hang up. It took us another 3 hours to figure out it was some issues with our kernel and now have to recompile the kernel. So 9PM we still did not have the renderfarm started. We thought of using the Animux Stable Version to bring up some nodes. So, we recreated those Animux USBs with the stable version and started bring up some slaves. Things went well for about 30 minutes and the linksys router stopped to work and everyone connected from outside to the animux network were dropped. (AHHHHHHHHHHH!!!!!)
Filipe was ready with his changes and ready to test out the first render so he could check a few renders and go to sleep. It was not close to 10PM EST and midnight for him in Brazil. Mark brought another Linksys router connected them to the internet and Filipe finished his SVN commit and we were ready to run our first render on a few slaves that were brought up. Around 1PM EST the first frames came online for Filipe to check out, he was tired looked at a few of them and went to bed.
Mark & Rexy started working on the kernel fixes. Rexy was ready with the kernel fix and now needed the same farmto compile the kernel that were rendering on. A decision was made to do exactly that, run them at the risk of killing these diskless “Animux Rackmount Renderfarm”. We did it with great trepidation only to find out they handled the task well. It was the 32bit disk nodes that caused us grave problems. Now that the kernel was fixed and installed into the new image, we did started bringing up the nodes with Animux V2.0 and they were stable. It was around 4AM EST before all the nodes were up and running. We have partial connectivity from the outside, just enough to SVN and monitor the renders. Ted (blendertorenderman.org) volunteered to check to see how the renders were going. Rexy, Mark called it a day went to get a much needed nap for a few hours.
Team was back online at 8AMEST. Woke up to Filipe message that he missed linking some feathers from the turkey. He assured this is nothing to worry and we had 2 choices, render the whole thing or Filipe created a blend file which we render only the missing feather and Dan composites them. The decision to go with the later. By now we had completed 60% of the frames. So, we knew there was no way we could have the preview done before Mark left for Siggraph. But could be done before the Animux BOF. We will know in a day or 2.
Leave a Reply