Een toevallige vondst: een filmpje van Pixar dat duidelijk maakt hoe ze bijna een goot gedeelte van Toy Story 2 kwijtraakten, doordat ‘iemand’ een rm * commando ingaf (‘remove all’). En toen kwam de conclusie dat de backups niet hadden gewerkt ! De technisch directeur van de filmproductie bleek een kopie van de gehele film te hebben gemaakt en die op haar eigen computer thuis te hebben geplaatst. Op deze manier kon het meeste van de film worden gered.
Twee opmerkingen:
- Hoe is het mogelijk dat er maar een backupsysteem werkte en dat niet voortdurend werd geverifieerd of de backup juist was. In de film wordt gesuggereerd dat er 20 tot 30 mensjaren werk verloren hadden kunnen gaan.
- Het is verwonderlijk dat een film moet worden gered door een illegale kopie: het kan niet anders dan in een bedrijf waarin men uiterst voorzichtig is met ‘things getting out’ in haar beleid blij is met kopieen buiten de muren van het bedrijf. Het redt de film, maar daartoe had men wat anders moeten regelenn
Wat gebeurde er werkelijk ?
‘The only prospect then was to roll back about 2 months to the last full backup that we thought might work. In that meeting, Galyn mentioned she might have a copy at her house. So we went home to get that machine, and you can watch the video for how that went… With Galyn’s machine now back in the building, we dupe’d that data immediately, then set about the task of trying to verify and validate this tree, which we thought might be about 2 weeks old. We compared Galyn’s restoral with a much older one (from 2 months prior) and couldn’t determine a clear winner, there were too many inconsistencies. So, instead, we set about the task of assembling what effectively amounted to a new source tree, by hand, one file at a time. The total number of files involved was well into the six figures, but we’ll round down to 100,000 for the sake of the rest of this discussion to make the math easier.
We identified the files that hadn’t changed between the two, and took those straight away. Then there were the files that were on Galyn’s but not on the older one; we took Galyn’s and assumed they were new. Then there were files that were on the older one but not on Galyn’s; we put those in the “hand check” pile, since it is unusual for files to be deleted within a production source tree, and we were suspicious of those deletions. Then there were the files that were different across the two backups, those also went into the “hand check” pile along with any files that were touched more recently than Galyn’s version.
Given that, we had something like 70,000 files that we felt good about, and we poured those into a new source tree. For the remaining 30,000 files, it was all hands on deck.
We checked things across 3 partially complete, partially correct trees… the 2 month old full backup (A) , Galyn’s (B, which we thought was the best one), and another cobbled together tree (C) from the stray files left around from failed renders, backup directories on animator’s machines, some heads of source history that were left untouched, verbose test renders, and other random stuff we could find via NFS elsewhere in the building.
We invited a select few members of the crew back to work straight from Friday -> Monday morning. We took rolling shifts to sleep and eat and kept plowing through, file by file, comparing each of the files in the “to be checked” list from A, B, and C, doing the best to verify and validate them, one at a time, by looking at them in xdiff. In the end, human eyes scanned, read, understood, looked for weirdness, and made a decision on something like 30,000 files that weekend.
Having taken our best guesses at those suspect files, we assembled a new master of ToyStory2. Many source histories were lost as a result, but we had the best version we could pull together. We invited the crew back, and started working again. Every shot went through a test render, and surprisingly, only a dozen or so failed.
I know full well that the following statement will likely blow people’s heads up, but the truth is that more than several percentage points of the show (as measured in numbers of files) were never recovered at all. So how could ToyStory2 work at all? We don’t know. The frames were rendering (other than that dozen shots), so we just carried on, fixed those shots, and charged ahead. At that point, there was nothing more that could be done.
And then, some months later, Pixar rewrote the film from almost the ground up, and we made ToyStory2 again. That rewritten film was the one you saw in theatres and that you can watch now on BluRay’.
