Took down the entire transmission again, this makes it, I dunno, the 6th time? Lost count. Stuck the gear carrier in the jig, this was when everything became crystal clear.
The downshift grind problem was because the reverse had a whole lot of slop, and with me jiggling the reverse linkages, I can get reverse gear to hit the 1st-2nd slider while in 2nd gear.
I removed reverse, tightened the I bolt a little to try and reduce play. Adjusted 1st and 2nd slider (this was pretty close), adjusted 3rd and 4th slider (this was quite far off, I had set the slider to engage 4th too deep, and as a consequence wouldn't engage 3rd fully). Stuck reverse back in, adjusted that to engage fully with the 1st-2nd slider while in reverse, and not hit 2nd even with me jiggling the linkages on reverse while in 2nd (sounds really messy, but once you're in there you'll figure it out, I know I did).
The reverse gear system in the stock VW transaxle is really weak (at least it seems so to me), just not a very good design. I would like to thank Eric (http://www.shoptalkforums.com/profile.php?mode=viewprofile&u=5765) for responding to his email when I asked about the reverse gear adjustment, didn't know what to do at that time. His website at http://www.geocities.com/Baja/Dunes/9702/transtalk.html was also pretty helpful, they have some pictures too.
I spent about an hour jigging everything up. It was already past 2AM by the time I had the entire transmission back into the car. Muscle fatigue was also already setting in, so I left the engine on the stand and went home, best to have a helper, especially during the trying-to-balance-transmission-on-jackstand part.