As a test, I went to the twit site and downloaded the mp3. Only 50 seconds using Internet Explorer.
I went back and checked with iTunes and it had timed out and only grabbed 10 megs of the episode.
I thought maybe the podcast description points to a different location than the web site, by checking the podcast xml that shows up in the podcast description in itunes, I see:
<title>TWiT 84: Hahn, I'm Home!</title>
which has the same url that I manually tried.
So I tried googling around and couldn't find a reason why iTunes is downloading so slowly when IE (and Firefox) can grab files quickly from the same URLs.
I'm going to test this out from my OSX version of iTunes tonight and see if Mac iTunes downloads any quicker.