If they aren't in rut and peeing on themselves, and if you are very clean with your butchering (I've heard that some people bathe their bucks before butchering), then it probably won't be too bad. We butchered a yearling buck because he broke his leg, and there was a little gaminess even though he was neither in rut or even completely mature. He went in curries so it was unnoticeable. I wouldn't eat my current buck unless I was really desperate, that's a horrible stench in breeding season, and even in summer he has a distinct presence. Also, they can, as someone else pointed out, breed very young and if you are not using them for breeding, there's no point to their their testicles. I wouldn't castrate a mature buck, but we band the little guys as early as we can get away with it. I like to leave them on their mothers until late fall when we butcher, and I don't want to take chances with their mothers or sisters getting bred.
Also, on a side note, it's not just the castrating that gives them a high voice. My tall and macho buck with a huge... beard, has the nicest soprano voice you could hope for.
