The bumper doesn't need to be removed, but yes it's still a real PITA to remove the guards/fenders. It's a good 20 minute job each side especially if your splash guards are in good shape and you don't want to drop the bumper.
It's the only way to do it though mate as the hinges are under the guards and there really is no other access! Apart from which, I would WANT to get in there and see what the problem is. From the look of those photos I'd hazard a guess that all the bolts on at least one hinge have either backed out or rusted through/sheared off. You can adjust door height and position by loosening the hinge bolts usually but you shouldn't get that much movement. I've certainly never seen anything like that before!
Edit: If I remember correctly, one of the bolts that holds on the guards/fenders is behind the indicators on each side, so you will have to remove those. I already had my headlights out to do a swap when we did this (which made it much easier) but I can't remember if you can get to those bolts without removing the grill/headlights!
