Quoting Singlemalt:
I've got one that is noisy that I only notice when slowing down. It quiets back down when ever it has some load on it.
If you're running an unsprung clutch, this may be normal, especially with solid engine mounts. If you've neither, then something is probably about to sh*t the bed.