As far as I'm concerned, if the builder has a problem with it, then that is NOT who I'd go with. $40k may be beans to some people, but to most it's a freakin ton of money!! Heck,in my field we might get down money most if the time but until my project is done, I don't get the remainder of the money.
I remember the line in Three Amigos, "No dough..... No show!"