Don't feel bad. I have a 5 year old and I go through the same thing. In the house with him or in the garage without? I usually end up inside with him.
Which makes us both happy.
Unless there is something he can help with like grabbing me tools. Even if they are the wrong ones.