upon further review and really studying the problem, the rear taken down pin hole in the lower itself is the problem, those holes don't line up, I did not drill the take down holes, those came from factory drilled at an angle.
I got it to fit with some finagling and doings, the upper and lower fit snuggly, not wiggle at all, there is a small gap toward the rear where the upper and lower mate, you can actually see light from the other side through the gap. This is an issue I have read about with this lower as well.
Seems like I did the research, I knew there were issues with the lower and I still used it.... it was cheap... get what you pay for.
Now in the future if I find a nice lower I can swap it out, I could even swap out my lower parts kit, so that might be something I look at doing maybe next year, but this was a really good learning experience for me.
I can't say enough about the upper I got from farmer rick, compare next to a buddies colt, its every bit as nice.