Did you use converters to go from line level to rca to connect the amp? If so, most of those have a gain control on them as well. You may need to adjust. That's what happened to me. The only problem was I had to turn the gain on the converters all the way down to get rid of the noise which meant I had to crank the volume way up and then you could still hear the white noise a little. So, I fixed the problem by going with an aftermarket HU!! Solved the problem!!
But in answer to your question, no you should not have white noise. I would have given most of the same advice as bluedog. Good luck!!