"maths is hard" says Barbie
The real problem is that teachers who are poor at math are expected to teach it and they pass their prejudice onto children who subsequently do poorly at math.
The surprising part is how many do well IN SPITE of poor teachers - and those tend to be the kind who are on the autistic spectrum who like numbers and focus on them no matter what.
To fix the problem, fix the teaching. There's no inherent reason why mathematicians should predominantly be socially unaware misfits, it's all down to a lack of inspiration at young ages.