Is my child's health visitor allowed access to my medical records? Or even his?

yes she is, a health visitor is there to ensure the non medical health of you and your child so any issue that may have an impact on that she should know especially with relation to post natal depression. She is just doing her job