No Google Manual Penalty, but lost rankings. New domain move? Any advice appreciated

First off – Thank You for your time and trouble.
I did try the search function, and did find some info, but not exact. And I will try and keep this as far from a review as I can.
I have a site that had no manual google penalty, yet I did have some bad backlinks from a 2011 bad seo strategy. So I am assuming based on my dramatic fall from the google search, that it was all algorithm or some sort of bad link based issue.
I know everyone has a Dr. Phil like story, but my website is my families…
No Google Manual Penalty, but lost rankings. New domain move? Any advice appreciated

See original post by colochris