You are running into a compatibility issue that is happening more and more often. There are two manifestations of the compatibility issue,
- Video Card has expectations of the motherboard that cannot be met. For example, requiring PCIe 3.0 support but the board only has PCIe 2.0 support.
- Motherboard's BIOS has expectations of the Video Card's Option-ROM (Op-ROM, which contains the card-specific portion of the Video support for the BIOS) that are not being met. For example, no legacy (i.e. non-UEFI) Op-ROM support.
For the first issue, there is no recourse but to use an older video card (remember, your board is 4 generations and 5 years old out of date; some of the features of today's video cards didn't exist back then). For the second issue, enabling UEFI in the BIOS will sometimes help - but this may mean reinstalling your OS for UEFI boot.
I wish I had a better answer; it is what it is...
Yea, you could look at a 7 Series (B75, H77, Z77) motherboard as an upgrade. Everything else you have should work ok with it. I would add, however, that these motherboards are getting harder and harder to find reasonably priced and you are only putting off the compatibility issues for a short while. You might want to look at replacing both the motherboard and processor...