I asked one of our experts. He said you cannot use both PHY and SerDes on the same port at the same time; therefore, you won't have a conflict.
Below is the exact reply I got from the expert:
One can mix the interfaces, both PHY and SerDes but they can’t be used simultaneously on the same port. For example; ports 0&1 = PHY, Ports 2&3 = SerDes will work but you can’t have Port 0&1 being both PHY and SerDes.
The EEPROM image must also be compatible with the port layout so you should check the available images before starting on a design.
I hope this helps.