Actually, The Major difference is the Processor which is defined for Server and Desktop. The Server based Processor is Xeon and that has highest cache like 6Mb and 8MB vice versa. And the Desktop Processor are having 2mb to 3 mb cache. The Motherboards are also different to compaitable with Processors. The Server Motherboards having slot of RAM is different with Desktop like they have ECC RAM, You will not get ECC RAM on Desktop Mother Boards.
Generally speaking, server systems are designed for RAS - Reliability, Availability and Scalability.
Normally on a server system, you could have RAID, redundant fans and power supplies, ECC memory, etc, which builds a robust system for 24x7 operation and minimal down time. You can also have more CPU/memory/PCI slots to expand the capability.