From what it seems, I suspect that your application is not really consuming all the cores of the processor. Just check how many maximum cores your application can actually support cause most of the times, an application that does not support dual or quad cores is unable to benefit from all the available cores of the processor. In your case here, you are using a quad-core Xeon meanwhile a lot of applications still are not programmed for quad-core processors which is why, you are not gaining maximum core performance with all cores enabled.
You however do have the option of turning off individual cores from the BIOS limiting your Xeon 5500 to work on lesser number of cores, even two or one depending on the number of cores your application supports.
Hope this answers your question, however for any further information, feel free to contact us and we'll be glad to help.
Thank you for using Intel, have a great day!
Intel Go Green, Save The Environment!