I think it would not be vary simple... there are some threads about this issue, and maybe this one will help you
(I've not checked anything)
Just curious: why do you need X/what do you plan to do?
This message was posted on behalf of Intel Corporation
I have never tried to install xserver on my Galileo, however, as Fernando mentioned, this might not ba a very simple thing to achieve. However, I must point out that Galileo does not have a dedicated video interface therefore you will have to install the drivers of what you are going to use as output device (i.e. your monitor).
That being said, you can find the source of xserver in https://www.x.org/releases/X11R7.7/src/xserver/ and its documentation in https://www.x.org/releases/X11R7.7/doc/index.html.
When you try to build xserver remember to install its dependencies, otherwise it will fail during the building process, they are:
OpenSSL-1.0.2h or Nettle-3.2 or libgcrypt-1.7.0
This is a very interesting project and I would like to see how it develops. I invite you to try to install xserver on Galileo and share your results on the community, they might be of much help for other users.