Marcin Juszkiewicz, the ARM developer at Red Hat responsible for a lot of RHEL/Fedora ARM work, has finally managed to get an X11 Server running on real AArch64 hardware.
Marcin has been working on bringing Linux packages to AArch64 for two years. Like other distributions, the AArch64 bring-up initially started with using ARM's emulator, but these days there's the APM Mustang and AMD Seattle hardware for early developers to play with real 64-bit ARM hardware.
While Marcin Juszkiewicz has had ARM hardware for a while, getting an X.Org Server up and running has taken longer. Initially there wasn't 64-bit ARM PCI Express support, but that's now addressed with the Linux kernel.