I built my own UI that connects directly to the pypilotServer; see link below. In my case I connect through IPC but the same can be done through the 21311 socket. The meat of the code is in handleKey() at line 233:
https://github.com/marcobergman/pypilot_ray
https://github.com/marcobergman/pypilot_ray