Disclaimer - I found this solution thanks to my friend Piotr who needed it for different use case, described below.

Recently I had to use multiple repositories on one server with different deploy keys. Solution suggested by github docs was not viable, because I could not alter hostname easilly, as it suggest. It turns out, that actually you can use different keys on per directory basis, when using git. It’s doable thanks to git conditional includes: