If the E2EE is enabled and the client software source is available and reproducible, then, indeed, it could be called Telegram or anything else, it doesn’t matter.
The particular issue with Telegram is, as you say, the default setting. And also that its encryption algo is not universally trusted.
The message is encrypted using a key. The key exchange was done over a direct secure channel to the other client, in much the same way as you connect to your bank’s website using HTTPS. The server therefore does not have the key and can only see encrypted text.
Assuming the client software has not been compromised at either end, then the server will never see anything other than garbled ciphertext.
BTW, this is also the case with Whatsapp, for example. But the problem with Whatsapp is that the client software is closed source. So you have to trust them not to, for example, surreptitiously phone home with a separate copy of your message. Very unlikely but you have no way to check when the client software is a black box.
But what’s running on the server is not the issue in either case.