The NVR's output capability via HDMI (up to 4K) will definitely be better than via VGA (up to 1920x1080).
We display 4K HikVision cameras on a 55" LG TV and they look great.
So, I'd guess, there may be an issue with your TV input managing/scaling the signal from your NVR e.g. some TVs have multiple HDMI inputs and if you check the manual they have different performance capabilities (it was common a couple of years ago for only one HDMI port to be to the newer HDMI 2.0 standard.