If I may pop in, AVI works with Vegas with no problem, it is just a container, holding whatever codec you set, inside - what codec/settings for recording? I can try help Tweak them with you, if you'd like.~It was a 60 Fps originally, I was just testing some things so please ignore anything in the background.
~Tried, gave the same error.
~I would like to stay as my current codec because it is the one that has the best quality for me (personally). I also need it as mp3 because avi does not work with Sony Vegas and has a lower quality.
~Have yet to try this, will inform you once I do.
The container (AVI/MKV/MP4) does not affect Quality, the codec used inside of the container and the settings only affect Quality.
I would be happy to help you to get it working with Vegas, if you have the time (around this other problem you have). Good luck
Edit:
I see your settings in the Screenshot of Bandicam - with NVENC/CUDA/etc you need to use a Keyframe Interval of "1" or else Vegas can't handle the VFR (Variable Frame Rate) and GOP (Group Of Pictures), and it will either not import, or result in 'glitchy' looking 'trails' in the video when things move (things change between the frames). It will take up more space (and need higher bitrate to keep the same amount of Quality), because it does not have the 'extra room' between frames to compress the little differences between them - but it will work (ie. NVENC codec inside of the AVI container, using a Keyframe Interval of "1").