When I record League of Legends(LOL) by Bandicam. When I use "Rectangle on a screen" mode, it will crash the game in a minute. When I use "DirectX/OpenGL window" mode, it will crash in the last moment before the "continue" button appear(I know that if I don't stop the recording and press the continue button will crash the recording), and it will said cannot create dump file.

How can I fix that to record LOL?

I think it is not a problem of my computer hardware, but here is my computer spec:
CPU: i7 4790K
GPU: GTX 770 4G
RAM: 32GB
MB: ROG Z97 Hero

Bandicam setting:
H.264 - NVENC (VBR)
Full Size, 60.00fps, 100q
48.0KNz, stereo, 192kbps

A few more questions/tasks so that others can help you out.

Whats your Operating system?

Are your nvidia drivers up to date? You can download the latest drivers from nvidias site here - http://www.nvidia.com/Download/index.aspx?lang=en-us If you get any errors, do a clean install.

Check and see if bandicam runs fine by disabling unnecessary programs. start-->run Msconfig (win 7). Make sure you hide Microsoft services and disable everything else. If it works, then one of your programs are conflicting with bandicam.

Other methods - Your using h264, I don't use it often but if there is an option for virtual dub (don't know if there is), enable it and see if that works

When I record League of Legends(LOL) by Bandicam. When I use "Rectangle on a screen" mode, it will crash the game in a minute.

It seems that a certain program such as an antivirus program, toolbar program, etc. is blocking Bandicam.
We don't have a good solution for this issue.

Please let us know if you fix the problem.

I am using windows 8.1. The driver is the latest one. I think it is not the problems h264 because it crash in other format too.