Another user with the same error here. Like the OP, I just bought the program after flawless runs with the free trial.
The program just would not lock onto any window, so it only worked in rectangle frame mode. It has been like this since I put my registration code in about a day ago.
As I type this, attempting different windows, enabling the firefox hooking in the options (just switched back from chrome recently), reinstalling/updating java and directX, restarting bandicam, the problem actually fixed itself! I'm not sure if any of the above is really related as it hasn't worked for over a day, but that is my short-term experience thus far.
update: yeah. it is alt-tabbing breaking it for me. I get it back when I restart bandicam usually. Hope to see this bug fixed soon. Program is a little expensive to be breaking whenever I switch screens.