Sometimes people don't know what to say in serious situations and are uncomfortable talking about things like illness and death, which is unfortunate because death is a very natural part of life. Your BIL probably did not take an actual offense to your remarks, but I am sure you felt as though you had painted yourself into a corner. Don't beat yourself up about it and just remember next time, rather than trying to find something profound to say, just say "I'm glad your _____ (fill in appropriate name) is doing better" or "I'm very sorry for your loss." Either is perfectly safe yet conveys that you do care about the person's situation.