Hi, I have been married for almost 6 years, I received my permanent resident card in 2006. I had an affair, left husband and trying to divorcing now. However husband said he wants my permanent residency taken away because of my adultery, is that possible?