If your name is on the lease he cannot evict you. If he does not let you in when you go then call the police.
You can discuss with the apartment management what it would take to remove your name from the lease, but whatever you do get anything from thim in writing so they can't tell you one...