It all depends. If your board isn't over the property line then there is nothing that the other property owner can do, despite the board being flush against his fence. If you really want to know then you should spend money on a surveyor who will be able to tell you exactly where the property ends and whether the board is over the line (and you can determine if it crosses the line in the "air" as well where it touches the neighbor's fence.)