"It makes no difference whether a good man has defrauded a bad man, or a bad man defrauded a good man, or whether a good or bad man has committed adultery: the law can look only to the amount of damage done."