1 Corinthians 8:11

Christ gave up his life for that person. Wouldn't you at least be willing to give up going to dinner for him--because, as you say, it doesn't really make any difference? But it does make a difference if you hurt your friend terribly, risking his eternal ruin!