Should I Just Let It Go, Or Still Be Upset About It?

Just let him go. Couples need their space and don't take offense to him not wanting you there it's not that. Talk to him and let him know you want him to have a good time and you love him, and suggest you 2 taking a little vaca after that alone :)