Stepchild visitation question.?

It's my boyfriend's son,so he's not technically my stepson.Anyway,he's 11 and lives with his Mom and b/f gets him every other weekend during the school year, and for 5 (tered) weeks during the summer. At the beginning of the summer it was fine, we would hang out together some of the time (with my 3 kids) and do things separately some of the time so they could have alone time together. The last time he had him for a week and now this week too,are different. He just wants to hunker down with his son and not visit with us very much at all. I get the feeling it's because he knows this is the last of his "extended visits" until Christmas. While I understand that, it is hurtful to me,and my kids keep asking why we aren't hanging out with them. We kind of bickered about it this evening and then later I texted him,"I love you" (as a peace offering) and he didn't write back. Later he said,"I don't like to text every 5 minutes when I'm with him."(not I love you too) Am I being too sensitive?