For now, you do not need any smartphone for joining party chats of Xbox even when you are away from your console. Apparently, Microsoft has introduced its party chats to the beta version of Xbox for iOS.
Well, if you have been approved into any program then first you need to sign up so that you can be able to connect with your teammates even if you are away from your home and trying to join them shortly in multiplayer matches.
However, we do not have any clue about the official release of party chats for Xbox mobile app. Although, the beta testing has been in progress.
And, it is expected that the official update make takes few months or weeks to arrive in the market. Of course, it is a good news for gamers who want to connect with their teammates every time even when they are away from their home.