You bet! ‘Tis easy. :o)

Click the Contacts menu.

Click Show Offline Contacts once to de-select the option (and show only online contacts). Click it again to display ALL your contacts, even the ones who are offline.

Remember that you can send messages to contacts who are offline—they just won’t get them until they sign back in.