You bet! It’s easy. :o)
Click the down-arrow in the upper-right corner to open the menu, then click Show Offline Contacts. That’s it!
To get back to viewing only the contacts who are online, do the same thing. One click shows them, the next click turns them off again. On, off. Hello, Goodbye. Love ya, miss ya. ;o)
Note: Remember that you can send messages to contacts who are offline—they just won’t get them until they sign back in.