You know, I have never had any real difficulty using my TWIC at the airport and I use it for all domestic travel. I have what I thought were confused looks at it at some non-maritime airports, but I have never been questioned about it, nor has it NOT been accepted. And I travel quite a bit each year. . . . Yes, I have read posts where others have had problems with it, but that has not been my personal experience.
As far as traveling internationally on a Z card without a passport, I had that experience back in 1981. I was injured on a ship coming down the English Channel and got of with the Pilot at Brixham. After a couple of days recovery (milked by me, but finally tossed out of town. . . ) I flew back to the States to re-board. Having NO passport at the time (finally got one in 84), I had no problem with immigration in either the UK or the US. And yes, I did rejoin the ship. . . just a bit disappointed because, let’s face it, where would you rather wait for a ship, as the only 22 year old Yank in Torbay, or just another punk in New Jersey. . . . .