The Collaborative International Dictionary of English v.0.48:

Portman \Port"man\, n.; pl. Portmen.
   An inhabitant or burgess of a port, esp. of one of the Cinque
   Ports.
   [1913 Webster]