The Collaborative International Dictionary of English v.0.48:

Compunctiously \Com*punc"tious*ly\, adv.
   With compunction.
   [1913 Webster]