The Jargon File (version 4.4.7, 29 Dec 2003):
out-of-band
 adj.
    [from telecommunications and network theory]
    1. In software, describes values of a function which are not in its
    ?natural? range of return values, but are rather signals that some kind of
    exception has occurred. Many C functions, for example, return a nonnegative
    integral value, but indicate failure with an out-of-band return value of
    ?1. Compare hidden flag, green bytes, fence.
    2. Also sometimes used to describe what communications people call shift
    characters, such as the ESC that leads control sequences for many
    terminals, or the level shift indicators in the old 5-bit Baudot codes.
    3. In personal communication, using methods other than email, such as
    telephones or snail-mail.
The Free On-line Dictionary of Computing (30 December 2018):
out-of-band
   1.  The exchange of call control information
   on a dedicated channel, separate from that used by the
   telephone call or data transmission.
   2. Sometimes used to describe what communications people call
   "shift characters", such as the ESC that leads control
   sequences for many terminals, or the level shift indicators in
   the old 5-bit Baudot codes.
   3. In personal communication, using methods other than
   electronic mail, such as telephone or snail-mail.
   4.  Values returned by a function that are not in
   its "natural" range of return values, but rather signal some
   kind of exception.  Many C functions that normally return
   a non-negative integer return -1 to indicate failure.
   This use confuses "out-of-band" with "out-of-range".  It is
   actually a clear example of in-band signalling since it uses
   the same "channel" for control and data.
   Compare hidden flag, green bytes, fence.
   [Jargon File]
   (2001-04-08)