The Jargon File (version 4.4.7, 29 Dec 2003):

ill-behaved
 adj.

    1. [numerical analysis] Said of an algorithm or computational method that
    tends to blow up because of accumulated roundoff error or poor convergence
    properties.

    2. [obs.] Software that bypasses the defined OS interfaces to do things
    (like screen, keyboard, and disk I/O) itself, often in a way that depends
    on the hardware of the machine it is running on or which is nonportable or
    incompatible with other pieces of software. In the MS-DOS world, there was
    a folk theorem (nearly true) to the effect that (owing to gross
    inadequacies and performance penalties in the OS interface) all interesting
    applications were ill-behaved. See also bare metal. Oppose well-behaved
    . See also mess-dos.

    3. In modern usage, a program is called ill-behaved if it uses interfaces
    to the OS or other programs that are private, undocumented, or grossly
    non-portable. Another way to be ill-behaved is to use headers or files that
    are theoretically private to another application.


The Free On-line Dictionary of Computing (30 December 2018):

ill-behaved

   1. [numerical analysis] Said of an algorithm or
   computational method that tends to blow up because of
   accumulated roundoff error or poor convergence properties.

   2. Software that bypasses the defined operating system
   interfaces to do things (like screen, keyboard, and disk I/O)
   itself, often in a way that depends on the hardware of the
   machine it is running on or which is nonportable or
   incompatible with other pieces of software.

   In the IBM PC/mess-dos world, there is a folk theorem
   (nearly true) to the effect that (owing to gross inadequacies
   and performance penalties in the OS interface) all interesting
   applications are ill-behaved.

   See also bare metal. Opposite: well-behaved, compare
   PC-ism.

   [Jargon File]