The Collaborative International Dictionary of English v.0.48:

Riffraff \Riff"raff`\, n. [OE. rif and raf every particle, OF.
   rif et raf. CF. Raff, and 1st Rifle.]
   Sweepings; refuse; the lowest order of society. --Beau. & Fl.
   [1913 Webster]

WordNet (r) 3.0 (2006):

riffraff
    n 1: disparaging terms for the common people [syn: rabble,
         riffraff, ragtag, ragtag and bobtail]