mod is a file format for music.
Simply, it's like midi in that you can edit notes and lengths and pitch and such. But with the realism of using wavs as samples. And many more controls. If done right, you could make a commercial quality song, including singing, etc.
But more commonly it is the generic term to refer to video game music, or "chip tunes".