/Users/(username)/Library/Arduino15
On Windows (you have to type it in the explorer address bar because its a hidden folder):
Arduino IDE 1.6.5r5 and previous:
C:\Users\(username)\AppData\Roaming\Arduino15
C:\Users\(username)\AppData\Local\Arduino15
On Linux:
/home/(username)/.arduino15 (a.k.a. ~/.arduino15)
If you delete this folder then you will need to reinstall any Boards you have installed using Boards Manager after you reinstall the Arduino IDE.