Though Buffett, known as "The Oracle of Omaha," has never been destitute, his wealth has changed dramatically in the past five years.