Should I Pay off My Mortgage?

On trying to take my own advice.

Being debt free used to be a fundamental financial goal. For generations parents, pastors, and even bankers generally advised against borrowing and considered loans a temporary tool at best. Even when structured responsibly, loans were meant to be minimized and repaid promptly.

Fast forward and today we have interest only mortgages, student loans with negative amortization, and financial advisors who actively discourage mortgage repayment.

How did we get here?

After the Great Recession in 2008, the 2 year treasury yield hovered near zero for nearly a decade, barley starting to rise before being hammered down again by the covid crisis. This encouraged spending and borrowing by pushing short term loan and savings rates to historic lows.

Link to source

Mortgage rates spent most of the last half century dropping until the last few years. The sub-5% rates that came to feel normal are historically anything but.

Link to source

At the same time, the stock market has been on an unusually steady and long run since 2009, hitting new highs almost like clockwork.

S&P 500 Performance - Link to Source

Equity growth may feel inevitable if you’re considering recent performance - or the oft-cited “long-term average” of 10%+ which harkens all the way back to 1926. Take a glance at this chart though, and you’ll be reminded of the relatively recent “lost decade” where we enjoyed negative stock returns over a long period.

As the kids say, YMMV.

For understandable reasons we have gotten uncomfortably comfortable with leverage, and somewhere along the way the formerly obvious aim of paying off your house became practically passé.

It may not be a bad time to recall the second part of Buffet’s classic advice: be fearful when others are greedy.

My Financial Journey

One afternoon back in college, I found myself splayed out on the floor of a Barnes & Noble pouring over some new revelations, as one does. I was seized by Robert Kiyosaki’s book Rich Dad, Poor Dad, which was high on inspiration but frustratingly low on directions.

I bought his whole collection, eagerly skimming for the promised “secrets” - which turned out to be available only in a $10,000 real estate seminar the books were hawking. I briefly considered tapping my college fund but headed back to the bookstore instead, where Robert G. Allen schooled me on creating Multiple Streams of Income and becoming a One Minute Millionaire.

Back in those days I was naive enough to believe everything I read, especially if it was published by a bona fide Author. I was also idealistic (and obedient) enough to implement what I learned without much questioning.

Luckily this served me well, at least when it came to money. At 13 I’d discovered The Motley Fool Investment Guide in the back of my grandfather’s car and promptly begged my dad to help me open a “discount brokerage account.” I used their “foolish four” investing formula to select a handful of stocks, buying a few shares each of DIS, MMM, INTC and GE.

I couldn’t wait to see how much faster I’d get rich than the brothers in the compound interest chart examples who started investing at 25 or 30.

My grandfather (from whom I inherited all my fastidious tracking tendencies), took me to meet his financial advisor a couple of years later. I left with a stack of books including Bogle on Mutual Funds and gleaned enough to open a Roth IRA at Vanguard as soon as I turned 18. I contributed the max (then $2000) into a handful of index funds, a newish investment product that nobody at my summer internship at Smith Barney was talking about.

Surprise, I majored in Finance. I did start out as a Philosophy and Psychology double major though (with minors in Business and Spanish), only switching after doubt struck halfway through junior year about how much grad school I’d need to become a lawyer or psychologist.

I had also bought stacks of books on writing, and I was passionate enough about it to occasionally skip class to write op-eds for the campus newspaper. When they offered me a regular column though, I turned it down.

Despite a dream to become a journalist (basically I wanted any white collar job where I could use my overthinking skills and tell people what to do), I did what had already become my habit: I followed the money.

So when I discovered real estate investing just before graduation under those fluorescent lights, surrounded by comforting rows of published wisdom, I didn’t question the strategy. Here was the second leg of my diversified wealth stool. You know, in case the whole stock market thing didn’t work out. I followed instructions and devised a plan.

Lest you doubt - behold the Plan

I fell in love with Excel spreadsheets and financial planning during that Smith Barney internship, which stretched into a part-time job my senior year. I thought I’d discovered the ultimate cheat code when I realized they simply sorted every client into one of 9 slightly differing asset allocation models (and I pretty much had).

I couldn’t believe they had me, at 21, in charge of determining how multi-millionaires should rebalance their holdings. I would never have to pay for my own financial advisor now that I had the models!

My new favorite hobby was plugging numbers into their monte carlo simulator and tinkering with my own spreadsheets during any downtime. I’ll spare you the overwrought, pages-long analysis of my then $12,150 retirement portfolio, but here are the bones of my very first financial plan.

Nerd alert

I guess I’ve always dreamed big! (Though in my defense I did understand how inflation would impact my targets.) But what stands out is that paying off my mortgage - and all the rental mortgages - was a primary stated goal.

Detailed mortgage acceleration plans

Long story short, I followed my plan. I bought a condo and four rental duplexes in my 20s and refinanced them periodically as rates drifted down. I worked as a private banker, primarily originating and managing loans for a living. I was as comfortable with leverage as you can get.

Yet I put 20% down on every purchase (with money left in my college fund), and all the while I paid extra on my home mortgage. This despite the books that inspired me hyping the use of leverage (Nothing Down! Use OPM!) and also despite eventually refinancing my condo into a 15 year fixed loan at 3.25%.

Side note - want to see the budget for my first year as a full-time Working Professional??

I was not this frugal; fun money was supplied by overtime, annual bonus, and holiday monetary gifts.

I quit paying extra on my condo only after renting it out when I got married in 2014. For reasons relating to the bucket mentality (and maybe Dave Ramsey), rental mortgages felt like a different category that didn’t warrant prepayment, despite their higher rates.

Switching my first home to a rental made me realize that distinction was illogical - and once that line item was in a different part of my spreadsheet, I suddenly regretted all those dollars spent paying down my cheapest loan.

After refinancing into a 2.75% 15 year fixed mortgage on our townhome in 2016, my husband and I quit paying extra on that as well. I still have that loan (after a grueling assumption process during our divorce), and I’m not prepaying it now that it’s less than the inflation rate. But in the decade since, my attention has turned toward my most expensive rental mortgage.

The Mortgage in Question

To cut to the chase (after 1250 words - thanks for hanging with me!) I have a rental mortgage with the following terms that I am am debating paying off:

• Loan Amount: Under $250K

• Loan Term: 5 years (matures 2030)

• Payments: Due monthly according to a 30 year amortization that reset in 2025 when I last modified the loan.

• Interest Rate: 5.75% fixed until the end of this year. Rate adjusts annually to the 1 year Treasury rate + 2.25% (it would be 6.15% if it adjusted today).

• Collateral: Original purchase price $205K; current value over $500K.

My lender is easy to work with and has let me amend the loan without modification fees. I switched to a floating rate 6 years ago and did a cash out refi (both for the first time) to snag an initial rate under 3% which was fixed for 3 years.

My rationale was that I could pay it off any time if rates went up. So has the time come?

Here are a few concepts I’m wrestling with to help frame this decision and which might be helpful if you still have a mortgage.

Feelings matter.

Some people hate debt. Others have a religious opposition to it. Certain cultures frown upon it more than others. Personal finance is personal, and nobody is truly rational. It’s OK to follow your feelings on this and not search for the answer in a spreadsheet.

Personally, probably given my experience as a banker, I have no emotional issue with debt. I once paid off a rental property with the proceeds from the sale of another one, and I was surprised by my lack of reaction. I had no sense of pride or accomplishment whatsoever.

In fact the added “problem” of what to do with the extra cash flow (and later, with the higher proceeds from the sale of THAT rental) actually added some stress to my financial life.

TLDR - I don’t have any moral qualms about debt, and I don’t anticipate any emotional response either way. It’s simply a balance sheet decision for me.

Rental performance doesn’t matter.

People tend to conflate the performance of an asset with the financing used to purchase it. In reality, they are two separate things.

This property barely breaks even now that the mortgage interest has more than doubled. But there is a reason cap rates don’t take into account interest expense: the quality of an investment is distinct from your ability to obtain good financing.

Even if I pay off the loan and the cash flow rises, it’s still a relatively low-performing property with a low single digit cap rate. The value has risen much more than rents over time, and I could invest the equity elsewhere for a similar return and less risk - or for a higher return.

TLDR - I’ve already decided to sell (I’m thinking 2027, but I’m not in a hurry if the market is soft). Whether to pay the loan off is a separate decision.

A mortgage is a negative bond.

Your mortgage was likely sold to a government agency like Fannie Mae or Freddie Mac on the secondary market, packaged with thousands of others into mortgage-backed securities, sold to investment companies and dispersed into various bond funds, and then delivered to individuals by pension funds, 401k administrators, or retail brokerage firms.

Long story short, your mortgage payment is literally someone else’s monthly income. And in a winding circle of modern monetary magic, you are probably paying your lender interest with one hand while earning bond dividends with the other each month.

This is generally irrational. At every exchange a layer of fees is imposed, so typically you’re not going to earn as much on a comparable bond or bond fund as you’re paying on your mortgage.

“Typically” - exceptions used to be more rare, but millions of people now have older mortgages that were fixed at historically low rates, whereas today’s bonds and even cash rates are paying more than that. Fine - I’m not prepaying my 2.75% mortgage either.

“Comparable” - When evaluating two fixed income options you should compare the risk, yield, duration, and management. Then there are the tax implications. But remember that bonds can and do lose money (see below), so it’s not as simple as comparing interest rates.

Link to source

By contrast, paying off a mortgage is as risk free as it gets. You know the exact return (your interest rate) and time horizon (the remainder of your loan term). Management risk and hassle factor (the possibility of missing a payment, bank or website issues, filing the tax forms, arguing about their escrow calculations, etc.) go to zero.

Note on stocks - Many people compare their mortgage rate to “what they could earn in the market,” generally referring to the average S&P 500 return of 10.4% since records began in 1926. But unless your portfolio is 100% S&P 500 and your holding period is 100 years, that comparison is irrelevant (and even then, it’s still a bet that your specific returns over the remaining duration of your loan term will equate to that average).

This boils down to a holdings decision. If you look at your whole financial picture instead of just your investment portfolio, selling bonds to pay off a mortgage does not change your total allocation. You’re decreasing fixed income on the left side of the ledger and increasing fixed income (or decreasing negative fixed income) on the right.

TLDR - My mortgage costs 5.75%, and I don’t expect my existing bond funds to outperform that; so rationally I should sell bonds and pay off my loan.

Having a mortgage impacts the retirement calculus.

There are psychological and financial repercussions of hanging on to a mortgage in retirement. A mortgage will generally delay your retirement date because you need more liquidity invested to cover the payment than it would take to just pay it off.

Example: A $100,000 30 year mortgage at 5% has a monthly payment of $537, or $6,442 per year. If you’ve decided on a safe withdrawal rate of 4%, then you need $161,046 invested (25 x $6,442) to cover that expense.

Leverage also increases your sequence of return risk (SORR). If you need more invested in order to service debt expenses, then you’re more vulnerable to market corrections.

Most people don’t run those numbers, but they will intuitively experience added stress and spending compression with a mortgage. Even if you know you can pay it off, monthly cash flow inevitably feels tighter with a mortgage payment and looser without one.

Note on taxes - the tax benefit of holding a mortgage is phantom for most people, but do your own math if you consistently itemize and consider your effective mortgage when making decisions.

TLDR - Now that I’m primarily living off my assets, I’d probably feel freer and also reduce sequence of return risk by tapping investments to pay off the mortgage.

When in doubt, choose simplicity.

Managing financial complexity is one of my favorite ways to fritter away my time and brain power, but there is something to be said for having one less tax form, one less login, one less automatic payment, one less expense.

This is particularly true as we age. Even if you’re not worried about cognitive decline (which you probably should be), how many accounts and forms and financial firms do you want your heirs to have to grapple with?

If I pay off this loan, here’s a non-comprehensive list of things I won’t have to deliberate anymore:

• Whether and how to refinance

• Tracking bond rates, guessing at my tax bracket, and wondering whether the arbitrage is worth it in this particular moment and over the long run.

• 3am musings about whether it’s market timing to think the S&P’s 25%+ average annual return over the last decade makes this a good time to take some chips off the table.

• The imputed loan payoff return on cash compared to my money market rate. E.g, paying off a 5.75% loan gives me a 7% “return” on cash because of the additional amount going to principal.

• Reframing - if the home was paid off would I take a cash out 5.75% floating rate loan against it in order to invest, even if there were no origination fees? No.

• Debating what to sell to pay it off.

  • Since retiring I’ve held 3 years of expenses in liquid cash/bonds; using that to pay off the loan will reduce my safe reserve to 1 year.

  • Selling stocks in brokerage will realize capital gains and push me out of the 0% tax bracket this year.

  • My target asset allocation has always been 50% stocks / 50% fixed income (real estate or bonds), and by that measure I should sell stocks.

  • I could sell taxable bonds and then sell stocks/buy bonds in retirement to maintain my target allocation without realizing capital gains...

I’ve been debating this for years now as the mortgage rate has gradually ticked up. So far keeping leverage has worked in my favor. But reflecting back on my early goals and the progress I’ve been lucky enough to make, I think the best move at this stage is to go ahead pay it off.

I’ll sell bonds in my brokerage account (at a LOSS), then rebalance in my traditional IRA to maintain my asset allocation (ugh - holding bonds in retirement at my age feels very sub-optimal - but at least it’s more tax efficient).

Tune in here for an over-analyis of what to do with any excess proceeds after I eventually sell and replenish my 3 year taxable cash/bond reserve.

TLDR - If the market keeps rising, then my wealth will grow; if it falls I’ll feel smart for taking some chips off the table. Paying the mortgage off is a win-win.

Right??

---

I help people get organized and use wealth to design a life that feels secure and aligned. A former Wall Street banker and CERTIFIED FINANCIAL PLANNER™, I act as an unbiased advocate without selling products or managing investments. To learn more, visit my website.

---

DISCLAIMER: I love writing about the personal, emotional, and practical sides of money, but please remember that my Substack is strictly for educational and coaching purposes. The insights shared here are general in nature and do not constitute specific investment, tax, or legal advice. While I am a CFP® certificant, reading this does not create an official advisory relationship, and any comments or likes should not be interpreted as client testimonials. For personalized investment advice, please consult a registered financial professional.