The Taxman Leaveth: Taxes in Early Retirement

During our years of wealth accumulation, a.k.a. working, we pay a pretty penny in taxes.  We become accustomed to knowing that after a certain point, we might only see about half of each additional dollar earned.  Adding up federal & state income tax and property taxes, many physicians will have annual tax bills exceeding $100,000.  If you’ve managed to accumulate a sizable nest egg over 20 years or less, you’ve no doubt contributed at least $1 million to the coffers of the taxman.

Fear not. Much, much lower tax rates are on the horizon for the aspiring early retiree.  Let’s crunch some numbers and examine what you might expect to pay when you hang up the stethoscope for the last time.

 

The Taxman Leaveth

 

Fox & Company CPA

Take the example of someone like Dr. Benson who was on track to retire with $3 million in about 20 years with $120,000 in annual spending.  As you’ll see below, he ended up with $3.3 million.  Will he require $120,000 a year in retirement?  Of course not.  He paid off his $36,000 a year mortgage, he’s no longer contributing to a 529, and his children are off on their own.

To maintain the lifestyle he and his wife have enjoyed, his spending will be closer to $70,000.  Since they expect to travel more, let’s give them $80,000 a year for a comfortable retirement.  This represents a super-safe 2.4% withdrawal rate.  With $100,000 in annual spending, the withdrawal rate would still be a paltry 3%.  They can expect to watch their nest egg grow most years in retirement with this level of spending.

Taking a look at how Dr. B arrived here in his early fifties, we see that he wisely has his nest egg spread out among different account types.  He maxed out his tax-deferred savings, while contributing to  personal and spousal backdoor Roth IRAs.  His HSA has grown nicely, and more than a third of his nest egg is in a taxable account.  Allow me to display this saving and compounding in a handy little spreadsheet.  I do like spreadsheets.

 

4DocsBdiverse

 

We’re assuming 4% real (inflation adjusted) returns, so spending power is preserved.  For the taxable account, I accounted for a half percent tax drag*, so that account has returned 3.5%.

*This tax drag assumes a portfolio of passive index funds with 2% qualified dividends taxed at 25%.  The tax on qualified dividends could be as low as 15% if you have no state tax and keep AGI under $250,000, avoiding the 3.8% medicare surcharge.  In that best case scenario, the Bensons could have had a 3.7% real return on their taxable account in the working years.

 

$80,000 to Spend. Zero Tax.

 

The Bensons, having paid a little over $1 million in federal income tax alone, don’t want to pay that anymore.  Like, not at all.  Zero. Zilch.  Can we get them $80,000 to spend without incurring federal income tax?  Sure.  Why not aim for $100,000?  Plugging some reasonable numbers into Intuit’s TaxCaster using 2015 tax rates gives us the following results.

 

4DocsB100k

 

In this example, the Bensons get their spending money from the following sources:

  • They receive $1000 in interest from the emergency account where they keep ready cash.
  • They set up the 457(b) to deliver $1500 a month, or $18,000 for the year.
  • Since they know that they can remove Roth contributions without penalty, they plan on taking out $11,000 a year for the next 20 years.
  • They sold $48,000 worth of index funds which had doubled in value, creating $24,000 in capital gains.
  • Their $1.1 million dollar taxable fund distributed $22,000 in qualified dividends.

They owe 0 federal income tax.  In fact, with only $44,400 in taxable income, they could have had a much higher taxable income and still paid no income tax.  This could be considered a wasted opportunity.  Nice going, Bensons. Way to go.

How much more capital gains could they have taken without owing federal income tax?

 

$161,000 to Spend. Still no tax.

 

4DocsB161k

 

The Bensons sold a whopping $109,000 worth of mutual funds that had doubled.  Their cost basis being $54,500 meant a long-term capital gain of $54,500.  Still no tax. Why?  If you have a taxable income of $74,900 or less in 2015, your long-term capital gains and qualified dividends have a 0% tax rate.  If you get lazy or goof up and end up with $75,000 in taxable income, don’t worry, you haven’t fallen off a cliff.  You won’t owe 15% on all of your capital gains and qualified dividends, you’ll just owe on the portion that exceeds the limit.  If you exceed the limit by $100, you owe $15 in taxes.

 

Kids Offer Additional Benefits

 

In the first 2 examples, we assumed the kids were long gone.  Not in college, not dependents.  But what it that weren’t the case?  What if they were in college, considered dependents, and the Bensons paid $4000 out of pocket towards their education?

 

4DocsBcollege

 

The American Opportunity Tax Credit (available to married couples with MAGI under $160,000) will match the first $2,000 paid toward tuition with a $2,000 tax credit (that’s free money, folks) and provide an additional $500 credit for the next $2,000.  In this case, the Bensons can take a lot more from the 457(b) or do some Roth conversions from the 401(k), provided it is rolled over to a traditional IRA first.

Rather than increasing the 457(b) withdrawal, they could have maintained it at $18,000 and converted $32,000 of traditional IRA (previously 401(k)) money to a Roth IRA.  What is the advantage of doing this?  Reducing required mandatory distributions (RMDs) which will be enforced at age 70.5, thereby avoiding future taxable income.

What if the Bensons still had children in junior high when they retired?  Say Hello to the child tax credit of $1000 per child.  The children must live at home, be under 17 years of age, and taxable income (MAGI) must remain below $110,000 for the married joint filers.  Easy enough.

 

4DocsB2kids

 

The child tax credit only applies to taxes due, so the Bensons either took more from the 457(b) or did some Roth conversions in December to get their “taxes due” as close to $2000 as possible.  Since you most likely will not have the ability to adjust your 457(b) income on an annual basis, it is probably best to use Roth conversions to keep taxable income flexible during early retirement.

Note that in 2 of these examples, the Bensons had spending money exceeding 4% of their nest egg of $3.3 million (= $132,000).  It might be OK for them to do so in a year with good market returns, particularly if they are planning on using a variable withdrawal strategy.  The point of this exercise is not to show much they can spend each year without depleting their nest egg, it is to show how much money can be made available without paying federal income tax in early retirement.

 

 

 

The take home lessons from this exercise are many:

  • You can live well without paying federal income tax in an early retirement scenario.
  • It’s important to diversify your retirement dollars among different account types, some of which have already been taxed (Roth, taxable account).
  • Roth contributions can be withdrawn without penalty at any age. Growth cannot. Keep track of contributions if you think you might want to access the account prior to age 59.5.
  • Keeping taxable income in the 15% bracket makes your long-term capital gains (on equities purchased at least a year ago) and qualified dividends tax free.
  • Retiring while your kids are at home or in college will allow you to take advantage of tax credits that are generally not be available to working physicians; they are phased out due to high income.
  • We didn’t touch the 401(k) other than for the sake of making Roth conversions. If you feel you will need to access this money before age 59.5, there are a couple ways. One is to retire in the year in which you turn 55 (or 56 – 59). By law, you should have immediate access.  Before that age, you can rollover to an IRA and set up Substantial Equal Periodic Payments (SEPP) to access the money without penalty. If you plan well and have monies in other accounts, you probably won’t be touching this money until at least 59.5 or perhaps 70.5 when RMDs become mandatory.
  • Social Security never entered the discussion. Again, if you planned like the Bensons, you’re not relying on it and might delay collecting until age 70 to get the maximum dollar benefit.
  • If you live in a state with an income tax, expect to pay a few thousand dollars a year in the above scenarios. Tax-Rates.org has a tax calculator that includes state taxes for every state.
  • You will not be paying FICA taxes if you have no earned income as was the case with the Bensons.
  • There is a net worth above which avoiding federal income tax is no longer possible. You know, Mo Money Mo Problems. It depends on how your dollars are distributed. If you’ve got that much, which I estimate is north of $5 million, paying taxes should be no problem at all. There is also an age at which it becomes unavoidable (70 due to RMDs) unless you’ve managed to convert most or all tax deferred dollars into Roth (or spent it all).

 

What is the take-home message for you?  Do these analyses make you more or less likely to consider an early retirement?  How likely is it that the tax code will remain largely intact by the time you will be ready?

Subscribe for Free Calculators & More!

No spam guarantee.

38 comments

  • Nice post.
    It is important to plan on tax strategies during accumulation, based on how you plan to spend/distribute in retirement. I personally am hoping to have little to no tax while converting 401(k) to Roth. Granted there is no guarantee that the tax laws will stay the same, but you have to plan based on current laws.

  • outdoorgirls

    I am really enjoying your blog. Very useful at this point in my life (contemplating ER). In any case what are the tax cutoff in the above scenarios if you are single?

    • Thanks, and good luck in your decision and future! Filing single does change the equation significantly. Whereas the 15% tax bracket goes up to $74,900 for married joint filers, it only goes half as high ($37,450) for single filers. If you want to play around with numbers, plug some in to taxcaster, which is how I came up with the different scenarios. People like to talk about the “marriage penalty” and some couples do indeed see their tax situation worsen with marriage, but for many, marriage will improve their standing when it comes to paying taxes.

  • Ross

    Thanks for the great article. So if a single person is in the 15% tax bracket for anything under $37,650, what happens if they (theoretically) earn $37,650 and have $100 in capital gains? Is their $100 over the $37,650 taxed at 25% or at the capital gains rate? (i.e. does the government benevolently try to take as little as possible by saying that your income above the last bracket was that of capital gains rather than the capital gains being part of your “first $1,000” of the year and the rest of your earnings being your earnings). I’ve always had trouble finding clear answers about this “ordering” stuff on the internet.

    • Excellent question, Ross. I’m no CPA, but my understanding is that the capital gains tax rate of 15% (plus state income tax) would apply to the $100 in capital gains. Taxcaster can be a useful tool to investigate the effect of different income and deductions on your federal income taxes.

  • If your capital gains push your taxable income above $37,450 for single taxpayers or $74,900 for married taxpayers, the overage will be taxed at the 15% rate (or worse).

    Assume you are married, have a long-term capital gain of $50,000 and your adjusted gross income is $275,000. You will pay a capital gain rate of 15% on the $50,000 gain and a 3.8% Medicare surtax on $25,000 of the gain (the amount in excess of $250,000 of adjusted gross income).

  • Anonymous

    So I went to the tax rates.org and put in the following:

    90000 in income plus 3% from an all taxable account of 3300000.

    I kept my deductions as standard.

    My state and federal tax rates was just over 40K. over 21%. This includes state but if you take state our it’s still well over 30K.

    I’m not sure how you came up with zero taxes.

    Can you please shed some light on this. Perhaps I’m missing a big part of this.

    • With $90,000 in earned income, PLUS another $99,000 in dividends, you’re going to owe some taxes. You’ve also got $189,000 to spend.

      In my examples, there was no EARNED income. The spending money comes from a combination of dividends and capital gains, Roth withdrawals, and distributions from a tax-deferred account.

      It’s important to keep TAXABLE income below about $75,000 so that the dividends and capital gains will not be taxed.

      I hope that makes sense. I typically use Taxcaster, but I’ve used tax-rates.org, which is nice for the state tax calculation.

      Best,
      -PoF

      • Anonymous

        So you are saying if I take 70K out of a taxable account it should not be taxed but the earned income will be taxed at regular income taxes.

        So if I took out 90 K from a taxable account, then 15K will be taxed at capital gain. Then if on top of that I make another 90K it will be taxed are regular state and federal taxes.

        Did I understand that correctly?

        • The best way to test different scenarios is to run them through TaxCaster. I have lost faith in tax-rates.org, but will be interested to see their response.

          If you have enough earned income to bump you out of the 15% tax bracket, you can’t capture any capital gains at 0% unfortunately.

          • Anonymous

            The problem is social security. Because I’m a physician I haven’t worked enough years to get full SS at 65. So by working part time I can achieve that. I’ll have to see if 90K per year hits a 15% tax rate.

          • Jeff

            I suggest earning just enough Social Security AIME (Average Indexed Monthly Earnings) to read the second bend point. That’s not hard to do, and extracts the majority of your benefit to effort ratio.

  • Anonymous

    So I tried to put some numbers into the tax calc. tax rates.org. I placed a 3% withdrawl from 3500000. The outcome was still 15% in capital gains. It did not adjust for 74900. It taxed the whole thing. Am I missing something?

    • That’s strange. TaxCaster seems to give a more accurate number. I’ll have to run a side-by-side and figure out what’s going on. I got the same result @ tax-rates, but the result is not consistent with the tax code.

      To be continued…

      • Message sent to Tax-rates.org:

        Plugging in Married Filing Jointly, standard deduction / exemption, with $95,499 capital gains, I come up with $0 in taxes owed, as all of my capital gains fall in the 15% federal income tax bracket, where gains are not taxed.

        Adding ONE dollar gives me a tax owed of $14,325. As in every dollar of gains would be taxed at 15%. My understanding is that only the additional dollar would be taxed at 15%. Am I missing something or is it a software glitch?

        -Physician on FIRE

        • Anonymous

          That’s what I was wondering. If it goes over by one dollar it should tax 15cents just like the marginal vs. effective tax rate.

          Perhaps it is a glitch. I hope so because it makes a huge difference in my taxes and my FI.

  • Steve

    First, great post. Recently discovered your site. All spot on! Not sure if you’ve resolved the $1 over issue, but… The 0% qualified capital gains / dividends rate only applies if you’re in the 15% bracket. If you’re over, even by $1, the rate jumps to 15% on all of it. This can complicate partial roth conversions. It’s a tricky balancing act. You want to convert the maximum amount while still retaining the 0% rate on all the qualified dividend income. Convert too much, even by $1 and opps, your taxes owe jump substantially. The sneaky solution – rough ballpark it, converting more than enough then once you know you’re exact numbers when filing, re-characterize any excess, ensuring you’re at the absolute top of the 15% bracket (not $1 over into the 25% bracket). For example, if you think you can covert about $5K, convert $10K. When filing, you discover you really only had $4K left in the 15% bracket. Re-characterize $6K of the $10K conversion and you’re golden.

    • That’s a great strategy to ensure you can be in that desirable 15% tax bracket. Glad you found the site!

      I’ve read a number of times that the 15% LTCG rate would only apply to the dollars that spilled over into the 25% federal income tax bracket. And I think Taxcaster bears that out when plugging in numbers.

      See answers here, here, and here.

      Best,
      -PoF

  • Sean Grand

    How would you avoid paying taxes on the 457 b distributions?
    Thanks.

    • Just don’t take too much at a time. Tax owed can be offset by exemptions and tax credits. The income will be taxed (or not) the same as 401(k) withdrawals. I recommend Turbotax Taxcaster to play around with different scenarios.

      Of course, the tax code could change dramatically before you or I retire.

      Best,
      -PoF

  • I am semi-retired, just teaching as a clinical professor one day a week. My taxable income for 2016 was $65,000, putting me in the 15% tax bracket. My brother sold his company and I realized a $230,000 long term capital gain on stock I owned in the company. Is there a cap on the paying zero taxes on this long term capital gain or am I allowed to claim this entire amount and pay no taxes on it?

    • I’ve got good news and bad news, Ted.

      The good news is you came into a nice pile of money! But you knew that already.

      The bad news is the capital gains will bump you up a few tax brackets. If you’re married and have some deductions and exemptions, it won’t be the top bracket, but you will owe the 15% capital gains tax, 3.8% ACA surtax, and any state income tax depending on where you live. Since you know much more about your situation than I do, plug your info into TaxCaster to get an idea of the federal income tax owed. Plugging in a few rough numbers gives me just under $60,000 in taxes owed (not including state tax).

      Best,
      -PoF

      Best,
      -PoF

  • ken

    I have about 4.4 mil in Ira and 800k in munis
    No issue with me paying about 20% federal and zero state(fla)

    Any way to rduce the tax bite other than using taxable account of 800k

    • If you’re not yet taking RMD’s, you could consider doing some Roth conversions now to fill up the lower tax brackets, lowering the size of your annual RMDs eventually.

      If you’ve already turned 70, there’s not much to do but pay the tax, enjoy the money from the big RMD, and bask in the Florida sunshine. Go Gators!

      Best,
      -PoF

  • Pingback: The Impact of Lifestyle Deflation - Stocks Masters

  • Pingback: Why This Index Fund Investor Purchased an Individual Stock - Physician on FIRE

  • Pingback: Selling Shares Beats Collecting Dividends - Physician on FIRE

  • Pingback: Our Drawdown Plan in Early Retirement - Physician on FIRE

  • Pingback: 2017 Q3 PoF Portfolio, Spending, and Blog Performance Update - Physician on FIRE

Leave a Reply

Your email address will not be published. Required fields are marked *