During our years of wealth accumulation, a.k.a. working, we pay a pretty penny in taxes. We become accustomed to knowing that after a certain point, we might only see about half of each additional dollar earned.
Adding up federal & state income tax and property taxes, many physicians will have annual tax bills exceeding $100,000.
If you’ve managed to accumulate a sizable nest egg over 20 years or less, you’ve no doubt contributed at least $1 million to the coffers of the taxman. I’ve paid closer to $2 Million in the last 15 years.
That tax burden doesn’t have to last indefinitely, though. Much, much lower tax rates are on the horizon for the aspiring early retiree. Let’s crunch some numbers and examine what you might expect to pay when you make the transition from wage earner to early retiree.
Take the example of someone like Dr. Benson from the Tale of 4 Physicians who was on track to retire with $4 million in about 20 to 25 years with $160,000 in annual spending.
Will he continue to require $160,000 a year in retirement?
Of course not. Before long, he will have paid off his $36,000 a year mortgage, he will no longer be paying off his own student loans or contributing to his childrens’ 529 Plans, and his grown children will be off on their own. With the household of four dropping to two, a number of line items on the annual budget will shrink.
The vast majority of the savings comes from becoming debt-free and no longer having the combined $60,000 in debt service to the mortgage and student loans. We’ll assume any cost savings due to the kids being gone is negated by increased costs for health insurance and more frequent travel. Truth be told, his expenses would likely be lower; the fact that he no longer has a job will lower some expenses like commuting and professional clothing.
To maintain the lifestyle he and his wife have enjoyed, we’ll say that their spending will likely be closer to $100,000. This represents a super-safe 2.5% withdrawal rate.
Bumping that up to $120,000 with some luxury spending, the withdrawal rate would still be a paltry 3%. They can expect to watch their nest egg grow most years in retirement with this level of spending.
Taking a look at how Dr. B arrived here in his early-to-mid fifties, we see that he wisely has his nest egg spread out among different account types. He maxed out his tax-deferred retirement accounts while contributing to personal and spousal backdoor Roth IRAs.
His HSA has grown nicely, he’s got nearly half-a-million in the Roth accounts and more than a quarter of his nest egg is in a taxable brokerage account. Allow me to display this saving and compounding in a handy little spreadsheet.
We’re assuming 4% real (inflation adjusted) returns, so spending power is preserved. For the taxable account, I accounted for a half percent tax drag*, so that account has returned 3.5%.
*This tax drag assumes a portfolio of passive index funds with 2% qualified dividends taxed at 25%. The tax on qualified dividends could be as low as 15% if you have no state tax and keep AGI under $250,000, avoiding the 3.8% medicare surcharge. In that best case scenario, the Bensons could have had a 3.7% real return on their taxable account in the working years. When retired, they will likely pay no federal capital gains taxes.
$100,000 to Spend. Zero Tax.
The Bensons, having paid well over $1 million in federal income tax alone, don’t want to pay that anymore. Like, not at all. Zero. Zilch.
Can we get them $100,000 to spend without incurring federal income tax? Sure, why not?
For the purposes of this exercise, we’ll assume that the couple had higher income when working in 2019 and therefore have not yet received their “stimmies,” or stimulus checks paid out in 2020 and 2021 to stimulate the economy. They will show up as credits when calculating their 2020 tax due.
Plugging some reasonable numbers into TurboTax TaxCaster using 2020 tax rates gives us the following results.
In this example, the Bensons get their spending money from the following sources:
- They receive $500 in interest from the high-yield savings account where they keep an emergency fund of about 6 months’ spending in cash.
- They set up the 457(b) to deliver $3,000 a month, or $36,000 for the year. With a balance of over $730,000, this is roughly 5%, a withdrawal rate that could conceivably make the money last for decades barring a terrible sequence of returns early in retirement.
- They can remove Roth contributions without penalty, but knowing how valuable that money is, they leave it be.
- They sold $43,500 worth of index funds which had more than tripled in value, having a cost basis of only $13,500, creating $30,000 in long-term capital gains.
- Their million dollar taxable account full of index funds distributed $20,000 in qualified dividends, or about 2%, which is typical.
They owe 0 federal income tax. In fact, with only $61,700 in taxable income, they received a refund of $1,227 despite paying no federal income taxes throughout the year.
If the stimulus credit hadn’t been issued, their tax burden would have been a paltry $1,173.
Dr. B could have had substantially higher taxable income and still avoided federal income taxes completely. How much more capital gains could they have taken without owing federal income tax?
$136,500 to Spend. Still no tax.
In this scenario the Bensons sold a whopping $80,000 worth of mutual funds that had roughly tripled. Their cost basis being $33,600 meant a long-term capital gain of $56,400.
They still pay no tax, instead receiving a refund of $12. Why?
If you have a taxable income of $80,000 or less in 2020, your long-term capital gains and qualified dividends have a 0% tax rate. If you get lazy or goof up and end up with $81,000 in taxable income, don’t worry, you haven’t fallen off a cliff.
You won’t owe 15% on all of your capital gains and qualified dividends, you’ll just owe on the portion that exceeds the limit. If you exceed the limit by $100, you owe $15 in capital gains taxes.
Interestingly, in this example, having a taxable income $8,100 over the $80,000 ceiling for the 0% long-term capital gains tax bracket, they used up almost their entire $2,400 stimulus payment. In other words, that $8,100 was taxed at close to 30%.
This is additional dollars of ordinary income (from the 457(b) withdrawals and interest) being taxed in the 12% and 22% brackets while simultaneously pushing capital gains dollars into the 15% bracket. It’s a strangely high marginal tax bracket that can occur as you are bumped out of the 0% capital gains tax bracket and have ordinary income that exceeds the standard deduction.
$160,000 to Spend. Still No tax.
Could we let them spend what they used to spend when they were putting $60,000 of their $160,000 towards student loans and a mortgage?
Here, they are only taking out $24,000 a year from the 457(b) and they sold just over $115,500 of highly appreciated funds from the taxable account, funds that only cost them $39,200 to purchase.
If they sold funds that had been purchased in recent years, they would have far fewer capital gains working against them and could withdraw even more. However, with a starting balance just over $1 Million, they would eventually deplete the taxable account by taking out six figures annually. On the plus side, their Roth account and 401(k), with a combined $1.9 Million or so, would continue to grow.
Without the Stimmies
For this exercise, I entered “2,400” in “stimulus checks received” into Taxcaster to take that out of the final equation in TaxCaster. I’d like to think that doing so will make this calculation valid for 2022 and beyond, but we both know that changes to the tax code could very well be coming soon enough.
We’ll stick with the $24,000 in total 457(b) withdrawals and assume Dr. Benson is selling low-cost-basis funds from taxable. In this example, he sells off $90,500 in funds with a cost basis of $30,200 to generate a total of $135,000 to spend, no stimulus credit required.
This puts him right at the peak of the 0% capital gains tax bracket.
Interestingly, an additional $1,000 of taxable income does not cause that odd double taxation issue that we saw before. This is because the standard deduction more than covers his ordinary income.
Let’s dig into this a bit deeper.
Add $1,000 in long-term capital gains, and he’s taxed %15 on those additional realized capital gains, owing a total of $150 in federal income tax for 2020.
Add $10,000 in capital gains, and it will cost him $1,500.
Keeping income from interest and 401(k), 403(b), or 457(b), Roth conversions, or ordinary income from work below the standard deduction has a real benefit, especially if your taxable income puts you just over the 0% long-term capital gains bracket at $80,000 in 2020 for a married couple filing jointly.
You’ll see what happens if instead of increasing the realized capital gains by $10,000, we increase the 457(b) withdrawals. Here’s that calculation.
We have the same total income and taxable income as above, yet the tax due has increased by nearly $1,000. Rather than a 15% marginal tax increase, it’s closer to 25%.
Kids Offer Additional Benefits
In the first 2 examples, we assumed the kids were long gone. Not in college, not dependents. But what it that weren’t the case? What if they were in college, considered dependents, and the Bensons paid $4,000 out of pocket towards their education?
We learned earlier that it’s best to keep ordinary income on the low side, so we’ll stick with the $2,000 monthly withdrawals from the 457(b), giving them a sub-4% withdrawal rate from this account taking $24,000 a year from an initial $734,000 balance. There’s a good chance that this withdrawal rate will result in their withdrawals lasting their lifetime as long as the employer doesn’t go belly-up.
We’ll do the calculation with and without the $2,400 stimulus tax credit for the grownups and the $500 per dependent that families at this income level received in 2020.
Well, now. The Bensons could have twice their budget in spending money and still get $5 bucks back thanks to the combination of $3,400 in stimulus money they receive in the form of a tax credit and the credit for paying for some of their kids’ schooling.
The American Opportunity Tax Credit (available to married couples with MAGI under $160,000) will match the first $2,000 paid toward tuition with a $2,000 tax credit (that’s free money) and provide an additional $500 credit for the next $2,000.
Let’s re-do that calculation without the stimulus money.
He could still go way over budget with $170,000 in cash to spend in 2020 and get a tiny tax refund of $5.
Let’s change things around and rather than selling so much from the taxable account, he can convert some of that 401(k) money to Roth. This might be a one-step transaction within his 401(k) if the plan allows, or it might require a rollover to an IRA followed by a Roth conversion.
Without the stimulus credit and with no selling from the taxable account, Dr. B doesn’t have nearly enough cash to cover his anticipated expenses of about $100,000. However, he might have cash leftover from prior years when he did things differently. Also, that $32,700 converted to Roth will be available for tax-free and penatly-free withdrawal five years later.
Children at Home
What if the Bensons still had children in junior high when they retired? Say hello to the child tax credit of $3,000 per child. Note that this was recently increased temporarily from $2,000 per child (and is $3,600 per child under 6 years old) for couples with incomes up to $150,000 or individuals with income up to $75,000. It may go back to $2,000 in 2022, but it may not.
To qualify for the $2,000 per year, the children must live at home, be under 17 years of age, and taxable income (MAGI) must remain below $400,000 for the married joint filers. Easy enough.
This possibly temporary increase also made the child tax credit refundable. That is, you could receive money back from the federal government even if you didn’t have tax due to offset it.
In this scenario, with kids at home, the Bensons receive $6,000 in child tax credits for their two teenagers, and they also receive a total of $3,400 in stimulus money that was either paid by check or direct depost in 2020 or will be credited on their 2020 tax return if it wasn’t.
Let’s say they’re getting all the tax credits on their 2020 return and received no stimulus money beforehand.
If we assume the $52,700 from the 401(k) is entirely a Roth conversion, we won’t consider that to be spending money. If, however, that $52,700 is a 401(k) withdrawal, they would have nearly $100,000 to spend and still pay no tax in to the federal government.
The take-home lessons from this exercise are many.
Six Figures to Spend, No Income Tax
You can live quite well without paying federal income tax in an early retirement scenario. We went through many scenarios in which the Bensons had more than enough money to spend without owing a penny in federal income tax.
This is much easier to do as a married couple. Single filers, including widows, will not have it so good, unfortunately.
Tax Diversification Matters
It’s important to diversify your retirement dollars among different account types, some of which have already been taxed (Roth, taxable account).
When you can draw from different account types, you’re better in control of your tax rate. If the vast majority of your retirement dollars are tax-deferred, you have much less autonomy in determining your taxation.
Roth Money is Valuable
Roth contributions can be withdrawn without penalty at any age. Growth (earnings) cannot. Keep track of contributions if you think you might want to access the account prior to age 59.5.
In the examples above, we didn’t withdraw any of it, but it’s a splendid store of reserve funds to tap if you want to increase your spending money without increasing your tax burden.
The 0% Capital Gains Bracket is Awesome
Keeping taxable income low enough to make your long-term capital gains and qualified dividends tax free is powerful.
As exemplified above, low taxable income does not mean having little money to spend in retirement, especially with good tax diversification.
Your Good-for-Nothing Kids are Actually Worth Something
Retiring while your kids are at home or in college will allow you to take advantage of tax credits that are generally not be available to working physicians who are phased out due to high income.
The American Opportunity Tax Credit is one every early retiree should understand; you’d be a fool not to spend $4,000 a year on tuition out of pocket before tapping any 529 plans, assuming your MAGI is $160,000 or lower.
The Child Tax Credit used to be unavailable to most physicians, but the phase-out range was increased greatly by the Tax Cuts and Jobs Act passed in 2017. A MAGI of under $400,000 gives you $2,000 or more per child, with an additional $1,000 to $1,600 per child if income is significantly lower in 2020 and 2021.
Your 401(k) Can Be Accessed in Early Retirement
If you feel you will need to access your 401(k) before age 59.5, there are several ways in which to do so. One is to retire in the year in which you turn 55 (or 56 – 59), even if you’re only 54 and a handful of days, retiring in January with your 55th birthday coming up in December. By law, you will have immediate access.
Before that age, you can roll the 401(k) over to an IRA and set up Substantial Equal Periodic Payments (SEPP) to access the money without penalty. If you plan well and have monies in other accounts, you probably won’t be touching this money until at least 59.5 or perhaps 72 when RMDs become mandatory.
Social Security Will Change Your Tax Picture
Social Security never entered the discussion. Again, if you planned like the Bensons, you’re not relying on it and might delay collecting until age 70 to get the maximum dollar benefit.
If you’re in poor health or have a family history of early demise, you may want to start collecting as soon as you can, at age 62.
Not All Taxes Can be Avoided
If you live in a state with an income tax, expect to pay a few thousand dollars a year in the above scenarios. Tax-Rates.org has a tax calculator that includes state taxes for every state.
If you buy stuff, you’ll pay sales tax on most of that stuff.
Own a home? Expect to pay property taxes. If you’ve got a car, the state will expect some money based on your car ownership annually, as well.
This post focused on federal income tax, which for most people is the biggest tax they pay, but it’s also one of the most easily avoided in retirement.
Note that we didn’t touch on the tax benefits of real estate. That’s a book worth of information, but it’s not necessary to take advantage of any of it in these scenarios.
You will not be paying FICA taxes if you have no earned income as was the case with the Bensons. That includes Social Security and Medicare taxes, which are five figures annually for most working physician.
If you’re self-employed, those taxes are about double what an employed physician pays.
The Really, Really Rich Will Pay
There is a net worth above which avoiding federal income tax is no longer possible. You know, Mo Money Mo Problems.
It depends on how your retirement dollars are distributed among various accounts. If you’ve got “too much,” which I estimate is a nest egg north of $5 million, paying any taxes due should be no problem at all.
There is also an age at which taxation becomes unavoidable. By age 70, you’re collecting Social Security, and at age 72, RMDs will kick in unless you’ve managed to convert most or all tax-deferred dollars into Roth dollars(or spent it all).
You can also donate $100,000 per person as a Qualified Charitable Distribution rather than take the money as an RMD. Doing so won’t make you wealthier, but every dollar donated is a dollar of income that doesn’t appear “above the line” on your 1040 at tax time.
Retirement Tax Planning
If you want to play around with hypotheticals, I encourage you to spend a few minutes with Taxcaster, as I have to come up with the scenarios above. It’s enlightening.
If you want to get more serious with your own complex tax situation, I recommend downloading my favorite tax planning tool that I use to dial in my taxable income, determine my charitable giving, and help ensure I don’t make any costly mistakes at tax time. The bundle does a lot more, but you can see how I use it for tax planning here.
If you’re more inclined to use professional help, there are people who specialize in tax strategies for retirees. View our short list of recommended tax strategists.
Figure this out before you retire, and with proper planning, you too may be positioned to enjoy a nearly tax-free retirement.
What is the take-home message for you? Do these analyses make you more or less likely to consider an early retirement?