In 2014, Fabien Lehagre was 29 when he received a letter from his bank that left him “shocked”. First, Lehagre was a French citizen living in France; there was no reason, he thought, for his bank to ask for his US Social Security number—at the time, he didn’t even know he had one.

The bigger shock in the letter, however, came when the bank informed Lehagre that he could be a “potential tax evader” because he had never filed income tax returns with the Internal Revenue Service. Lehagre had been born in the US to an American mother and a French father, but had not lived there since he was 18 months old. He doesn’t speak English (spoke with wealth via ChatGPT translation) and has never worked in the US

Lehagre, now 39, is a so-called “accidental American,” or someone born in the U.S. who hasn’t spent much time there, if at all. These accidental Americans may have no ties to the country, but they’re still on the hook to file and pay US taxes.

Shortly after learning of his unwanted citizenship, Lehagre founded the Association of Accidental Americans, which lobbies to end citizenship-based taxation in favor of residency-based taxation. He would renounce his American citizenship, but after years of stress and campaigning, he refuse to pay the fee to do so. “It’s out of the question to pay $2,350,” says Lehagre.

Every year, thousands more Americans—accidentally or not—decide they’ve had enough. Fed up with what they see as greedy US tax collectors and a meddlesome bureaucracy, they are taking the most drastic steps – tearing up their passports and renouncing their US citizenship.

It’s difficult to quantify exactly how many do so—the IRS publishes a quarterly list of Americans who have expatriated, but it’s not updated, and many of the names published this year actually renounced their citizenship more than a year ago – but more have in recent years. According to a 2020 study from government data, 5,816 Americans renounced their citizenship in the first half of that year, an increase of more than tenfold from the last six months of 2019. The IRS did not immediately respond to a request for updated figures.

Those who move to renounce their citizenship quickly discover that the process is harder – and more expensive – than it seems. It’s often a years-long process that could consume thousands of dollars or more, depending on the individual’s situation. And the former American could also face unintended consequences, lawyers say, such as not being allowed to even visit the US again.

The ultra-rich on the move

Americans renounce their citizenship for many reasons. Some, like Lehagre, have no real ties to the US outside of their parents. Others do it for political reasons. But there is a growing cohort of the ultra-rich who have sought greener pastures abroad because of what they see as an overly onerous American tax code, immigration experts say.

However, renunciation of citizenship is very rare; the ultra-rich are more likely to get second citizenships or residencies in places like Portugal or Malta than to give up their US passports entirely.

“It’s such a big move, and once you do it, you can’t take it back,” says Peter Roskam, leader of the federal policy team at BakerHostetler and a former Republican member of the US Congress. “It’s not like you can give up your citizenship for this year. It’s gone. In a world that’s less and less stable, having that blue passport is a pretty wonderful thing.”

But it does happen, says David Lesperance, a lawyer who helps wealthy Americans give up their U.S. tax status. Lesperance previously said wealth that many of the wealthiest citizens are looking for a “backup plan” due to the perception of growing political and social instability in the US. Some fear the rise of anti-Semitic, anti-LGBTQ and anti-minority sentiments. in some parts of the country.

Others do so in reaction to US “double taxation”. or the fact that the US is one of the only countries that tax citizens regardless of where they live in the world. Although few Americans living abroad actually pay double taxes after credits and exclusions (although some do), the hassle — and high filing fees — are enough to prompt some to give up their citizenship. Others may also try to get ahead other fiscal and patrimonial changes they fear they will be pursued by Democratic presidential administrations.

“Although the tax benefits are not the biggest driver of interest, these benefits do play a role, and the double taxation policy of the United States adds an additional layer of complexity for American citizens,” says Armand Arton, president of Arton Capital, a global company. financial consultancy services company for citizenship. For many Americans living abroad, “the United States will always be home, (but) a policy of double taxation is pushing many to consider giving up their American citizenship in favor of better deals found abroad.”

More likely, though, is the accidental opt-out of Americans like Lehagre, says Max Reed, a Canada-based cross-border tax lawyer.

“It’s not like there’s this wave of people moving out of the U.S. and giving up their citizenship,” Reed says. “The vast, vast majority of them are long-term non-U.S. residents.”

One of the big factors driving these Americans to surrender their passports since 2010 was the implementation of the Foreign Account Tax Compliance Act, or FACTA. This federal law not only requires all foreign financial institutions to report customers with a U.S. connection and their assets to the U.S. Treasury Department, but also requires U.S. citizens and green card holders to report foreign financial assets to the Internal Revenue Service annually Service.

The law was intended to end tax evasion, but many Americans living abroad found the burden of compliance unbearable. It has also led many foreign banks to deny services like mortgages and basic bank accounts to Americans because they don’t want to report the headaches.

How to renounce citizenship

To voluntarily renounce US citizenship, Americans must first secure citizenship in another country. There are a number of ways to do this, depending on the country, including by birthright, family ties or marriage; for the rich there is citizenship through investment programs which make it much easier, assuming they have the funds. Some countries offer other ways, such as buying a house or through work.

Then the soon-to-be American will head to a US embassy to take the oath of renunciation. There, he will also have to pay all outstanding fees, surrender his passport and pay the $2,350 fee (State Department recently announced which will drop to $450). Although it seems simple enough, the whole process can take years.

It gets trickier for wealthy Americans. The US may impose an exit tax if the net worth exceeds $2 million or the average annual income exceeds a certain threshold. The exit tax applies to assets worldwide and is levied to ensure that soon-to-be ex-Americans have met all their tax obligations, including for income and capital gains. Retirement accounts like IRAs are liquidated and taxes are paid on those as well. The former American might want to evade tax responsibility, but the IRS has other plans.

Another pitfall to watch out for, Reed says: A permanent ban on re-entering the U.S. While those who obtain second passports from countries like Canada, France and the U.K. won’t have much trouble visiting the U.S. in the future, those who obtain passports through other means, including potential investment programs in some countries, may have a harder time, he says.

“That’s why it’s not really a strategy for Americans who are just trying to avoid income tax, it doesn’t work that well,” Reed says of giving up citizenship just to save taxes. “What second citizenship are you getting and will you be able to come back?”

This story was originally presented on Fortune.com