Americans are frustrated with more than just Washington politicians. A recent ABC News poll found that a majority of Americans are “angry” at banks for their role in the recession and more than three-quarters feel banks should do more to “make amends.” Undoubtedly, many who blame big banks for the financial crisis like the idea of those institutions-particularly bailout recipients-paying higher taxes. Yet tempting as it may sound, Americans should be warned that collecting money from “big banks” is more difficult than one might expect. Taxes levied on business tend to find their way to costumers, workers, and anyone with a 401k. In other words, you’ll be the one paying taxes that are supposed to hit Wall Street.

In March, the Congressional Budget Office (CBO) provided a helpful analysis of this dynamic when evaluating a proposed tax on banks with assets in excess of $50 billion. The tax was supposed to bring in $90 billion over the next decade and make good on President Obama’s promise that “We want our money back and we are going to get it.”

Yet the CBO revealed that big banks alone wouldn’t bear the burden of the tax. The CBO wrote, “The ultimate cost of a tax or fee is not necessarily borne by the entity that writes the check to the government. The cost of the proposed fee would ultimately be borne to varying degrees by an institution’s customers, employees, and investors…”

Customers would pay the tax through higher borrowing rates and other fees. Bank employees might see their compensation fall. And the CBO warns “the fee would probably lower the total supply of credit in the financial system to a slight degree. It would also probably slightly decrease the availability of credit for small businesses.” Less credit for small businesses will make it harder for those businesses to grow and expand, and-most importantly given our near double digit unemployment rate-to hire workers.

The CBO report reminds us that anytime government levies a tax, it takes money out of the private sector. Taxes on corporations never just hit some boogeyman called “big business.” Businesses are ultimately people: customers, employees, and investors. We end up paying those business taxes, even when they are officially directed at unpopular entities from Wall Street to oil companies.

The American people should keep that in mind as Congress considers new ways to raise revenue. A Value Added Tax (VAT), for example, has been mentioned as a possible way to boost federal receipts and lower the deficit. The VAT, which is common in Europe, taxes goods and services at every stage of production. While VAT proponents make it sound as though “businesses” pay the fees, ultimately the costs are passed on to consumers. Consumers will pay higher prices which means they will be able to afford less. Businesses will face higher costs meaning they’ll have less for hiring workers, expanding production, or investing in new research.

Even taxes on “millionaires”-a policy being considered in many states and localities facing budget gaps-tend to miss the mark. Certainly some millionaires will pay higher taxes. But that means they’ll have less to spend on goods and services and less to invest, which hurts businesses, their employees, and ultimately all of us. And millionaires can also relocate. When taxes rise in one city or state, many of these highly mobile individuals will leave-taking their money and consumer power-in favor of a lower-tax jurisdiction.

Economists can make it sound complicated, but the principle is really simple: when government takes more, the private sector has less. Politicians may try to target taxes on politically unpopular individuals and entities, but inevitably the effects ripple through the economy sparing no one.

Raising taxes isn’t the answer to our deficit crisis. Government needs to cut spending. Historically, our government has consumed about 20 percent of GDP. Today, it consumes about 24 percent. This isn’t just because of the economic crisis, the CBO estimates that government will still account for 24 percent of GDP in 2020, long after this crisis presumably has passed. (For more on the government’s out of control spending, see this video by Bankrupting America).

It’s understandable that Americans are angry with the banks, but raising taxes on them won’t undo the misguided bailouts and could backfire on all of us. Citizens shouldn’t be fooled by the populist rhetoric that politicians can make the other guys pay while sparing us. We all pay for too-big government, which is why reducing government spending–not increasing taxes–must be our priority.