Ever since Nicholas Clement defined the calorie as a unit of heat back in the 19th century, we've used it as a measure for the energy available to our bodies in the food we eat. The so-called "kilogram calorie" (or kilocalorie), which you see on nutritional labels, equates to the energy needed to raise the temperature of one kilogram of water by one degree Celsius.
These capital-C Calories serve as an approximation for the amount of energy we take in from our food, as well as the amount of energy we expend over the course of our day, through a combination of essential life-processes and any additional physical or mental activity we add on top of the baseline.
Later on in the 19th century, chemist Wilbur Atwater used oxidation reactions to test the energy content of various nutrients, inclusive of corrections for rates of digestion and the production of urea. Atwater's values, roughly 4 calories per gram for protein and carbs and 9 kcals/gram for fats, remain in use today.
Lately, however, there's been a trend towards rejecting this model. Not only are calories thought to be insufficient -- or outright irrelevant -- in explaining the continuing rise in obesity, but the deeper reason is that "a calorie isn't a calorie".
The bomb calorimeter, used by Atwater in deriving his original values, isn't an accurate representation of the body, dissenters say. We don't just "burn" food and extract calories from it in that way.
They're quite right in that sense. While our cells do oxidize nutrients in a process much like burning, there's no real analogy between those chemical processes and the image of burning a piece of bread in an open flame.
More importantly, living organisms do things with the material they ingest, independent of its energy value. Protein, for example, is a metabolically costly entity. Many things our bodies do with protein require energy, such as protein turnover in muscle, and as such it's not always evident that the net energy derived from a gram of protein is identical to the net energy derived from a gram of carbohydrate, even though they have the same value on paper.
There are inefficiencies in digestion and absorption that distort the picture as well. Fiber and cellulose are poorly handled by our GI tract and thus we don't obtain the on-paper Atwater value of energy for those foods.
It would seem that calories aren't calories after all. Right?
Not quite. As a unit of energy available to our bodies, a calorie is always a calorie. The issue here isn't really about calories, but about nutrients and the way living organisms put them to use, including the extraction of energy from them.
In ideal models, calorie balance -- the balance of energy after energy expended is subtracted from energy ingested -- is always the globally permissive variable: if the balance is a net positive, you're gaining weight. If it's not, you aren't. All other factors aside, there must be energy to put in the adipose tissue. If there's no calories to store, then you aren't getting fatter. Your insulin can be spiked every hour of every day and you will still never store air as fat.
The devil's in the details, though, and that's where the dissenters raise a valid point. It's important to realize that nutrients have functions independent of their calorie value. Protein, as mentioned, is a key structural compound, necessary to build new cells and rebuild muscles after exercise. Even without any form of exercise, protein turnover still happens throughout your body. Fats, particularly the essential omega-6 and omega-3 fats, are likewise a requirement for good health. Even carbs, as a source of blood glucose and glycogen stores, have positive effects on neurological and immune-system function among other things. Not all nutrients are created equally, and the ideal values may not resemble the actual calorie values you receive from any food. A 100g dose of protein and a 100g dose of glucose probably don't deliver the same net amount of energy, nor are those nutrients used in the same way.
Pay attention to the choice of words here: nutrients aren't created equally. This is entirely oblique to the matter of energy or calories. What we're talking about here is not whether "calories are calories" but whether or not different nutrients return the same net energy or contribute to essential life processes. That is, given the same amount of on-paper calories, are you receiving the same amount of actual net calories once you factor in digestive and metabolic efficiency?
The amount of food that actually constitutes a positive or negative calorie balance depends entirely on the amount of energy ingested, the amount of energy used up (or lost to heat), and the amount put aside for a future rainy day. All of three of these terms in the balance equation can fluctuate according to what you eat and what you do, as well as presets established by your individual biology.
A calorie is a calorie; it can't be anything else. Calories are the globally-permissive variable. They have to be there to gain, and they have to be missing in order to lose. But a gram of protein and a gram of carbohydrate are indeed two different things, and your body will do different things with them, which in turn affects the net energy extracted from them. So no, you can't entirely ignore the composition of a diet and expect to see identical changes (up or down) solely based on calorie values; but no one was saying that from the outset.
As I've said before, so many of these problems are not real but conceptual, a matter of perspective and how you approach the problem. Here, there's a fundamental divide between what the Calories Aren't Calories side believes, and what's actually happening.
Fanatics who "don't believe in calorie balance" insist that, because of the inherent uncertainty in energy intake -- because you're not a calorimeter -- and in your body's varied usage of calories from nutrients, we must throw out the calorie balance model as invalid.
In science, when you encounter an inconsistency, you don't resolve it by making even more outrageous and unsupportable arguments. "Thermodynamics doesn't apply to diets" transgresses on the truth to such a degree that it's not even wrong. Thermodynamic energy balance does apply; it has to apply.
That said, I'm not aware of anyone on the "calories balance" side that thinks of the human body as a magical incinerator burning up whatever we shovel down our throats, like we're stoking a coal fire in our guts, in the first place. Everyone I know of acknowledges that different nutrients play different roles, and in doing so adjusts both the Energy In and Energy Out sides of the calorie equation. Even a complex system like a living organism must, ultimately, respect the conservation of mass and energy. We can therefore say that, in some way, the energy that goes into a living system must either be stored or else come out of it in some form or another.
The wiggle-room comes in when defining the how and why of the "in some form or another" clause.
Your body, any body, can be described as a form of complex adaptive system. We all have at least a passing familiarity with homeostasis, the mechanisms used by living organisms to maintain stability and order. As I've described elsewhere, this isn't quite right. We aren't like clockwork with orderly gears and springs that fit together into a marvel of Victorian engineering. There's no single switch that flips when insulin is too low or when we eat too much.
We're networks of molecules, and living cells and tissues and organs. They all talk to each other with nerves and hormones and more localized signals, interlocking to such a degree that the relationships between them are more important than the pieces themselves. No individual molecule, or individual cell, matters in your body. It's the constitution of the whole, your body as a thing itself, that is important. Think pattern, not pieces.
Energy balance in a complex adaptive system depends on the state of the entire system. When you overeat, you don't "just" stimulate insulin production. You alter the metabolic activity in muscle and fat and other organs like the liver. Eating signals a profile of events in the brain, which will typically blunt appetite and assert a "healthy" metabolic state. A similar sequence occurs when you fast between meals, with opposite consequences.
These state-changes all affect the way that your body reacts to any influx of energy, and they average out over any 24-hour period. Overeat during that interval, and the net outcome is more fat stored. Under-eat, and fat is lost. Gains and losses represent the net outcome, not the response to any individual meal.
Energy comes in, bounces around, and either stays or leaves according to the rules of the system -- but the rules can change. In a complex system, the energy input and way the energy bounces around and leaves can vary depending on the inputs and the environment around it. Calorie intake and calorie expenditure are moving targets.
Modeling that in detail would be a nightmare. But just as we can summarize the way a mass of jiggling gas molecules behaves as a flow of warm air, we can abstract away the details and look at the high-level behavior. This is where calorie balance becomes relevant. You can obsess over the details all you want, but the meaningful behaviors happen at the high level, in the net balance between energy ingested and energy expended or wasted.
In general, energy intake has to be less than energy usage, but this is only the most abstract sense of the equation.
Your tendency to store fat will, in reality, depend on a variety of things: your activity level and past training, the foods you eat (inclusive of both calorie and nutrient contents), and your biological tendencies to store fat (or not). There are people much more prone to both overeating and storing fat, and thus more vulnerable to becoming obese than the average member of the population.
Some authors have attempted to describe this "fattening" process as a singular physiological mechanism: high intakes of carbohydrate stimulate insulin production and kick off a vicious cycle, by which fat tissue is stimulated to store more fat, which in turn increases carb consumption, further stimulating insulin, and so on.
This is plausible, at least as a thought experiment, but I find it too simplistic given the obvious role of what obesity researchers call the "obesogenic environment" (literally "obesity causing"). Prone as some of us may be to store fat, we still need to put calories in those fat cells. Contrary to popular belief, insulin cannot store what isn't there in the first place.
The widespread availability of cheap, tasty, high-calorie foods coupled with mostly-sedentary lifestyles means that it's depressingly easy to overshoot your daily calorie needs. Even active people and regular exercisers can quickly find themselves on the wrong side of the calorie balance equation.
Consider the "settling point", which obesity researchers define as the on-going balance between expenditure and intake, in which compensation in either value results in the appearance of long-term stability. You diet, drop a few pounds, and your metabolic rate drops accordingly. At the new lower calorie intake, you've now matched your energy expenditure and weight loss stops again. Compensatory effects can make it appear that you're stuck, but in reality you're caught in a trick of accounting.
We've also identified what are likely to be homeostatic systems working to keep you within certain happy ranges of total body mass and bodyfat percentages, and moving outside of these boundaries -- either gaining or losing weight -- isn't always easy. A recent review paper from Westerterp-Plantenga and a whole lot of colleagues presents these arguments in much more detail (see "Set points, settling points and some alternative models: theoretical options to understand how genes and environments combine to regulate body adiposity").
Sugary processed foods certainly don't help matters, and I'd be inclined to agree with the hypothesis that these foods work to increase their own consumption, perhaps driving up that set point (or settling point) in vulnerable individuals. The regulation of food intake is a complex process involving fat cells, the gut, and the brain (among other things), and insulin aside, tasty foods light up the brain in ways that trigger a strong "want" sensation. This in itself might be enough to set off a cascade of ever-increasing adiposity.
Eating tasty sugary or fatty things makes you want to eat more tasty sugary or fatty things. This is no guesswork. Brain-imaging experiments have shown that the brains of the obese respond to food differently from normal-weight individuals. While establishing cause and effect is a delicate matter in this kind of experiment -- the obese might be wired differently from birth, or the different wiring might itself be a consequence of obesity -- there are suggestions from research into, of all things, drug and gambling addictions that shed some light on the matter.
The genetic (or rather, biological) variation in certain critical pieces of the brain is currently a topic of interest among addiction researchers. Some people are born with variants of the dopamine receptors and transporters, with dopamine being the neurotransmitter responsible for feelings of enjoyment, desire, and motivation. If true, it's likely that some people really are born with tendencies to overeat, being "hyper-responders" to the stimulus of nice-tasting treats. Sugary, fatty foods would, to them, create an especially strong desire to consume more, in the same way that problem gamblers or drug addicts demonstrate an extraordinary affinity for their vices.
I used the term "biological" in favor of genetic because as yet it's not entirely certain how much of this is innate, rather than brought out by the environment. As epigenetic research indicates, the interaction between genes and environment is far from clear-cut. Two people born with identical genes but developing in entirely different environments may exhibit equally different traits. Some traits, such as height, are almost entirely determined by genes; others, like certain personality traits, show much less correlation to genetic causes. It may well turn out that children brought up on donuts and french fries and soft drinks wind up with more of these "obesity genes" activated than would have been the case in a more calorie-austere environment, including a neurally-wired desire to eat more.
If there's one thing we can take away from complexity science, it's that the concept of One Single Cause is worthless. Cause and effect just don't work like that in exponential networks. Given the complexity just within our own bodies, let alone the extra tangles that arise when situating that body in a network of social and nutritional influences, I have a hard time seeing the obesity problem as being an exception, a condition with any easily-identified cause.
The physiological explanation, which paints obesity as a problem of dysfunctional fat tissue caused by chronically elevated insulin, may well have a role to play. However I have a hard time accepting it as the mechanism of blame, mainly because there are many people who can eat processed junk foods just fine, without developing either intense cravings or the pattern of "mindless" eating that typically accompanies obesity (or even without getting fatter for that matter).
Not just that, but experience has taught me that merely cutting or eliminating carbs, while an effective strategy in the short term, tends not to bring about magical long-term effects. Eventually a person will have to make an effort to cut calories and add in regular physical activity to maintain progress. Part of this may be down to whether or not the person develops habits and truly changes his or her lifestyle, although if that's the case, that's all the more reason to avoid painting any single food as The Cause -- or the life-saving panacea.
I find it more plausible that the brain's food cravings, whether caused by genetics or long-term changes caused by a lifetime of eating junk food, play a central role, possibly in conjunction with physiological changes caused by "bad" nutrient profiles (that is, inadequate protein and essential fats). Putting brains with a sweet-tooth in a world full of cheap junk food is just asking for it; consequently, while we probably skew the energy balance equation towards fat storage with the foods we choose, we're also much more given to overeating -- relative to our moving-targets of energy expenditure, whatever they may be -- than many are willing to admit.
The networks of interaction between physical and psychological, besides being my hobby-horse at the moment, provides a much more comprehensive explanation than "insulin!" and "carbs bad".
So where does this leave us?
Calories are calories. A unit of energy can't be anything but a unit of energy. How accurate we are at estimating the calorie content of any given food, and what our body does with those calories, is up for debate. Both the energy-in and energy-out variables are dependent on the interplay between our biological tendencies, how the food's nutrient content affects the energy available to our bodies, and our psychological and behavioral relationship with food.
If you want to drop weight, and preferentially drop excess fat in the process, then you need to concentrate on getting essential nutrients and keep an eye on your net calorie balance as the overarching permissive variable. This need not be "counting calories" (although that certainly can work), but an eye towards making sure that you (really) burn up more than you (really) eat over any given 24-hour interval. Part of that is choosing better foods: the lean beef over the chicken fried steak, or the green veggies over the potato chips. Part of that is getting regular activity. Part of that is getting into the habit of eating better, and immersing yourself in a lifestyle that reinforces that mindset.
Calorie-intensive exercise and a coherent strategy to manage your food intake (think "habits" instead of "diets") are the solution, rather than demonizing any food and hoping for the best. The picture is certainly far more complicated than any simplistic models would allow, but this is not equivalent to the rampant belief that "the calorie hypothesis has been debunked".