What is the intuition behind quantization?

Question

It has long irked me that quantum mechanics appears to be so bound up in mysticism, if that isn't too strong a word.

It shouldn't be like that: it should be possible to follow the reasoning all the way back to some intuitively obvious axioms - "intuitively obvious" , at least if you invest enough in understanding the mathematics.

The way I was introduced to quantum mechanics years ago, it went something like: "First, you calculate the Hamiltonian, then mumble mumble and then you have the Hamiltonian Operator". I have never come across a good, intuitive explanation for why we should expect the apparently magical process of translating the classical Hamiltonian to an operator on (a dense subset of) a Hilbert space, $L^2(R^d)$, to work.

Where did the idea come from, that the physical state of a system should be described by a complex-valued "wave-function" and the observables are differential operators that act on wave-functions - and what was the intuitive reasoning behind? What is the mathematical justification - other than "it seems to work"?

The maths don't scare me - I have some understanding of manifolds, bundles, Lie algebras and -groups, I understand that wave-functions are sections of the base manifold into the complex line bundle and that you can, locally, consider them simply to be complex valued functions.

Answer 1

You might enjoy Todorov's 2012 classic. As his Nelson quip summarizes, it is not a functor. Given a classical hamiltonian phase-space function, there are several "prescriptions" or "recipes" that yield the same operator-on-Hilbert-space answer. But, of course, there are counterexamples, which are a source of an almost century-old fascination. There is no "should". By ingenuity, luck, and trial-and-error, superheroes in the 1920s stumbled on it, and were pleased to note nature listened.

The cited article reminds you why, for flat geometry phase spaces the answers agree. Half of your problem is cultural: you are asked to compare and contrast theories defined in phase space and Hilbert space, respectively. There is a wonderful (but computationally demanding/barely-tractable) formulation of quantum mechanics in phase space which eliminates the culture problem and permits comparison and contrast of apples with apples. (But it assumes you are familiar with concepts of the Hilbert space/oranges formulation, really... Take a look at this introduction.)

The engine of this purely formal map from Hilbert space to phase space is the Wigner map, with an inverse given by the Weyl map.

Using the Weyl map, you may go from a classical hamiltonian to a unique quantum one, but, contrary to Weyl's 1927 wild guess, it is not superior, except computationally. By trial and error, people have discovered systems whose hamiltonian is not the Weyl image of a classical hamiltonian, and, in fact, physical systems with different hamiltonians, whose Wigner maps all have the same $\hbar\to 0$ classical hamiltonian limit.

So, as Todorov quips: "Quantization is a mystery".