The Hermite polynomial, as a special function, is very useful in quantum mechanics and mathematical physics. The one-variable Hermite polynomials are just eigenfunctions of the Harmonic oscillator, while the two-variable Hermite polynomials can represent transition amplitude from number state | n⟩ to | m⟩ in time evolution of a forced Harmonic oscillator.^[1] The Hermite– Gaussian modes exist in optical propagation in quadratic gradient index (GRIN) lenses.^[2] Mathematically, the one-variable Hermite polynomial H_n(x) can be introduced by its generating function

or

while the generating function of two-variable Hermite polynomial H_{n, m}(x, y) is^[3]

or

An interesting question thus challenges us: if the power function x^{n− l}y^{m− l} in Eq.  (4) is replaced by suitable Hermite polynomials, like H_{n− l}(ix)H_{m− l}(iy), then will a new meaningful special function appear? If yes, is it useful in some physics theories? The answer is affirmative, we shall employ the operator-Hermite polynomial method (OHPM)^[4] to tackle this problem. The rest of the paper is organized as follows. In Section  2, we briefly review OHPM. In Section  3, we derive a new special function by using OHPM. In Section  4, we obtain some new operator identities and integration formulas. Their applications in deriving the normalization for some quantum optical states are presented in Section  5. Finally, some conclusions are presented in Section  6.

The essential point of OHPM is to replace H_n(x) by H_n(X), where X is the coordinate operator in quantum mechanics and expressed as

and then operator ordering (normal, anti-normal or Weyl ordering) and representation theory are used to obtain many operator identities. For instance, from Eq.  (1) we have

where : : denotes normal ordering, then we have the useful and concise operator identity

For a two-mode case, when we introduce the second-mode coordinate operator

and noting [a + b^† , b + a^† ] = 0, from Eq.  (3) we have

which indicates

which is another concise operator identity. Hence, the OHPM is powerful in deriving new operator identities.

If we replace x → X, y → Y in Eqs.  (3) and (4), by noting [X, Y] = 0, we see

and

Then noticing

which is a result of the following derivation

and substituting Eq.  (13) into Eq.  (12), we have

On the other hand

thus

where both sides are in normal ordering, which means

where

is the new special function we are searching for, its generating function is exp (sx + ty − st + s²/4 + t²/4).

According to

from Eq.  (19) we have

Similarly, we see

from which it then follows that

According to Eqs. (12), (15), and (19), we obtain a new operator identity

Further, from Eq.  (18) we know

which yields another operator identity

Using the completeness relation of coordinate representation^[5]

we have

which implies a new integration formula

On the other hand, from Eq.  (23) we have

which implies another integration formula

Using Eqs.  (19) and (26), we have the following two-variable Hermite excitation state

which is an entangled state. Further, from Eq.  (18) and [(a + b^† ), (b + a^† ) = 0, we know

Using the Baker– Hausdorff formula

and comparing Eq.  (30) with Eq.  (31), we obtain the following operator identity

Operating 𝔉 _{n, m}(a + b^† , b + a^† ) on the two-mode vacuum state leads to

On the other hand, from Eq.  (18) we have

Using the Baker– Hausdorff formula and Eq.  (1), we see that the left hand side of Eq.  (35) is equal to

so comparing Eq.  (35) with Eq.  (36), we deduce another operator identity

from which it follows that

Using the entangled state representation^[6]

where

obeys the eigenvector equations

Using Eqs.  (38), (40), and (37), we have

from which we deduce the integration formula

As its application, we consider how to normalize the Hermite-polynomial excitation state H_m(ia^† ) | 0⟩ . Using the completeness relation of coherent state representation

and Eq.  (42), we have

then the normalization for the Hermite-polynomial excitation state H_m(ia^† ) | 0⟩ is

In this paper, by virtue of the operator Hermite polynomial method and the technique of integration within ordered product of operators, we find a new kind of special function, which is closely related to one- and two-variable Hermite polynomials. Its application in deriving the normalization for some quantum optical states is presented.