Suppose G is a transitive permutation group of finite degree n≥2, with associated permutation character π=π(G) (which counts fixed points).
Let H be a point stabilizer (so that |G:H|=π(1)=n), and let ρ=ρG and 1G denote the regular and 1-character of G, respectively.
For every positive integer t, we consider the generalized character
ψt=ρ-t(π-1G)
of G.
Since π=IndHG(1H) is part of ρ=Ind1G(1) by transitivity of character induction, for t=1, this ψt=ρ-π+1G is an (ordinary) character of G.
We are going to examine when this happens in general.
This work is motivated by the note [6] of the authors on Frobenius groups (where t=|H|).
Unfortunately, there is a misprint in that note (due to type-setting).
One of the objectives of the present article is to correct this and to make the corresponding result somewhat more precise.
One may ask whether G can be found such that ψt is a character of G for some t>|H|, or when t=n.
Both questions will be answered in the negative.
If G is regular (H=1), then ρ=π and ψt=t1G-(t-1)ρ is a character only when t=1 (as n≥2 and so π≠1G).
Excluding this trivial case, we assume throughout that |H|>1 .
We let Irr(G) denote the set of (complex) irreducible characters of G.
Recall that the cardinality |Irr(G)|=k(G) is the class number of G.
Since G is transitive, the multiplicity (scalar product) 〈1G,π〉=1 (Huppert [3, Satz V.20.2]).
There is an irreducible character ξ≠1G of G which occurs in π, and we let Ξ be the (non-empty) set of these irreducible characters of G.
Of course, Ξ=Ξ(G) is determined by the permutation group G, as is
t*=t*(G)=minξ∈Ξ⌊ξ(1)〈ξ,π〉⌋.
We have 〈ξ,π〉≤〈ξ,ρ〉=ξ(1) for all ξ∈Ξ.
Hence t* is a positive integer.
Note.
By definition, Ξ=Ξ(G) and t*=t*(G) actually depend only on the permutation character π=π(G).
Observe that G is determined by π as a linear group but in general not as a permutation group.
Indeed, Wielandt’s example in [3, V.20.10] shows that it can happen that π=IndKG(1K) for some subgroup K of G which is not G-conjugate to H.
When n=p is a prime, all such possibilities are listed by Feit [1] (in view of the classification theorem; see also [4, XII.10.10] and [4, XII.10.11]).
The case n=p2 has been treated by Guralnick [2].
When G is a Frobenius group to H, π determines G as a permutation group since any complement in G to the Frobenius kernel is G-conjugate to H by the Schur–Zassenhaus theorem (see below).
Lemma.
For ξ∈Ξ, we have 〈ξ,ψt〉=ξ(1)-t〈ξ,π〉, whereas 〈χ,ψt〉=χ(1) whenever χ∈Irr(G)∖Ξ.
Hence ψt is a character of G if and only if t≤t*.
Proof.
Let χ∈Irr(G) be an irreducible character of G.
Since the scalar product is hermitian, and since 〈χ,ρ〉=χ(1) and 〈1G,π〉=1, we have 〈1G,ψt〉=1 and
〈χ,ψt〉=χ(1)-t〈χ,π〉
when χ≠1G.
By definition, 〈χ,π〉=0 if and only if χ∉Ξ and χ≠1G, in which case 〈χ,ψt〉=χ(1).
It follows that ψt is a character of G precisely when t〈ξ,π〉≤ξ(1) for all ξ∈Ξ, that is, when the positive integer t≤t*.
∎
Let us write ψ*=ψt for t=t*.
By the lemma, this ψ*=ψ*(G) is a character of G; we shall describe that explicitly in
two extremal cases.
Theorem 1.
We have t*≤n-1, and t*=n-1 if and only if G is 2-transitive.
In this extremal case (t*=n-1), the following hold:
We have Ξ={ξ} for some unique irreducible character ξ≠1G of G, with ξ(1)=n-1 dividing |H|.
Proof.
The integer t*=t*(G) is as large as possible just when π=1G+ξ for some irreducible character ξ≠1G of G.
Then t*=ξ(1)=n-1 and G is 2-transitive [3, Satz V.20.2]) so that t* is a divisor of |H|.
In general, t*≤n-1, and if t=n-1, then ψt=ρ-(n-1)ξ=∑χ≠ξχ(1)χ is a character of G.
Hence, then t=t* and ψ*=ψt is as claimed.
∎
One can have t*=⌊n-12⌋ when G is a suitable rank 3 permutation group, but it cannot happen that ⌊n-12⌋<t*<n-1.
If n=p is a prime, by a celebrated theorem of Burnside, G is 2-transitive or a (solvable) Frobenius group (for proofs, see [3, Satz V.21.3] and [4, Theorem XII.10.8]).
In this case, t*=p-1 or t*=|H|, with |H| dividing p-1 (see below).
Let D=Der(G)=G∖⋃g∈GHg denote the (normal) subset of derangements in G, the elements acting fixed-point-freely (Hg=g-1Hg).
Thus
D={x∈G∣π(x)=0},
and if π=IndKG(1K) for some subgroup K of G, then ⋃g∈GKg=⋃g∈GHg by definition of induced characters.
It is known and easy to see that |D|≥n-1 (Jordan) and that |D|≥|H| (Cameron–Cohen); see for instance [6, Theorems 1 and 2].
The transitive permutation group G is a Frobenius group (to H≠1) provided |D|=n-1, i.e. if H∩Hg=1 for each g∈G∖H.
The famous theorem of Frobenius [3, Hauptsatz V.7.6] tells us that then D∪{1} is a (normal) subgroup of G (Frobenius kernel).
The Frobenius kernel has order n, whereas the complement H has order dividing n-1 since all nontrivial H-orbits have size |H|.
Theorem 2.
We have t*≤|H|, and t*=|H| if and only if G is a Frobenius group.
In this extremal case (t*=|H|), one has k(G)=k(H)+kG(D), where kG(D) is the number of G-conjugacy classes in D, and the following hold:
|Ξ|=kG(D) and ξ(1)〈ξ,π〉=|H| for each ξ∈Ξ.
ψ*=∑χ∈Irr(G)∖Ξχ(1)χ and Ker(ψ*)=D∪{1} is the Frobenius kernel of G.
Moreover, ResHG(ψ*)=ρH is the regular character of H, and |Irr(G)∖Ξ|=k(H) is equal to the class number of H.
Proof.
Assume that t*>|H|.
Then, by the lemma, there is an integer r≥1 such that ψt is a character of G for t=|H|+r.
Using that |G|=|H|n, we then have
ψt(1)=|H|n-(|H|+r)(n-1)>0.
It follows that |H|+r<|H|⋅nn-1, and this yields the estimate r(n-1)<|H|.
On the other hand, we know from Theorem 1 that |H|+r=t≤n-1.
We obtain that rn=r(n-1)+r<|H|+r≤n-1, which is impossible.
Suppose that t*=|H|.
Recall that H≠1 by assumption.
We know from the lemma that ψ*=ψ*(G) is a character of G (ψ*=ψt for t=t*).
Hence the kernel
F=Ker(ψ*)={x∈G∣ψ*(x)=ψ*(1)}
is a normal subgroup of G.
We have
ψ*(1)=|G|-|H|(n-1)=|H|
(as |G|=|H|n), and ψ*(x)=|H| if and only if x=1 or π(x)=0 (as ρ(x)=0 for x≠1).
Hence F=D∪{1}.
It follows that F∩H=1.
Since
|D|≥n-1=|G:H|-1,
we get G=FH and |D|=n-1.
Thus G is a Frobenius group, with Frobenius kernel F.
From now on, we suppose that G is a Frobenius group to H, and we write F=D∪{1} (an n-set) and ψ=ψt for t=|H|.
In order to show that t*=|H| and ψ=ψ*, we have to verify that ψ is a character of G.
As before, ψ(1)=|H| and ψ(x)=|H| if and only if x∈F.
Since by hypothesis no element ≠1 of G has more than one fixed point, by definition,
ψ(x)={|H|ifx∈F,0otherwise.
In particular, the restriction ResHG(ψ)=ρH is the regular character of H.
We claim that the assignments hH↦hG for h∈H give rise to a bijection from the conjugacy classes of H to those of G which do not meet D.
Indeed, if hG=h0G for nontrivial elements h,h0 in H, then h=h0g for some g∈G, and this forces that g∈H and hH=h0H.
This shows that k(G)=k(H)+kG(D).
For any irreducible character χ of G, by the above,
sχ:=〈χ,ψ〉=1|G|∑x∈Fχ(x)|H|=1n∑x∈Fχ(x).
By the lemma, 〈1G,ψ〉=1.
So let χ≠1G.
It has been shown in the course of the proof for [6, Theorem 3], by elementary means based on the Cauchy–Schwarz inequality, that sχ≠0 implies that χ(x)=χ(1) for all x∈F.
(The misprint mentioned above appears on page 397, line 5, where parentheses are not put correctly.)
Thus sχ≠0 implies that Ker(χ)⊇F and that sχ=χ(1).
Consequently, ψ is a character of G, and F=Ker(ψ) is a normal subgroup of G (Frobenius kernel).
Let again χ≠1G be an irreducible character of G.
We know that from sχ≠0 it follows that sχ=χ(1), and
χ(1)=sχ=〈χ,ρ〉-|H|〈χ,π〉=χ(1)-|H|〈χ,π〉
implies that 〈χ,π〉=0 and so χ∉Ξ.
Conversely, if 〈χ,π〉=0, then sχ=χ(1).
So Ξ consists precisely of the irreducible characters ξ of G satisfying 〈ξ,ψ〉=0, which implies that ξ(1)=|H|〈ξ,π〉 for all ξ∈Ξ.
The number |Irr(G)∖Ξ|=k(G)-|Ξ| of irreducible characters of G occurring in ψ is just the number of irreducible characters having F in its kernel.
Using that G/F≅H, we obtain that k(G)-|Ξ|=|Irr(H)|=k(H), which in turn yields that |Ξ|=kG(D) (see [3, Satz 16.13] for a corresponding result).
The proof is complete.
∎
Remark.
Let G be a Frobenius group (to H), and let F=Der(G)∪{1} (an n-set).
We know that ψ=ρ-|H|(π-1G) is a character of G with Ker(ψ)=F and ResHG(ψ)=ρH.
Note that IndHG(ρH)=ρ=ρG.
Associate to every character θ of H the generalized character
θ^=IndHG(θ)-θ(1)(π-1G)
of G.
This the unique class function of G extending θ and satisfying θ^(x)=θ(1) for all x∈F.
Using that G/F≅H, we see that θ^ is a character of G.
In particular, θ^=ψ for θ=ρH, and if χ is an irreducible character of G occurring in ψ, then Ker(χ)⊇F and ResHG(χ)=θ is irreducible, and we have χ=θ^ and 〈χ,ψ〉=χ(1)=θ(1)=〈θ,ρH〉.
Of course, one can establish the theorem of Frobenius by showing directly that, for any θ∈Irr(H), the class function θ^ is an irreducible character of G, and this is the approach given in the proof for [5, Theorem 7.2].
