What’s in a name? – gender classification of names with character based machine learning models

Abstract

Gender information is no longer a mandatory input when registering for an account at many leading Internet companies. However, prediction of demographic information such as gender and age remains an important task, especially in intervention of unintentional gender/age bias in recommender systems. Therefore it is necessary to infer the gender of those users who did not to provide this information during registration. We consider the problem of predicting the gender of registered users based on their declared name. By analyzing the first names of 100M+ users, we found that genders can be very effectively classified using the composition of the name strings. We propose a number of character based machine learning models, and demonstrate that our models are able to infer the gender of users with much higher accuracy than baseline models. Moreover, we show that using the last names in addition to the first names improves classification performance further.

A Appendix

1.1 A.1 Fusion model that combines content information and names for gender prediction

We have seen in Sect. 4 (Table 4) that the content based model is inferior to the name based models. However, it is reasonable to expect that because content consumption by users captures behavior signals, it may offer useful information that can complement the name based models. This is especially true if the user has a unisex first name, for which it is not possible to predict with certainty the gender of the user by first name alone.

We conducted a study to investigate a simple ensemble model. We took a sample of 1M users from Yahoo Full Name Data. For each user, we predicted the male probability using the first name and the NBLR model. For the same user, we predicted the probability based on the content the user consumed, using the CONTENT model.

Fig. 5

Fusion model that combines name and content information to predict gender

We experimented with the following ways of fusing the results of the content and name based models:

Logistic regression: we use $P(\text {M}|\text {first})$ and $P(\text {M}|\text {content})$ as features and declared gender as labels to fit a logistic regression model.

Logistic regression with logits: we first convert $P(\text {M}|\text {first})$ and $P(\text {M}|\text {content})$ to logits, then fit a logistic regression model. Specifically, we convert the probability to logits via the function $p\rightarrow \ln (p/(1-p))$.

Xgboost: we use $P(\text {M}|\text {first})$ and $P(\text {M}|\text {content})$ as features and declared gender as labels to fit an xgboost model.

Table 8 Performance comparison of 3 fusion models, vs the baseline name and content models

We did a 5-fold cross-validation to derive the average results in Table 8. We do not report the standard deviation since it is quite small (under 0.0006). From the table, we see that the fusion model using xgboost can improve AUC by 0.008, accuracy by 0.012, confirming our belief that combining the content and names improves gender prediction. Logistic regression performed slightly worse than xgboost. Logistic regression with logits does not perform as well as logistic regression, although still better than the name based model alone.

1.2 A.2 Dictionary look-up vs model prediction

For users with popular names, it is unnecessary to use a name-based machine learning model to predict their genders. Instead, a simple dictionary lookup is sufficient – it avoids any error that might be introduced in the modeling process. On the other hand, for unpopular names, a model-based method is more suitable: when a name appears only a few times, statistically we have a low confidence in deciding the gender based on the few occurrences. Furthermore, for names that have never appeared before in the data, a machine learning model is the only choice.

Therefore, when deploying in real world applications, we suggest taking a hierarchical approach for the gender classification: if a name appears $\ge k$ times in the population of users with declared gender, we decide the gender based on a dictionary lookup. For other names, we apply the machine learning based models. We will reason on a suitable choice of k next.

In general, we want to pick a name frequency k, above which we can trust the labels in the lookup dictionary with a high confidence. Suppose that a name is 30%/70% male/female in a very large population (e.g., the world). If there are k of our users with that name, drawn from that population randomly, then based on simple calculation using binomial distribution, we need at least $k \ge 16$ to be sure that the name will be labelled as female with $95\%$ or higher probability ³. For a name that is 40%/60% male/female, we need $k\ge 66$. Based on the above considerations, we decide to choose $k = {40}.$ With that name frequency, we know from Table 2 that a dictionary lookup can cover the gender of 86% of users. The gender for remaining users can be inferred by the machine learning models. We emphasize that dictionary lookup is proposed only for real-world applications. For the purpose of understanding our models for names of different frequencies, all results reported in this paper are based on the machine learning models. We have seen in Table 4 that our proposed models performed well for both popular and unpopular names, when compared with the baselines.

1.3 A.3 Understanding the models

In the previous subsection, we have seen that NBLR, LSTM and char-BERT all perform very similarly. Normally, a linear model such as NBLR is easier to understand than nonlinear models. By looking at the weights for the features, one can understand why a sample is predicted as positive or negative.

However, in the case of predicting the gender using the name string, the features are character n-grams extracted from the same name, hence they are not independent. This feature correlation can give rise to negative weights for some features that are associated with male names, which is compensated by larger positive weights for other correlated features (as a reminder, we use label 1 for male and 0 for female). Similarly positive weights may be observed for features that are associated with female names. This makes model interpretation from feature weights difficult. For example, when looking at the female name “_anna_” (we add prefix and suffix “_” to a name, see Sect. 3.2), we found that the 5-gram “anna_” has a positive weight of 0.36, which may be a surprise as one would expect that a name ends with “anna” should be a female name. However, a substring of “anna_” is “nna_”, and it has a negative weight of $-1.19$. When combining the weights of all 17 character grams, together with the intercept, we end up with a weight of $-9.01$ and a male probability of close to 0. Thus, looking at feature weights can be confusing and even misleading.

Instead, it is more instructive to apply one of our models to real datasets, and see how it behaves. We first apply NBLR to classify Chinese names. Note that the Chinese language has up to 100K characters. Of these, about 7000 are commonly used. A person’s given name could consist of any one or two characters. Therefore, the number of unique given names could be in the millions ($7000\times 7000=49\text {M}$). It is difficult to collect enough instances of each of these possible variations and their associated gender. This is where a character-based model is especially suited. As a sanity check, we looked at a list of 11 names from Lu (2018) that are considered especially beautiful, because of their poetic connotation or phonetic harmony. Table 9 shows the male probabilities for each of these names, together with a “note” column for their likely genders and explanation, taken from Lu (2018). We observe that our classifier does assign higher scores for names that are definitely male than those that are female.

Table 9 “Eleven Beautiful Chinese Names” from Lu (2018). P(M) is the probability of male, predicted by NBLR. The “label” column gives the likely gender, and the “note” column gives a description, both based on Lu (2018). Three of the eleven has frequency $\ge 40$ and their genders (in brackets) could have been looked up

Table 10 Words with a high female/male score among: (a) 3,000 most frequent English words that are not in SSA Data; (b) 236,736 words in nltk.corpus.words that are neither in SSA Data nor in Yahoo Data. (c) S&P 500 companies

It is also helpful to understand the NBLR classifier by looking at English words that scored high as male or female. We first take the top (3000 most common words in 3000 most common words in english 2020), filter out 443 words that are found in SSA Data (such as “Rose”), then list the top 11 highest scored male/female words in Table 10 (a). Interestingly, a lot of them make sense, e.g., “miss” or “women” are female words due to their meanings. However, we notice that the majority of these 3,000 words are found in Yahoo Data as names. Next, we take 236,736 words in nltk.corpus.words and filter out any names in either SSA Data or Yahoo Data, and end up with 169,902 words. The highest scoring masculine/feminine words are given in Table 10 (b). Many of the words in column “feminine” of Table 10 (b) do look very feminine. For example, many words end in “ia” (e.g., “anadenia”). Words on the right-hand-side of the table are classified as masculine, likely due to the ending of the words, such as “*boy”. Incidentally “knappan” is a word in Malayalam (a Dravidian language primarily spoken in the south Indian state of Kerala) that means “a good for nothing guy”.

Table 10 (c) gives the most feminine and masculine company names in S&P 500 companies (SP 500 Companies 2020). The results are also interesting. Corporations with feminine names include a number of ladies fashion companies (Tiffany, Ulta, Macy’s), while many with masculine names are in engineering or energy sectors (e.g., Duke Energy, General Motor, General Electric).

Overall, the performance of NBLR on these three datasets illustrates that the model can extend well to rare or even unseen strings, and to non-English languages.