Natural Language Processing with the Text of Wikipedia Articles

Many questions that researchers have about Wikipedia rely on the contents of text within edits or entire articles. Answering these questions requires techniques of natural language processing—software for for analyzing textual data.

Building on our analysis of women politicians in Wikipedia, we might want to look beyond the number of women to consider what kind of information Wikipedia tends to include about them. In analyses of women politicians in the US, researchers have found that US news publishers tend to prioritize coverage of the personal lives of women politicians—their hair, their husbands, and their fashion choices—more than that of men (Bauer & Taylor 2023). Is that true for Wikipedia and for a wider range of countries? To answer this question, we need to examine the actual text of the Wikipedia articles.

In this example, we illustrate how to access and analyze textual data from Wikipedia in Python using the pywikibot library. In a later unit, we show how to do this in R. If you would like to explore more techniques of natural language processing in Python, we recommend the book Natural Language Processing with Python by Steven Bird, Wean Klein, and Edward Loper.

Querying Wikipedia Article Text

We start by getting a single page to see what is available to us:

import pywikibot
from pywikibot import textlib # for parsing sections straight from the wiki pages

site = pywikibot.Site('en', 'wikipedia') # select the wikimedia project we want to interact with
page = pywikibot.Page(site, "Kalonzo Musyoka") # give name of page we want to use

With this page, we can now get the raw wiki text, the HTML and even get the sections:

print("start of wiki text:\n\n{}\n---".format(page.text[:500]))

print("start of html text:\n\n{}\n---".format(page.expand_text()[:500]))

sections = textlib.extract_sections(page.text, site)

for section in sections.sections:
    print(section.title)
start of wiki text:

{{short description|10th Vice President of Kenya}}
{{distinguish|Kennedy Musyoka Kalonzo}}
{{Use British English|date=May 2013}}
{{Use dmy dates|date=November 2022}}
{{Infobox officeholder
| name             = H.E. Dr. Stephen Kalonzo Musyoka
| nickname         =
| office           = 10th [[Vice President of Kenya]]
| order            = 
| honorific_prefix = [[His Excellency]]
| honorific_suffix = [[State Commendations of Kenya|E.G.H.]]
| image            = Kalonzo Musyoka1.jpg
| alt            
---
start of html text:

<div class="shortdescription nomobile noexcerpt noprint searchaux" style="display:none">10th Vice President of Kenya</div>[[Category:articles with short description]][[Category:Short description is different from Wikidata]]
<templatestyles src="Module:Hatnote/styles.css"></templatestyles><div role="note" class="hatnote navigation-not-searchable">Not to be confused with [[:Kennedy Musyoka Kalonzo]].</div>
<nowiki/>[[Category:Use British English from May 2013]][[Category:All Wikipedia articles wri
---
==Early life and education==
==Political career==
===1992–1997 Kenyan Parliament===
===1997–2002 Kenyan Parliament===
===2002–2007 Kenyan Parliament===
===After 2007===
===2013 Elections===
===2017 Elections===
===2022 Elections===
==Other responsibilities==
==Personal life==
==References==
==External links==

Do pages include a section about a person’s personal life?

We can iterate over the different sections present in the raw Wiki text using this method, getting both the headings and the content. Using this, we can explore the question of whether male or female biographies are more likely to include a section about a politician’s personal life. Do do this, we make a simplified query of the one we used in the previous end-to-end example, to get the actual text and parse the section headings:

import pywikibot # loading the pywikibot library
from pywikibot import pagegenerators as pg # import the page generator so we can iterate over all returned Wikidata items
from pywikibot import textlib # for parsing sections straight from the wiki pages

import pandas as pd # loading pandas for data manipulation

import hashlib # for de-identifying the contributors

WIKIPEDIA_SITE = pywikibot.Site("en", 'wikipedia') # creating an English WP site object to get page details

WIKIDATA_SITE = pywikibot.Site("wikidata", 'wikidata')


def get_gender(wikidata_item):
    '''
    Look at the property for gendre and return the
    english language gender label if present. If not,
    return "unknown" - useful when our queries do not
    presume the response
    '''
    if "P21" in wikidata_item.claims.keys():
        return wikidata_item.claims['P21'][0].target.labels['en']
    else:
        return "unknown"

def get_personal_life(sections):
    '''
    iterate over the section headings and return if a personal
    life section is present or not
    '''
    for s in sections.sections:
        if "personal life" in s.title.lower():
            return True
    return False
    
def run_query(QUERY):
    '''
    Run's the handed query and returns a dictionary that contains
    the following information, for politicians that have an article
    in the English language wikipedia
    1. Item ID in wikidata
    2. name of Wikipedia page
    3. gender of politician
    4. number of unique editors
    5. quality rating
    '''
    generator = pg.WikidataSPARQLPageGenerator(QUERY, site=WIKIDATA_SITE)

    politician_data = {}

    for item in generator:
        qid = item.getID() # get unique ID
        # check that has english language article:
        if 'enwiki' in item.sitelinks.keys():
            page_title = item.sitelinks['enwiki'].canonical_title()
            page = pywikibot.Page(WIKIPEDIA_SITE, page_title)

            politician_data[qid] = {} # initialize dictionary 
            politician_data[qid]['gender'] = get_gender(item) # add gender
            politician_data[qid]['name'] = page_title # add english article name
            sections = textlib.extract_sections(page.text, site)
            politician_data[qid]['personal_life_section'] = get_personal_life(sections)
            politician_data[qid]['text_wiki'] = page.text
            politician_data[qid]['text_html'] = page.expand_text()
            politician_data[qid]['text_sections'] = sections
    return politician_data

With this we can now query all politicians again and get the full list:

male_query = """
SELECT DISTINCT ?item
WHERE {
    ?item wdt:P31 wd:Q5;  # Any instance of a human;
          wdt:P106 wd:Q82955; # occupation politician;
          wdt:P27 wd:Q114; # citizen of kenya;
          wdt:P21 wd:Q6581097; # male
}
LIMIT 1200
"""

female_query = """
SELECT DISTINCT ?item
WHERE {
    ?item wdt:P31 wd:Q5;  # Any instance of a human;
          wdt:P106 wd:Q82955; # occupation politician;
          wdt:P27 wd:Q114; # citizen of kenya;
          wdt:P21 wd:Q6581072; # female
}
LIMIT 1200
"""

page_data_male = run_query(male_query)
page_data_female = run_query(female_query)
all_page_data = page_data_male | page_data_female
df_all_page_data = pd.DataFrame(all_page_data).T
df_all_page_data
gender name personal_life_section text_wiki text_html text_sections
Q4694639 male Agustino Neto False {{Short description|Kenyan politician and lawy... <div class="shortdescription nomobile noexcerp... ({{Short description|Kenyan politician and law...
Q16499778 male Joseph Lekuton False {{Short description|Kenyan politician}}\n{{Use... <div class="shortdescription nomobile noexcerp... ({{Short description|Kenyan politician}}\n{{Us...
Q4358037 male Moody Awori True {{Short description|9th Vice President of Keny... <div class="shortdescription nomobile noexcerp... ({{Short description|9th Vice President of Ken...
Q11854485 male Bonaya Godana False {{Short description|Kenyan politician}}\n{{Mor... <div class="shortdescription nomobile noexcerp... ({{Short description|Kenyan politician}}\n{{Mo...
Q4591769 male Koigi wa Wamwere False {{Short description|Kenyan politician and huma... <div class="shortdescription nomobile noexcerp... ({{Short description|Kenyan politician and hum...
... ... ... ... ... ... ...
Q47490046 female Purity Wangui Ngirici False {{short description|Kenyan politician }}\n{{Us... <div class="shortdescription nomobile noexcerp... ({{short description|Kenyan politician }}\n{{U...
Q116955662 female Juma Amriya Boy False {{Short description|Kenyan politician}}\n{{Orp... <div class="shortdescription nomobile noexcerp... ({{Short description|Kenyan politician}}\n{{Or...
Q17411041 female Fatuma Ibrahim Ali False {{Short description|Kenyan politician}}\n{{Use... <div class="shortdescription nomobile noexcerp... ({{Short description|Kenyan politician}}\n{{Us...
Q27980715 female Daisy Nyongesa False [[File:Daisy_Kanainza_Nyongesa.jpg|thumb|Daisy... [[File:Daisy_Kanainza_Nyongesa.jpg|thumb|Daisy... ([[File:Daisy_Kanainza_Nyongesa.jpg|thumb|Dais...
Q4897658 female Beth Wambui Mugo False {{Short description|Kenyan politician}}\n{{Exp... <div class="shortdescription nomobile noexcerp... ({{Short description|Kenyan politician}}\n{{Ex...

657 rows × 6 columns

df_all_page_data[['personal_life_section','gender','name']].groupby(['gender', 'personal_life_section']).count()
name
gender personal_life_section
female False 131
True 33
male False 404
True 89

We see that 33 of the 164 female articles have a Personal Life section (around 20 % of female), while 89 out of 493 male articles have that section (around 18 % of male articles), which comes out roughly the same.

Exploring references

Let’s now do some work to play with references using R and scraping the actual HTML from the web instead. To have a starting point, we save the list of pages we got in this step that we can re-use as a CSV:

df_all_page_data.to_csv('kenyan_politicians.csv')

With this we can move on to the next step!

References