While Wikipedia provides access to an API that simplifies researcher access to structured data, many online services do not offer such a easy-to-use information source. In those cases, you may need to scrape that information from the open web. In 2026 as we write this, the open web is declining in availability as news publishers and content delivery networks restrict access to evade AI bots. Yet scraping still remains a valuable tool for research.
In this example, we use Wikipedia as an example for how to scrape information from the open web. In practice, we encourage you to use the Wikipedia APIs when querying information from Wikipedia.
To continue our theme of studying how women and men politicians are depicted on Wikipedia, we now turn to the question of how Wikipedia relies on sources from other publishers in questions of representation. Based on the Global Media Monitoring Project, and research by an international group of scholars, Kenyan news publishers more often use words focused on leadership when describing male political leaders compared to female leaders and coverage toward women tends to be more negative.
In this analysis, we use the references section to investigate the headlines of articles cited in Wikipedia articles. This will require us to scrape those articles and then apply a number of NLP techniques to analyze the contents of those headlines.
Setting up software packages
We start by installing and loading the necessary libraries. We will use the rvest, which is part of the tidyverse set of libraries which you have likely already installed as part of the other SICSS instructions. If not, you can install just rvest using install.packages("rvest") before loading it. For other steps you’ll also need the wordcloud and tidytext libraries, which you can install as below:
── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
✔ dplyr 1.2.1 ✔ readr 2.2.0
✔ forcats 1.0.1 ✔ stringr 1.6.0
✔ ggplot2 4.0.3 ✔ tibble 3.3.1
✔ lubridate 1.9.5 ✔ tidyr 1.3.2
✔ purrr 1.2.2
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag() masks stats::lag()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors
library(rvest)
Attaching package: 'rvest'
The following object is masked from 'package:readr':
guess_encoding
In the last instructions, we created a CSV file (kenyan_politicians.csv), which has the names of the Wikipedia pages we want to scrape. We start by loading this file:
With the html now parsed, we can try to traverse the nodes of the file, to find the references of the page. If we look at an example of a Wiki page, we can see that those will be in a <section> tag, that will be labelled by aria-labelledby="References". Let’s try to see if we can get that section of the page:
ref_section <- html %>%html_elements(xpath="//section[contains(@aria-labelledby, 'References')]")ref_section
[1] "Profile of Moody Arthur Awori"
[2] "Page on Awori at Vice-President web site Archived 2007-09-27 at the Wayback Machine."
[3] "\"Awori Hands Over to Kalonzo\", The East African Standard, 10 January 1998."
[4] "Kenya Government Bio Archived 2005-07-28 at the Wayback Machine."
[5] ".mw-parser-output cite.citation{font-style:inherit;word-wrap:break-word}.mw-parser-output .citation q{quotes:\"\\\"\"\"\\\"\"\"'\"\"'\"}.mw-parser-output .citation:target{background-color:rgba(0,127,255,0.133)}.mw-parser-output .id-lock-free.id-lock-free a{background:url(\"//upload.wikimedia.org/wikipedia/commons/6/65/Lock-green.svg\")right 0.1em center/9px no-repeat}.mw-parser-output .id-lock-limited.id-lock-limited a,.mw-parser-output .id-lock-registration.id-lock-registration a{background:url(\"//upload.wikimedia.org/wikipedia/commons/d/d6/Lock-gray-alt-2.svg\")right 0.1em center/9px no-repeat}.mw-parser-output .id-lock-subscription.id-lock-subscription a{background:url(\"//upload.wikimedia.org/wikipedia/commons/a/aa/Lock-red-alt-2.svg\")right 0.1em center/9px no-repeat}.mw-parser-output .cs1-ws-icon a{background:url(\"//upload.wikimedia.org/wikipedia/commons/4/4c/Wikisource-logo.svg\")right 0.1em center/12px no-repeat}body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-free a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-limited a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-registration a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-subscription a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .cs1-ws-icon a{background-size:contain;padding:0 1em 0 0}.mw-parser-output .cs1-code{color:inherit;background:inherit;border:none;padding:inherit}.mw-parser-output .cs1-hidden-error{display:none;color:var(--color-error,#bf3c2c)}.mw-parser-output .cs1-visible-error{color:var(--color-error,#bf3c2c)}.mw-parser-output .cs1-maint{display:none;color:#085;margin-left:0.3em}.mw-parser-output .cs1-kern-left{padding-left:0.2em}.mw-parser-output .cs1-kern-right{padding-right:0.2em}.mw-parser-output .citation .mw-selflink{font-weight:inherit}@media screen{.mw-parser-output .cs1-format{font-size:95%}html.skin-theme-clientpref-night .mw-parser-output .cs1-maint{color:#18911f}}@media screen and (prefers-color-scheme:dark){html.skin-theme-clientpref-os .mw-parser-output .cs1-maint{color:#18911f}}reporter, Nairobian. \"The Immortals:Why the Aworis are the Kennedys of Kenya\". Standard Entertainment and Lifestyle. Retrieved 23 May 2021."
[6] "\"\"Kibaki's new cabinet\"\". Archived from the original on 16 January 2006. Retrieved 20 February 2006., The Standard (Kenya)."
[7] "\"Inside one famous family of scholars and top leaders\". Nation. 5 July 2020. Retrieved 28 July 2023."
[8] "\"Kenya Women Billionaires: The Story of Mary Okello - Kenyans.co.ke\". www.kenyans.co.ke. 5 October 2021. Retrieved 28 July 2023."
[9] "\"Prof. NO Awori, May 1986 - Nov 1986\". University of Nairobi - Department of Surgery."
[10] "Amalemba, Robert. \"Former VP Awori wins back late kin's 30-acre land from trespasser\". The Standard. Retrieved 21 February 2026."
[11] "\"Old boys\". Kakamegahighschool.com. Archived from the original on 9 October 2011. Retrieved 4 August 2011."
[12] "Kenya Parliament profile Archived 2006-04-14 at the Wayback Machine."
[13] "\"Reviews under way in Kenya vote\", BBC News, 30 December 2007."
[14] "\"Kalonzo VP in Kibaki’s new Cabinet\", The Standard, 9 January 2008."
[15] "Mutinda Mwanzia, \"Kenya: Awori Hands Over to Kalonzo\", The East African Standard, 10 January 2008."
[16] "\"Travel ban in Kenya scam inquiry\", BBC News, 14 February 2006."
[17] "\"Kenyans demand more graft scalps\", BBC News, 17 February 2006."
[18] "Nzau Musau, \"Groups declare war on Awori\", Kenya Times, 22 February 2006."
[19] "\"Kenyan VP passes buck over graft\", BBC News, 22 February 2006."
[20] "\"Association for the Physically Disabled of Kenya ( Home\". 3 August 2019. Retrieved 21 February 2026."
[21] "\"Moody Awori: Vice President Who Ensured Prisoners Were Treated With Dignity, His Rich Legacy - whownskenya\". 27 February 2023. Retrieved 22 February 2026."
[22] "\"President Kenyatta defends appointing 91-year-old Moody Awori\". Citizen Digital. 6 December 2018. Retrieved 21 February 2026."
This gives us the citations, but you see that there is some HTML formatting left, due to the way Wikipedia citations work depending on the template being used. This would require further processing, but we can leave this for future improvements (and highlights the issue with web scraping generally).
Doing some basic NLP
We can now use tidytext and wordcloud to first tokenize the references and explore what the most common words are for the given page:
library(tidytext) # for NLPlibrary(wordcloud) # to render wordclouds
references n
1 mw 24
2 output 23
3 parser 23
4 lock 15
5 id 12
6 skin 12
There’s still some of the problematic labels left from the way Wikipedia does the citation markup, we can manually remove some more by creating our own filter list:
references n
1 february 10
2 kenya 10
3 awori 9
4 retrieved 9
5 2006 7
6 background 7
With this, we can now create a word cloud:
# define a color palette to not just use black as uniform labelpal <-brewer.pal(8,"Dark2")# plot the 20 most common wordstokens_clean %>%with(wordcloud(references, n, random.order =FALSE, max.words =20, colors=pal))
We see there’s still some artifacts coming from the way we extracted the text, let’s try to do better by re-doing the citation extraction, but this time making sure we don’t end up with the markup text for some references:
# get all references, including the ones that have the markup included:references <- ref_section %>%html_elements(xpath="//ol/li//span[contains(@class, 'reference-text')]") %>%html_text2()references <-as.data.frame(references)# ignore rows that contain MW outputreferences <- references %>%filter(!str_detect(references, "mw-parser"))references
references
1 Profile of Moody Arthur Awori
2 Page on Awori at Vice-President web site Archived 2007-09-27 at the Wayback Machine.
3 "Awori Hands Over to Kalonzo", The East African Standard, 10 January 1998.
4 Kenya Government Bio Archived 2005-07-28 at the Wayback Machine.
5 ""Kibaki's new cabinet"". Archived from the original on 16 January 2006. Retrieved 20 February 2006., The Standard (Kenya).
6 "Inside one famous family of scholars and top leaders". Nation. 5 July 2020. Retrieved 28 July 2023.
7 "Kenya Women Billionaires: The Story of Mary Okello - Kenyans.co.ke". www.kenyans.co.ke. 5 October 2021. Retrieved 28 July 2023.
8 "Prof. NO Awori, May 1986 - Nov 1986". University of Nairobi - Department of Surgery.
9 Amalemba, Robert. "Former VP Awori wins back late kin's 30-acre land from trespasser". The Standard. Retrieved 21 February 2026.
10 "Old boys". Kakamegahighschool.com. Archived from the original on 9 October 2011. Retrieved 4 August 2011.
11 Kenya Parliament profile Archived 2006-04-14 at the Wayback Machine.
12 "Reviews under way in Kenya vote", BBC News, 30 December 2007.
13 "Kalonzo VP in Kibaki’s new Cabinet", The Standard, 9 January 2008.
14 Mutinda Mwanzia, "Kenya: Awori Hands Over to Kalonzo", The East African Standard, 10 January 2008.
15 "Travel ban in Kenya scam inquiry", BBC News, 14 February 2006.
16 "Kenyans demand more graft scalps", BBC News, 17 February 2006.
17 Nzau Musau, "Groups declare war on Awori", Kenya Times, 22 February 2006.
18 "Kenyan VP passes buck over graft", BBC News, 22 February 2006.
19 "Association for the Physically Disabled of Kenya ( Home". 3 August 2019. Retrieved 21 February 2026.
20 "Moody Awori: Vice President Who Ensured Prisoners Were Treated With Dignity, His Rich Legacy - whownskenya". 27 February 2023. Retrieved 22 February 2026.
21 "President Kenyatta defends appointing 91-year-old Moody Awori". Citizen Digital. 6 December 2018. Retrieved 21 February 2026.
Now we get the ones with the correct citations:
# now get rows with proper citation bit:references_html <- ref_section %>%html_elements(xpath="//ol/li//span[contains(@class, 'reference-text')]/cite") %>%html_text2()references_html <-as.data.frame(references_html)
references_joined
1 Profile of Moody Arthur Awori
2 Page on Awori at Vice-President web site Archived 2007-09-27 at the Wayback Machine.
3 "Awori Hands Over to Kalonzo", The East African Standard, 10 January 1998.
4 Kenya Government Bio Archived 2005-07-28 at the Wayback Machine.
5 ""Kibaki's new cabinet"". Archived from the original on 16 January 2006. Retrieved 20 February 2006., The Standard (Kenya).
6 "Inside one famous family of scholars and top leaders". Nation. 5 July 2020. Retrieved 28 July 2023.
7 "Kenya Women Billionaires: The Story of Mary Okello - Kenyans.co.ke". www.kenyans.co.ke. 5 October 2021. Retrieved 28 July 2023.
8 "Prof. NO Awori, May 1986 - Nov 1986". University of Nairobi - Department of Surgery.
9 Amalemba, Robert. "Former VP Awori wins back late kin's 30-acre land from trespasser". The Standard. Retrieved 21 February 2026.
10 "Old boys". Kakamegahighschool.com. Archived from the original on 9 October 2011. Retrieved 4 August 2011.
11 Kenya Parliament profile Archived 2006-04-14 at the Wayback Machine.
12 "Reviews under way in Kenya vote", BBC News, 30 December 2007.
13 "Kalonzo VP in Kibaki’s new Cabinet", The Standard, 9 January 2008.
14 Mutinda Mwanzia, "Kenya: Awori Hands Over to Kalonzo", The East African Standard, 10 January 2008.
15 "Travel ban in Kenya scam inquiry", BBC News, 14 February 2006.
16 "Kenyans demand more graft scalps", BBC News, 17 February 2006.
17 Nzau Musau, "Groups declare war on Awori", Kenya Times, 22 February 2006.
18 "Kenyan VP passes buck over graft", BBC News, 22 February 2006.
19 "Association for the Physically Disabled of Kenya ( Home". 3 August 2019. Retrieved 21 February 2026.
20 "Moody Awori: Vice President Who Ensured Prisoners Were Treated With Dignity, His Rich Legacy - whownskenya". 27 February 2023. Retrieved 22 February 2026.
21 "President Kenyatta defends appointing 91-year-old Moody Awori". Citizen Digital. 6 December 2018. Retrieved 21 February 2026.
22 reporter, Nairobian. "The Immortals:Why the Aworis are the Kennedys of Kenya". Standard Entertainment and Lifestyle. Retrieved 23 May 2021.
23 ""Kibaki's new cabinet"". Archived from the original on 16 January 2006. Retrieved 20 February 2006.
Now we can re-do the tokenization and make the word cloud:
Let’s write a function to get a dataframe of references for each of them:
getReferences <-function(politician){ url <-paste("https://en.wikipedia.org/wiki", str_replace_all(politician," ", "_"),sep="/")print(url) html <-read_html(url)# get ref section ref_section <- html %>%html_elements(xpath="//section[contains(@aria-labelledby, 'References')]")# get references that include the markup references <- ref_section %>%html_elements(xpath="//ol/li//span[contains(@class, 'reference-text')]") %>%html_text2() references <-as.data.frame(references)# ignore rows that contain MW output references <- references %>%filter(!str_detect(references, "mw-parser")) references# now get rows with proper citation bit: references_html <- ref_section %>%html_elements(xpath="//ol/li//span[contains(@class, 'reference-text')]/cite") %>%html_text2() references_html <-as.data.frame(references_html)# join references references_joined <-unique(c(references$references, references_html$references_html ) ) references_joined <-as.data.frame(references_joined)as.data.frame(references_joined)}
Now we can create our reference lists:
# get male refsmale_references <-data.frame(references_joined=character())for (mp in male_politicians) { male_references <-bind_rows(male_references, getReferences(mp))}
# plot the 20 most common words for mentokens_clean_male <-tokenize_df(male_references)tokens_clean_male %>%with(wordcloud(references_joined, n, random.order =FALSE, max.words =20, colors=pal))
Top words in references of women
# plot the 20 most common words for womentokens_clean_female <-tokenize_df(female_references)tokens_clean_female %>%with(wordcloud(references_joined, n, random.order =FALSE, max.words =20, colors=pal))
Some simple sentiment analysis
We can now do something similar, to explore the sentiment of the Wiki articles depending on whether those articles are about male or female politicians.
Sentiment analysis works by assigning ‘emotional scores’ to different words, which can then be used to score larger texts. This can be used to figure out if texts have a positive, neutral or negative ‘opinion’ on a text. Or even to get some insights into the emotionality of a text.
To achieve this, sentiment analysis is based on lexica, which have words scored in different ways, which can then be used for the scoring. The textdata package offers different lexica.
For now, we will fetch the full articles, instead of trying to parse only parts of it. Let’s make a function for that:
getBody <-function(politician){ url <-paste("https://en.wikipedia.org/wiki", str_replace_all(politician," ", "_"),sep="/")print(url) html <-read_html(url)# get ref section paragraphs <- html %>%html_elements("p") %>%html_text() # get all paragraphs# remove citations paragraphs <- paragraphs %>%str_replace_all("\\[[0-9]+]", '')as.data.frame(paragraphs)}
Now we can iterate over our male & female politicians as before to get the page content:
# get male refsmale_paragraphs <-data.frame(paragraphs=character())for (mp in male_politicians) { male_paragraphs <-bind_rows(male_paragraphs, getBody(mp))}
For each of those two data frames, we can now convert them into tidy text, as before, including the removal of stop words:
# write function to get the clean_words <-function(df) { df %>%unnest_tokens(paragraphs, paragraphs) %>%anti_join(stop_words, join_by('paragraphs'=='word'))}male_tidytext <-clean_words(male_paragraphs)female_tidytext <-clean_words(female_paragraphs)
Now we can now download a lexicon and get started with the sentiment analysis. Let’s use the bing data set, which scores each word as positive or negative.
# install.packages('textdata') # if necessary, install the libary on your computerbing <-get_sentiments("bing") # load lexicon
For simple demonstration purposes, we will now just create a summary score over the texts of the male and female politician articles:
We can see, the sentiments for the selected articles don’t really differ too much between female politicians and male politicians, with both words appearing roughly in equal parts
This just provides a very quick view into what can be done with these types of analyses
Potential problems and Ideas/suggestions for sentiment analysis
There are a few considerations to
Alternative Lexica
We have used a very simple scoring-lexicon, but there are alternatives, e.g. the afinn lexicon, which scores words on a range from -5 (for negative sentiment) to +5 (for positive sentiments), and nrc not only gives positive and negative as categories but also anger, anticipation, disgust, fear, joy, sadness, surprise, and trust.
Which lexicon you should, or can, use depends on your research question.
Negations
Currently, we do not take negations into account, e.g. it a sentence included not not happy, we would still count the happy as a positive sentiment. To get around this, you can use bigram analyses, where instead of tokenizing to the word-level, you look at the succession of two words, e.g using unnest_tokens(bigram, text, token = "ngrams", n = 2)
Granularity
Currently, we only did a very basic summary statistic of a single number for all articles. You could break it down by looking at per-article statistics. Or even on a per-paragraph level, to see how sentiment differs between sections in an article.
