Crawling DOI from a SAGE

Recently a friend told me they were doing a systematic review of the Hand Therapy Journal published by SAGE. They wanted a way to scrape the journal for all publications, without going mad in the process. Seemed like a good excuse to get them motivated to learn R and for me to practice web scraping with the rvest package.

We’ll go through the following steps:

1. Generate URLs for all issues of the journal
2. Inspect the source code for the page for the DOI
3. Scrape all the pages
4. Scrape all the pages using parallel processing via the future package

library("tidyverse")
library("rvest")
library("glue")
library("httr")
library("future")

URLs for each journal issue

All issues have the following URL structure:

https://journals.sagepub.com/toc/hthb/{volume}/{issue}

There have always been 4 issues a year, and the most recent volume is 23:

issue_urls <- tibble(
  volume = 1:23,
  issue_1 = 1,
  issue_2 = 2,
  issue_3 = 3,
  issue_4 = 4
)

issue_urls <- issue_urls |>
  pivot_longer(
    issue_1:issue_4,
    names_to = "issue.colname",
    values_to = "issue"
  ) |>
  select(-issue.colname) |>
  arrange(volume) |>
  mutate(
    issue_url = glue("https://journals.sagepub.com/toc/hthb/{volume}/{issue}")
  )

head(issue_urls)

# A tibble: 6 × 3
  volume issue issue_url                                
   <int> <dbl> <glue>                                   
1      1     1 https://journals.sagepub.com/toc/hthb/1/1
2      1     2 https://journals.sagepub.com/toc/hthb/1/2
3      1     3 https://journals.sagepub.com/toc/hthb/1/3
4      1     4 https://journals.sagepub.com/toc/hthb/1/4
5      2     1 https://journals.sagepub.com/toc/hthb/2/1
6      2     2 https://journals.sagepub.com/toc/hthb/2/2

Inspect the source code for the DOI

Inspecting the source code reveals that the title of each article has the attribute data-item-name with value click-article-title. Let’s use the most recent issue as a toy example:

"https://journals.sagepub.com/toc/hthb/23/4" |>
  read_html() %>%
  html_nodes("[data-item-name=click-article-title]") %>%
  html_attrs() %>%
  .[[1]]

The DOI for the article is almost the href value. Here’s our scraping function:

get_article_dois_from_issue <- function(issue_url) {
  issue_page <- tryCatch(issue_url |> read_html(), error = function(c) NA)

  if (is.na(issue_page[[1]])) {
    return(NA)
  }

  issue_url |>
    read_html() |>
    html_nodes("[data-item-name=click-article-title]") |>
    html_attr("href")
}

Scrape all the pages

The purrr package lets us insert the DOIs into the rows of our tibble as list columns:

example_dois <- issue_urls |>
  slice(52:54) |>
  mutate(doi = map(issue_url, get_article_dois_from_issue))

The unnest() function from tidyr unpacks these list columns, and all DOI begin with 10.¹ which we can use to tidy up the raw hrefs:

example_dois |>
  unnest(doi) |>
  mutate(doi = str_replace(doi, ".*/10.", "http://doi.org/10."))

Scrape all the pages with future

To run in parallel:

plan("multicore")

scraped_dois <- issue_urls |>
  mutate(dois = map(issue_url, ~ future(get_article_dois_from_issue(.x))))

scraped_dois |>
  mutate(dois = map(dois, value)) |>
  unnest(dois) |>
  filter(!is.na(dois)) |>
  rename(doi = dois) |>
  mutate(doi = str_replace(doi, ".*/10.", "http://doi.org/10.")) |>
  select(-issue_url)

This was the final output my friend needed — all 459 DOI-issued articles from the journal. If I was asked how to make this more rigorous I’d recommend:

• Programmatically discovering the most recent volume and issue
• Not assuming a max of 4 issues per volume

Footnotes

1. See the DOI Handbook DOI: 10.1000/182.