bi dashboard html5,html - Scrape website's Power BI dashboard using R - Stack Overflow

The problem is that the site you want to analyze relies on JavaScript to run and fetch the content for you. In such a case, httr::GET is of no help to you.

However, since manual work is also not an option, we have Selenium.

The following does what you're looking for:

library(dplyr)

library(purrr)

library(readr)

library(wdman)

library(RSelenium)

library(xml2)

library(selectr)

# using wdman to start a selenium server

selServ

port = 4444L,

version = 'latest',

chromever = '84.0.4147.30', # set this to a chrome version that's available on your machine

)

# using RSelenium to start a chrome on the selenium server

remDr

remoteServerAddr = 'localhost',

port = 4444L,

browserName = 'chrome'

)

# open a new Tag on Chrome

remDr$open()

# navigate to the site you wish to analyze

report_url

remDr$navigate(report_url)

# find and click the button leading to the Zip Code data

zipCodeBtn

zipCodeBtn$clickElement()

# fetch the site source in XML

zipcode_data_table %

querySelector("div.pivotTable")

Now we have the page source read into R, probably what you had in mind when you started your scraping task.

From here on it's smooth sailing and merely about converting that xml to a useable table:

col_headers %

querySelectorAll("div.columnHeaders div.pivotTableCellWrap") %>%

map_chr(xml_text)

rownames %

querySelectorAll("div.rowHeaders div.pivotTableCellWrap") %>%

map_chr(xml_text)

zipcode_data %

querySelectorAll("div.bodyCells div.pivotTableCellWrap") %>%

map(xml_parent) %>%

unique() %>%

map(~ .x %>% querySelectorAll("div.pivotTableCellWrap") %>% map_chr(xml_text)) %>%

setNames(col_headers) %>%

bind_cols()

# tadaa

df_final %

type_convert(trim_ws = T, na = c(""))

The resulting df looks like this:

> df_final

# A tibble: 15 x 5

zipcode `Confirmed Cases ` `% of Total Cases ` `Deaths ` `% of Total Deaths `

1 63301 1549 17.53% 40 28.99%

2 63366 1364 15.44% 38 27.54%

3 63303 1160 13.13% 21 15.22%

4 63385 1091 12.35% 12 8.70%

5 63304 1046 11.84% 3 2.17%

6 63368 896 10.14% 12 8.70%

7 63367 882 9.98% 9 6.52%

8 534 6.04% 1 0.72%

9 63348 105 1.19% 0 0.00%

10 63341 84 0.95% 1 0.72%

11 63332 64 0.72% 0 0.00%

12 63373 25 0.28% 1 0.72%

13 63386 17 0.19% 0 0.00%

14 63357 13 0.15% 0 0.00%

15 63376 5 0.06% 0 0.00%