D3 and R, a match made in heaven
A step by step tutorial for converting D3 Gallery examples into R Shiny applications, using your own data
The problem with D3, and how R can help.
You have seen the gorgeous D3 graphs galleries. You have seen the amazing infographics on the New York Times website. Maybe, you felt inspired and went through a few D3 tutorials, bought an online course or a book on the topic. And most likely you have thrown your hands up in frustration when you learned the cold, hard truth:
D3 is hard. Like, really hard.
You probably want to just take one of those amazing graphs on the gallery and get it to work for your project. However, there are many obstacles in your way:
- D3 can’t run in a vacuum. It needs to be presented by a server.
- Even if you get the gallery example to run, you don’t want to use their data, you want to use your data.
- D3 hates you.
Well I have good news for you: there is a way to bring D3 to the people with R. R’s Shiny library turns R code into a web application, acting as a web server. And it turns out, D3 needs a web server to make it spring to life. Were going to take one of those graphs from the D3 gallery, and get it working in Shiny. Were also going to use R’s awesome Tidyverse package to slice and dice our own data, and give the user parameters to change the D3 graph.
Why use D3 anyways?
Why would you want to use D3 with R? Doesn’t R already have like, 35 graphing packages? Why wouldn’t I just use ggplot2 instead? What makes D3 so special.
The answer to this is simple, D3 isn’t just another graphing library. In fact, D3 isn’t a graphing library at all! It is a JavaScript library that can manipulate HTML and make SVG’s (scalable images used by the web). Perhaps an analogy is in order.
If excel graphs were a Hyundai, that might make ggplot2 a Ferrari. D3 would be the factory that builds Ferraris. D3 can build all of the little components that make a great graph, and piece them together into one cohesive shape.
Here are a few reasons you may want to consider D3 over R’s other graphing packages:
- R’s graphing libraries are boxes for your mind. They give you all of these rules to live by. Do you want to put that label in the middle of the bar tilted upside down at 56 degrees in a font you pirated on the dark web? No, you cant do that.
- D3 has no box to live in. There are no rules. You can make the ugliest graph you want. Go ahead and put that label there, D3 doesn’t care.
- Why make a bar chart in 3 lines of ggplot2 code when you can make it in 300 lines of JavaScript instead? We don’t cut corners. This is data science.
Now that I’ve sold you, its time to get to work. Were going to take this example from the D3 gallery and modify it until it does what we want. The example is a bar chart made by the founder of D3 himself; Mike Bostock.
Now, I know a bar chart isn’t exactly exciting. But if you can get R to serve up a D3 bar chart, then you can get R to serve up a gradient boosted, back propagated, force directed radial sunburst D3 chart too. Rock on.
Step 0: Installing the R packages we need
We will need 3 R packages for this tutorial. If you already have these installed, skip to step 1. If not, go ahead and run the below code from the R console to install them.
What do they do? Shiny will make our web application, Tidyverse will slice and dice data, and jsonlite will turn our data into json; which is a format that D3 likes.
Step 1: Creating the Shiny App
In RStudio, create a new shiny app by going to File -> New Project -> New Directory -> Shiny Web Application. Enter a name for your app and select a folder where the project will live. Finally click “Create Project”.
Just like that you have a working shiny application in front of you! The next thing you need to do is create a folder called ‘www’ inside your project directory. You can either do this from your windows file system, or directly from RStudio in the files pane (by default this is in the lower right corner of RStudio)
What is the www folder? That is where web files go (html, css, javascript). Guess what a D3 script is?
Finally, lets clear out the example. Delete everything inside of the UI and Server functions so we have a fresh, blank slate to work from. Lets also add all of the packages we will be using for this tutorial.
The initial app.R:
Step 2: A place for everything, and everything in its place.
Now that we have a home for our D3 files, its time to put things in the right spot. Most D3 examples have 2 to 4 files. Typically these are:
- html file
- css file
- javascript file
- data (usually json or csv)
Sometimes these are all separated, and other times they are combined. In the case of our example, there is just one html file that contains the css and javascript, and one csv that stores the data. We need to split these out. We don’t need to create an html file, because that is what our UI function in Shiny replicates. But we do need to sort out the CSS and JavaScript
Open your code editor of choice (VS Code, Sublime, Notepad, whatever) and copy the CSS to a new file. Save this file into the www folder as ‘style.css’. This is the code you want:
The style.css:
Next up you will see two script sections. The first one is just this little line of code:
This script tells your application “Yo! We are using D3”. For a shiny application, we don't really need an index.html file. That is what our UI function emulates! So that makes it a great place to put any of these types of scripts, especially those inside of a <head> tag. While we are at it, lets go ahead and call our CSS file as well so our app knows where to look when it wants to get stylish!
Last but not least is the actual D3 code. This is everything between the final <script> </script> tags. Put this into a new file called “script.js” inside the www folder.
The initial script.js
If you have made it this far, we are all set up! Don’t worry about the csv file, rather than use that, we will be bringing our own data! Our www directory should now look like this:
Step 3: Lets make the front end
Now it is time to make a user interface for our application. We are going to be using the Tidyverse packages built in ‘mpg’ dataset for our example. This dataset lists many popular car makes and models and their fuel efficiency metrics. Here is the final code for the ui function:
The ‘titlePanel’ function does what you think, it creates a title for our app. The ‘selectInput’ function creates a dropdown list with a few choices for vehicle size. This is the input we will use to change our D3 graph.
Finally we have a ‘uiOutput’ function that is looking for something called ‘d3’. Whenever you see the word ‘Output’ in the ui function (tableOutput, plotOutput, etc), it means the UI function is looking for an object from the server function. In our case, it will be a renderUi output object named d3. We will build that next.
You can actually run this project now and get a super boring app with a title and a dropdown list. Humble beginnings.
Step 4: Build the back end
The server function is the best part of every Shiny app. This is where you get to do data science. This is also where you paste code from stack overflow and hope it works. Now that I think about it, those two things are the same.
Before we get started building the server function, lets look at our dataset. in the RStudio console, type this and hit enter:
Below are the results. This is the mpg dataset. We are going to take this dataset which has 234 rows and get the average city mpg for each manufacturer. This will get us down to 15 rows of data that we will graph. See the ‘class’ field (compact, suv, etc. )? This is the field we are going to hook up to our dropdown in the ui function. When the user selects ‘suv’, our server function will filter the data to just the manufacturers that produce suv’s.
We will acomplish this grouping and aggregation using R’s wonderful Tidyverse library. I don’t want to dive too deep into a lesson on how to slice and dice data with the Tidyverse, that is worthy of an article all on its own. I’m just going to show you the code and briefly describe how this works. Run this from the console:
Results:
What is this code doing? Its actually pretty simple. This is Tidyverse syntax, and it reads from top to bottom. The “%>%” syntax is how the Tidyverse library chains the current state of the code block to the next. You can think of it as ‘and then do this…’. In English it is:
Take the mpg dataset.
Filter to where the records in the class column = 'suv'.
Create a new column called avgCity that is the average of city mpg
Reduce the columns to just manufacturer and avgCity
Remove all duplicate records
Rename manufacturer to 'name' and avgCity to 'value'Now that you understand this, its time to show you the full code for the server function in Shiny.
The final server function
Let me explain this from the top down.
By default, the server function has only input and output. I had to add session to this list of parameters. The session parameter will allow Shiny to send information from the server function to the web page. In our case, we are trying to get our data from R to our JavaScript code in the www folder. This parameter is what makes that happen.
The observeEvent function is monitoring our dropdown in our ui function, recall that we gave it an inputId of ‘vehicleClass’. If the dropdown changes (say from ‘suv’ to ‘compact’), this observeEvent function will notice this change, invalidate the current state of affairs, and rerun this block of code to determine the new state of affairs. Ultimately it will update our D3 graph.
The next block of code is the same as the Tidyverse example we ran above, with one small change:
Rather than hardcode class to ‘suv’, we map it to the vehicleClass dropdown. When you change the dropdown from compact to suv, our filter for the mpg dataset also changes.
Now we use the jsonlite library. This simply takes the state of our aggregated and filtered data, and gets it into a json format. D3 likes json, so we like json.
Now THIS is the magic sauce! This little line of code takes our json payload, and sends it across the session so that it can be consumed by our D3 JavaScript! R is sending a message to our www folder, saying “Yo! here is the data! do something wonderful with it!!”
Finally we have this little bit of code. Recall that our UI function has this uiOutput(“d3”)? Well the same way we can send information from the ui to the server (through input$id syntax), we can also send information from the server to the ui. in this case, we create an object named “output$d3” which is just an html line saying, “render this JavaScript code in the web page”. The name of the script is “script.js”, which is what we named or D3 script!
Now this example wont run just yet. Even though R is now sending our data in a json format over to our D3 script, our D3 script isn’t yet configured to accept it. Only now are we finally ready to venture into the deep dark forest known as D3.
Here is our finished app.R:
Step 5: D3
2000 words later we have made it. We have climbed to the summit of Mount R, just to find the behemoth of Mount D3 before us. Do we despair and turn around as so many before us, or do we venture forth and go where no nerd has gone before.
Well lets not be dramatic about this. We really just need to change a few things and were done. I assume if you’ve actually read this far you’re more than willing to copy/paste your way to the bitter end.
I’m going to rewrite the D3 script so it works with R. When Mike Bostock published this D3 bar chart code, he assumed the person reading it was smart enough to know what it meant, thus he didn’t bother leaving any helpful comments telling us how it works. I am going to assume the person who reads my new version of this code is an idiot and explicitly tell you what it does. Why?
- I am a good person.
- 99% of the time the idiot reading my code is me, 6 months later, after I have forgotten everything. I know that future me is a busy person, has a life, and matters. My documentation and code comments are little gifts from present me, to future me.
Start by running the application. The app will actually pull up, and you can change the dropdown and notice that nothing happens. To troubleshoot D3, we need to click “Open in Browser” then inspect the code in Chrome.
Open the Chrome inspector by right clicking in the browser and going to “inspect” or by simply hitting cntrl + shift + j. Click on the console tab and you will see this error:
This is because our D3 has this bit of code:
We don’t want to use the example “sample-data.csv” file. We want to use our data sent by Shiny’s session object instead. We can fix this by wrapping our entire D3 script in a function designed to receive messages of the type ‘jsondata’.
If you recall, our server function sends out a message of this type (‘jsondata’), and contains the json payload that holds our mpg data:
In other words, Shiny is broadcasting our mpg data as json, and D3 is listening for it. If Shiny changes the dataset (say by altering an input), it will broadcast the new version of this dataset, and D3 will listen for that change and pick it up.
This function really is the magic sauce of making Shiny talk to D3. Once you wrap the entire D3 script in this cool function, you can remove the code that called the csv. We also want to make quite a few changes to make our graph look good in our UI, and map our data properly. Here I will show you the full, final, and commented version of the D3 script. The comments will annotate what all of the changes mean.
The final script.js file:
You will see that the code hasn’t been changed too much, with a few exceptions. I am now doing a few things with the html id of #d3Graph. The problem you will encounter, is that when data is invalidated (when you change the dropdown) and a new version is sent from Shiny to D3, a new graph will be made every time this happens. You will eventually end up with a page full of new graphs, one on top of the other. I added a few lines to the d3 that adds an id to our graph. Every time our function runs, it deletes the old graph solving the problem.
The other notable changes to the code is that I set our data to equal the message from our jsondata message handler at the top of the script:
I also modified the margins to make sure D3 doesn’t put our graph too close to the edge of the browser, giving us some room to breathe:
A final change was to make the bars purple instead of blue. I did this by opening the ‘style.css’ file and modifying the bar fill values:
Step 6: Run the app
Now we can finally run the app:
And changing the dropdown will update the graph!
That’s it! You now understand how to convert D3 gallery examples to a format Shiny likes. You can send your custom data from R over to JavaScript and make beautiful graphs.
Here is the full project on Github! Just download it, and open it in R. The readme will tell you the versions of R and the libraries I used to create this.
