Intermediate Streamlit
Some Tips and Tricks as You Construct an Evolving App
Streamlit is a great tool to give your data science work an interface. I’ve been using it as a lightweight dashboard tool to display simple visualizations, wrapping python packages in a UI, and exploring model evaluations for NLP models (live examples 1 and 2). It’s allowed me to better understand my data and created an interface that can help translate between code/data/analysis and communication with stakeholders and subject matter experts.
However, today’s prototypes become tomorrow’s production apps. There’s a risk in making things too easy — before you know it you’ve set up expectations that the next iteration will happen just as quickly, or that your app is robust and well tested, or that deploying it company-wide is around the corner, or that the new intern can take it from here to save some money.
As your app grows you’ll need to manage the growing pains. In this post I’ll provide some tips and ideas that I’ve found helpful as my streamit apps evolved. I’ll start with some simple tips to build better apps and end with some ideas for making apps modular and testable.
(As a functional note, any time you see a gist you should be able to run it as a streamlit app and explore yourself. If you’re willing to do a pip install streamlit pandas altair vega_datasets, you can run any of the gists below with streamlit run <gist_url>).
Display Clean Variable Names
The variable names in a DataFrame might be snake cased or formatted in a way not appropriate for end users, e.g. pointless_metric_my_boss_requested_and_i_reluctantly_included. Most of the streamlit widgets contain a format_func parameter which takes function that applies formatting for display to the option values you provide the widget. As a simple example, you could title case each of the variable names.
You can also use this functionality, combined with a dictionary, to explicitly handle the formatting of your values. The example below cleans up the column names from the birdstrikes dataset for use as a dropdown to describe each column.
Use Caching (But Benchmark It First)
It can be tempting to throw that handy @st.cache decorator on everything and hope for the best. However, mindlessly applying caching means that we're missing a great opportunity to get meta and use streamlit to understand where caching helps the most.
Rather than decorating every function, create two versions of each function: one with the decorator and one without. Then do some basic benchmarking of how long it takes to execute both the cached and uncached versions of that function.
In the example below, we simulate loading a large dataset by concatenating 100 copies of the airports dataset, then dynamically selecting the first n rows and describing them.
Since each step (data load, select rows, describe selection) of this is timed, we can see where caching provides a speedup. From my experience with this example, my heuristics for caching are:
- Always cache loading the dataset
- Probably cache functions that take longer than a half second
- Benchmark everything else
I think caching is one of streamlit’s killer features and I know they’re focusing on it and improving it. Caching intelligently is also complex problem, so it’s a good idea to lean more towards benchmarking and validating that the caching functionality is acting as expected.
Create Dynamic Widgets
Many examples focus on creating dynamic visualizations, but don’t forget you can also program dynamic widgets. The simplest example of this need is when two columns in a dataset have a nested relationship and there are two widgets to select values from those two columns. When building an app to filter data, the the dropdown for the first column should change the options available in the second dropdown.
Linking behavior of two dropdowns is a common use case. The example below builds a scatterplot with the cars dataset. We need a dynamic dropdown here because the variable we select for the x-axis doesn't need to be available for selection in the y-axis.
We can also go beyond this basic dynamic functionality: what if we sorted the available y-axis options by their correlation with the selected x variable? We can calculate the correlations and combining this with the widget’s format_func to display variables and their correlations in sorted order.
Make Heavy Use of f-strings & Markdown
In the above example, we used python’s f-strings to interpolate the variable names and their correlation values. In building an interface around the analysis, much of it requires creating or manipulating strings in variable names, widget values, axis labels, widget labels, or narrative description.
If we want to display some analysis in narrative form and there’s a few particular variables we want to highlight, f-strings and markdown can help us out. Beyond an easy way to fill strings with specific variable values, it’s also an easy way to format them inline. For example, we might use something like this to display basic info about a column in a dataset and highlight them in a markdown string.
mean = df["values"].mean()
n_rows = len(df)md_results = f"The mean is **{mean:.2f}** and there are **{n_rows:,}**."st.markdown(md_results)
We’ve used two formats here: .2f to round a float to two decimal places and , to use a comma as a thousands separator. We've also used markdown syntax to bold the values so that they're visually prominent in the text.
Consider Switching to Altair for Visualizations
If you’ve been prototyping visualizations with another library, consider switching to Altair to build your visualizations. In my experience, I think there are three key reasons a switch could be beneficial:
- Altair is probably faster (unless we’re plotting a lot of data)
- It operates directly on pandas DataFrames
- Interactive visualizations are easy to create
On the first point about speed, we can see a drastic speedup if we prototyped using matplotlib. Most of that speedup is just the fact that it takes more time to render a static image and place it in the app compared to rendering a javascript visualization. This is demonstrated in the example app below, which generates a scatterplot for some generated data and outputs the timing for the creation and rendering for each part of the visualization.
Working directly with DataFrames provides another benefit. It can ease the debugging process: if there’s an issue with the input data, we can use st.write(df) to display the DataFrame in a streamlit app and inspect it. This makes the feedback loop for debugging data issues much shorter. The second benefit is that it reduces the amount of transformational glue code sometimes required to create specific visualizations. For basic plots, we could use a DataFrame's plotting methods, but more complex visualizations might require us to restructure our dataset in a way that makes sense with the visualization API. This additional code between the dataset and visualization can be the source of additional complexity and can be a pain point as the app grows. Since Altair uses the Vega-Lite visualization grammar, the functions available in the transforms API can be used to make any visualization appropriate transformations.
Finally, interactive visualizations with Altair are easy. While an app might start by using streamlit widgets to filter and select data, an app could also use a visualization could as the selection mechanism. Rather than communicating information as a string in a widget or narrative, interactive visualizations allow visual communication of aspects of the data within a visualization.
Don’t Neglect Refactoring, Writing Modular Code, and Testing
It’s easy to spend a few hours with streamlit and have a 500 line app.py file that nobody but you understands. If you're handing off your code, deploying your app, or adding a some new functionality it's now possible that you'll be spending a significant amount of time trying to remember how your code works because you've neglected good code hygiene.
If an app is beyond 100 lines of code, it can probably benefit from a refactor. A good first step is to create functions from the code and put those functions in a separate helpers.py file. This also makes it easier to test and benchmark caching on these functions.
There’s no specific right way on how exactly to refactor code, but I’ve developed an exercise that can help when starting an app refactor.
Refactoring Exercise
In the app.py, try to:
- only import streamlit and helper functions (don’t forget to benchmark
@st.cacheon these helper functions) - never create a variable that isn’t input into a streamlit object, i.e. visualization or widget, in the next line of code (with the exception of the data loading function)
These aren’t hard and fast rules to always abide by: you could follow them specifically and have a poorly organized app because you’ve got large, complex functions that do too much. However, they are good objectives to start with when moving from everything in app.py to a more modular structure.
The example below highlights an app before and after going through this exercise.
Another benefit of reorganizing code in this way is that the functions in the helpers file are now easier to write tests for. Sometimes I struggle with coming up with ideas of what to test, but I’ve found that now it’s really easy to come up with tests for my apps because I’m more quickly discovering bugs and edge cases now that I’m interacting more closely with the data and code. Now, any time my app displays a traceback, I fix the function that caused it and write a test to make sure the new behavior is what I expect.
Wrap Up
I’ve been enjoying my time with streamlit — it’s a fantastic tool that’s addressed a clear need in the data science workflow. However, today’s prototypes are tomorrow’s production apps, and it’s easy for a simple app to become an unmaintainable nightmare for a data science team. The ideas covered in this article have helped me manage the pains associated with moving my apps beyond a prototype and I hope they do the same for you.
