Creating Automated Python Dashboards using Plotly, Datapane, and GitHub Actions
Today, most online data science courses teach you about machine learning and deep learning; you learn how to optimize hyperparameters rather than the fundamentals such as SQL, Data Analysis, Reporting, Cloud Services, etc. But in the real world, analyzing data, learning how to ‘automate the boring stuff,’ creating ETL pipelines and dashboards, and guiding stakeholders or clients to the correct path often delivers much more value. More data teams fail because data science never influences business decisions, than because their model wasn’t optimized.
In the real world, the problems data scientists face aren’t only how to tweak ML models and compare metrics, like they would in Kaggle competitions. The problem they face is to be stuck on local Jupyter Notebooks and don’t have a way to connect with the person consuming their analysis. If we build data products (such as ML models, automated platforms, or reports) that don’t help stakeholders’ or clients’ decision processes, we miss our original “reason for being.”
High-level executives are interested in numbers and insights; not which activation function you used in your neural networks.
The main driving factor of good data products is to collaborate with business units and get feedback regularly. The more you actively participate in the decision-making process, the more you learn about how to test hypotheses and solve “real” problems, not textbook ones. At the end of the day, your findings and stories will turn into actions that drive outcomes.
Unfortunately, you can’t send your notebook to a stakeholder and expect them to understand it and run it themselves. This means that, in a lot of companies, you get an email from a business unit with a request, re-run a Jupyter Notebook, and then manually build a PowerPoint or Excel spreadsheet. This manual process means that data scientists are the bottleneck for insights and spend their days manually building reports instead of solving hard problems.
The initial but critical step to interact with problems is to create KPIs and well-crafted, automated reports and dashboards which can deliver value and remove the need for you to be involved in every question or data refresh. After brainstorming with stakeholders (product, growth, marketing managers, etc) and collaborating with engineers, you could start this process. Luckily, the Python ecosystem provides some handy tools to build and workflows.
In this article, we will be using Python to pull the stock market prices live using pandas datareader and create an interactive report using Plotly and Datapane. We will then use GitHub actions to trigger our code to run every day to update our report.
There are 4 steps to create automated dashboards every day at 6 am (UTC):
- Create Plotly functions for our visualizations
2. Create the report in Python using Datapane’s library
3. Write a .yml file for GitHub actions
4. Share the report online or embed it on blogs
Plotly is an open-source library that makes it easy to create interactive visualizations.
Datapane is an open-source framework that helps us to share insights with team-mates and stakeholders easily without needing 3rd party BI tools or web services.
Github Actions is an automated platform for running code that makes it easy to create custom software development lifecycle workflows directly in your Github repository.
- Create Plotly functions for dashboards
- Created a scatter plot for Netflix stock price
- Added 10-day, 20-day moving averages
- Updated title layout
2. Create Datapane function for reporting
- Added all figures
- Defined cols and rows
You could reach all figures in the GitHub repo.
For more detailed information about layout and customization, please read the documentation.
3. Write a .yml file for GitHub actions
- Added requirements (pandas-reader)
- Scheduled cron jobs (every day at 6 am)
- Added script and token
For more detailed information about the building GitHub Actions, please read the documentation.
4. Share the report online or embed it on blogs
https://datapane.com/u/kaan/reports/stock-analysis/
https://github.com/datapane/gallery/blob/master/stock-reporting/stocks.ipynb
