The novel coronavirus SARS-CoV-2 seems to have taken over the world, and there is a lot of interest in tracking things like infection rates, death rates, measures being taken around the world in an effort to contain it… Like probably everybody else I’ve watched the mainstream media accounts hyping infection counts and death rates with very little in the way of context, and I’ve been very troubled what seems like a mainstream media effort to push FUD (Fear, Uncertainty, Doubt) on an unsophisticated public.
“A person is smart. People are dumb, panicky dangerous animals and you know it.” —Tommy Lee Jones as Agent K, Men in Black (1997).
Having an interest in data (especially visualizing data), I wanted to try pulling it on my own and producing some simple charts to try and wrap some context around what is being presented in mainstream media. I also had, when I started this project, a Raspberry Pi Model 3 B+ Linux “pocket computer” (for lack of a better description) that includes, as part of its standard Raspbian1 install, a free copy of Wolfram Mathematica with just about all of the features of its rather expensive Windows and Mac desktop cousins.
A relatively new part of the Wolfram platform is a scripting language called WolframScript, which allows writing plain-text scripts using the same language as Mathematica, that can be run from the command line, apparently without the need for the full Mathematica desktop. This can be used to pull the needed data, create
DateListPlots from that data, export said charts to graphical formats like
SVG, and then in turn call
git (the platform that underpins my website using Github Pages) to push those updates to my site on a defined schedule via a
cron job, all on my $35 Raspberry Pi.
Since I started developing this project I’ve scaled the ol’ Raspberry Pi 3 (1 GB RAM) to a Raspberry Pi 4 with 4 GB RAM and a vastly improved ARM CPU (improved to the point that it could, in theory, replace my desktop machine if I ever wanted to go exclusively to Linux as my platform), but the development process is the same.
A computer with WolframScript installed, which is available from here for Windows, Mac, and 64-bit Linux in both
.debpackages; or already installed on the Raspberry Pi as part of Raspbian.
Of course, access to where you host your
gitrepository. Since my web site runs on Github Pages, it’s a repository on my GitHub account.
Where to get the data
The more popular site for COVID data is Johns Hopkins University Coronavirus Resource Center, but the source data is complex and required some effort to figure out. Instead, I chose the simpler data feed and API from
Fortunately, Wolfram has taken care of putting the data in a format usable by Mathematica and has been hosting the data sets in the Wolfram Data Repository to make them easier to work with. To keep things simple (as I’m still learning the Wolfram Language, which has a steep learning curve), I opted to use Covidtracking’s data, which is accessible as a simple data function.
Processing the data in WolframScript
Below is the script that does all the heavy lifting:
- It pulls the covidtracking.com data from Wolfram’s Data Repository (line 2);
- It creates all of the relevant charts that I want to make from that data (lines 3–63);
- It exports those charts to
.svg2 graphics to a folder in the local copy of my site’s GitHub repository (lines 64–72);
- Finally, it calls a shell script that updates the site by calling
gitwith the necessary options to push the updates to the “remote” on GitHub, where you get to see it.
I have another version of this script that, instead of updating the site, combines all of these images into
PNG and posts them to my Twitter feed every day. The two scripts are similar except that the “Twitter” script exports all of the charts to a single
.png image and then uses Wolfram’s social media connectivity to send the image to Twitter.
#!/bin/bash cd /home/steven/Documents/XXXXXXXXXX.github.io git add img/svg/*.svg git commit --allow-empty -am "Daily update." git push
Automating the process
In Linux, automating the process is fairly straightforward using
cron (I have multiple scripts in the same folder that I use cron to run all of them at once, including the daily COVID update). The actual page source, in Markdown, that contains the references to these graphics is here.
The following is added to my
crontab as a
for loop triggered at 12:00 noon daily (covidtracking.com claims to update their data at 11:00 AM EDT each day; I give it an hour for Wolfram to update their connection before I update mine):
0 12 * * * for f in /home/steven/Documents/Scripts/WolframScript/*.wls; do $f; done
So that’s the whole process, start to finish, as to how the COVID-19 page gets its daily chart updates.