Steven W. Buehler

God, Games, and Geekery.

Compiling Daily COVID-19 Charts using Wolfram Script

The novel coronavirus SARS-CoV-2 seems to have taken over the world, and there is a lot of interest in tracking things like infection rates, death rates, measures being taken around the world in an effort to contain it… Like probably everybody else I’ve watched the mainstream media accounts hyping infection counts and death rates with very little in the way of context, and I’ve been very troubled what seems like a mainstream media effort to push FUD (Fear, Uncertainty, Doubt) on an unsophisticated public.

“A person is smart. People are dumb, panicky dangerous animals and you know it.” —Tommy Lee Jones as Agent K, Men in Black (1997).

Having an interest in data (especially visualizing data), I wanted to try pulling it on my own and producing some simple charts to try and wrap some context around what is being presented in mainstream media. I also had, when I started this project, a Raspberry Pi Model 3 B+ Linux “pocket computer” (for lack of a better description) that includes, as part of its standard Raspbian1 install, a free copy of Wolfram Mathematica with just about all of the features of its rather expensive Windows and Mac desktop cousins.

A relatively new part of the Wolfram platform is a scripting language called WolframScript, which allows writing plain-text scripts using the same language as Mathematica, that can be run from the command line, apparently without the need for the full Mathematica desktop. This can be used to pull the needed data, create DateListPlots from that data, export said charts to graphical formats like PNG and SVG, and then in turn call git (the platform that underpins my website using Github Pages) to push those updates to my site on a defined schedule via a cron job, all on my $35 Raspberry Pi.

Since I started developing this project I’ve scaled the ol’ Raspberry Pi 3 (1 GB RAM) to a Raspberry Pi 4 with 4 GB RAM and a vastly improved ARM CPU (improved to the point that it could, in theory, replace my desktop machine if I ever wanted to go exclusively to Linux as my platform), but the development process is the same.

Prerequisites

  1. A computer with WolframScript installed, which is available from here for Windows, Mac, and 64-bit Linux in both .rpm and .deb packages; or already installed on the Raspberry Pi as part of Raspbian.

  2. Git for your platform. If you use a code environment like the free Visual Studio Code it is usually part of that installation.

  3. Of course, access to where you host your git repository. Since my web site runs on Github Pages, it’s a repository on my GitHub account.

Where to get the data

The more popular site for COVID data is Johns Hopkins University Coronavirus Resource Center, but the source data is complex and required some effort to figure out. Instead, I chose the simpler data feed and API from , which is run by volunteers at [_The Atlantic_](https://www.theatlantic.com/) magazine. The Johns Hopkins data is much more detailed than Covidtracking's, but because it is much more detailed it is that much more difficult to process. The Johns Hopkins data also seems to be more aggressive in its countying methdology.

Fortunately, Wolfram has taken care of putting the data in a format usable by Mathematica and has been hosting the data sets in the Wolfram Data Repository to make them easier to work with. To keep things simple (as I’m still learning the Wolfram Language, which has a steep learning curve), I opted to use Covidtracking’s data, which is accessible as a simple data function.

Processing the data in WolframScript

Below is the script that does all the heavy lifting:

  • It pulls the covidtracking.com data from Wolfram’s Data Repository (line 2);
  • It creates all of the relevant charts that I want to make from that data (lines 3–63);
  • It exports those charts to .svg2 graphics to a folder in the local copy of my site’s GitHub repository (lines 64–72);
  • Finally, it calls a shell script that updates the site by calling git with the necessary options to push the updates to the “remote” on GitHub, where you get to see it.

I have another version of this script that, instead of updating the site, combines all of these images into PNG and posts them to my Twitter feed every day. The two scripts are similar except that the “Twitter” script exports all of the charts to a single .png image and then uses Wolfram’s social media connectivity to send the image to Twitter.

#!/bin/bash
cd /home/steven/Documents/XXXXXXXXXX.github.io
git add img/svg/*.svg
git commit --allow-empty -am "Daily update."
git push

Automating the process

In Linux, automating the process is fairly straightforward using cron (I have multiple scripts in the same folder that I use cron to run all of them at once, including the daily COVID update). The actual page source, in Markdown, that contains the references to these graphics is here.

The following is added to my crontab as a for loop triggered at 12:00 noon daily (covidtracking.com claims to update their data at 11:00 AM EDT each day; I give it an hour for Wolfram to update their connection before I update mine):

0 12 * * * for f in /home/steven/Documents/Scripts/WolframScript/*.wls; do $f; done

Conclusion

So that’s the whole process, start to finish, as to how the COVID-19 page gets its daily chart updates.

  1. Raspbian is the Raspberry Pi’s standard Linux distribution, based on Debian, built on ARM, and that takes advantage of its various experimental IO options available with the Pi. 

  2. Scalable Vector Graphics, which is now an Internet standard for vector graphics on the web. 

Github Twitter Facebook LinkedIn RSS