What's Coming Next from the COVID-19 Canada Open Data Working Group in 2021

What a year

The COVID-19 Canada Open Data Working Group was founded by myself and my colleague Isha Berry in early March of this year to fill a critical data gap—to provide a pan-Canadian picture of the developing COVID-19 pandemic. What began with a simple dataset of cases and deaths and a basic R Shiny dashboard has expanded into much more, most recently VaxView, our tracker for the vaccine rollout in Canada.

What’s next

This pandemic is far from over. This winter, we face what will in many ways be the most challenging phase of the pandemic, even as the hope offered by widespread vaccination in the spring grows ever closer. We will continue tracking this virus as long as it remains relevant.

That being said, some changes are coming in 2021. A lot, actually. This month, we ran a two-week survey on our dataset and dashboard and received a tremendous 242 complete responses—hearing from people from all over Canada who use and/or consume our data. The feedback from these surveys will help us shape our priorities going in 2021. In the coming year, we aim to bring you more data and more ways to explore and interact with these data.

Our datasets are changing and expanding

The growing number of cases in Canada’s second wave has created a growing demand for manual data input from our team. Together, we have identified many parts of our data workflow that could benefit from automation, especially as data offerings from the provinces have become more consistent. This will allow us to work more efficiently and redirect our energy toward expanding our data offerings.

For example, we have identified many provinces with health region-level recovered and testing data, which we do not currently collect. We have also made progress on bringing more demographic data into our individual-level datasets. Our plan is to gradually incorporate these additional datasets into our offerings in the new year.

Another feature of our dataset is that we generally report data based on the date it was publicly reported, since this is the only date that is consistently offered by all jurisdictions. This is sub-optimal for some uses, especially for users focused on a single province, and creates undesirable patterns in the data (for example, for jurisdictions that don’t report on weekends and then report 3 days of data on Mondays).

There is a solution: converting official datasets (e.g., the CSV files offered by many provinces) to be compatible with our dataset to serve as a drop-in replacement. Ultimately, this should be incorporated into the JSON API (e.g., allowing users to download our dataset but with one province’s data being substituted with the official version).

Some feedback in our evaluation surveys was focused on making our data easier to link to other datasets. For example, linking the health region names we use to alternate health region names through the use of the health region unique identifier (HR_UID). Our correspondence files provide this information and population values for all provinces and health regions, but we acknowledge it would be easier if these data were directly incorporated into the time series datasets themselves.

In the new year, we will draw up a proposal for a new data structure (which will only consist of adding new columns to the existing time series). We will then enter into a transition period where datasets will be provided in both old and new formats before the new format replaces the old format as the primary dataset.

If you’d like to be informed of upcoming changes to the dataset or have any comments or feedback, please send us an email at: ccodwg [at] gmail [dot] com.

We’re building an ecosytstem for our data

We’re building an R package ecosystem to make it easier to access, explore and use our data to generate insights. The GitHub repository for our dataset is called Covid19Canada, so it’s only natural that an derivatives of this project build off this naming scheme.

A Canadian COVID-19 data archive for the future

Since late August of 2021, I have been quietly amassing what is almost certainly the largest publicly available collection of Canadian COVID-19 datasettes. The Archive of COVID-19 Data from Canadian Government Sources is a collection of daily snapshots of COVID-19 data from various Canadian government sources (and select non-governmental sources). The basis of the project is a Python script that automatically archives nearly 140 datasets (and counting) every day.

I had toyed with the idea behind this archive for some time, but the final straw that actually convinced me to start it was this story out of Iowa regarding flawed COVID-19 data collection that led to thousands of cases being backdated, often by several months, distorting percent positivity statistics. The flaw was discovered by those tracking retroactive changes to the daily datasets provided the Iowa Department of Public Health.

Earlier this month, a huge flaw was discovered in #Iowa’s COVID-19 data which caused new cases to get backdated. This resulted in the positivity rate for testing in Iowa being underestimated in the present. (3/12)https://t.co/LQl1XVBf8K

— Jean-Paul R. Soucy (@JPSoucy) August 26, 2020

I believe this project will be my most enduring contribution to understanding the COVID-19 epidemic in Canada. My goal is to make it the go-to source for re-constructing the historical record when someone asks “What happened during the COVID-19 epidemic in Canada?”

The next steps for this project are to:

1. Continue to solicit contributions of new datasets and improve metadata for existing datasets.
2. Make the data accessible and searchable via the Covid19CanadaData package.
3. Create a website for publishing and exploring the data via Datasette.

You can find the project landing page and data catalogue on GitHub. Please contact me or make an issue/pull request on GitHub if you have suggestions for datasets to add or if you have data to contribute to the archival effort.

See you in 2021

I’d like to wish everyone a happy New Year. We at the COVID-19 Canada Open Data Working Group look forward to keeping everyone abreast of the latest trends in the COVID-19 epidemic in the year ahead. 2021 will start to look a lot rosier come spring, I promise.

You can discuss this post on Twitter.