Add geomap utilities, hosp integration #137

jsharpna · 2020-07-08T22:09:39Z

Summary of changes

Added geomap utility in _delphi_utils_python/delphi_utils/geomap.py (see tests/test_geomap.py)
Added geo data in _delphi_utils_python/delphi_utils/data
Added data processing scripts going from raw data -> cross tables in _delphi_utils_python/data_proc
Integrated geomap util into emr_hosp and tested by running and comparing receiving

krivard

Almost all of my beefs are with the documentation. Aside from a couple of optional nits (& a spurious commented-out line that should definitely be fixed) the only code thing to double-check is whether megacounties have a side effect of generating averages -- I didn't think they should, but the docs mention a window size for averages and that makes me nervous.

krivard · 2020-07-10T20:55:29Z

_delphi_utils_python/data_proc/geomap/README.md

+
+Authors: Jingjing Tang, James Sharpnack
+
+The data_proc/geomap directory contains original source data, processing scripts, and notes for processing from original source to crosswalk tables in the data directory for the delphi_utils package.


Suggested change

The data_proc/geomap directory contains original source data, processing scripts, and notes for processing from original source to crosswalk tables in the data directory for the delphi_utils package.

The `data_proc/geomap` directory contains original source data, processing scripts, and notes for generating the crosswalk tables in the data directory of the delphi_utils package.

krivard · 2020-07-10T20:56:38Z

_delphi_utils_python/data_proc/geomap/README.md

+Requires the following source files below.
+
+Run the following to write the cross files in the package data dir...


Suggested change

Requires the following source files below.

Run the following to write the cross files in the package data dir...

First, acquire the source files listed below.

Then run the following to write the crosswalk files in the package data dir...

krivard · 2020-07-10T21:02:08Z

_delphi_utils_python/data_proc/geomap/README.md

+
+Some of the source files were constructed by hand, most notably 02_20_uszips.csv.
+
+The 02_20_uszips.csv file is based on the newest consensus data including 5-digit zipcode, fips code, county name, state, population, HRR, HSA (I downloaded the original file from here https://simplemaps.com/data/us-zips. This file matches best to the most recent (2020) situation in terms of the population. But there still exist some matching problems. I manually checked and corrected those lines (~20) with zip-codes.com (https://www.zip-codes.com/zip-code/58439/zip-code-58439.asp). The mapping from 5-digit zipcode to HRR is based on the file in 2017 version downloaded from https://atlasdata.dartmouth.edu/static/supp_research_data


How much of this construction history do we need to retain in this repository?

Could some of this be replaced with shorter explanatory text (e.g. "We tried incorporating data from the cbsatocountycrosswalk.csv file at https://data.nber.org/data/cbsa-fips-county-crosswalk.html but it was worse" instead of the 4/15 and 4/19 entries)? The population files referenced in the 6/15 log are not included here; can we remove that log entry?

Dmitry will be cleaning this README up with his PR.

krivard · 2020-07-10T21:06:25Z

_delphi_utils_python/data_proc/geomap/README.md

+Introduced the March 2020 MSA file, source is [US Census Bureau](https://www.census.gov/geographies/reference-files/time-series/demo/metro-micro/delineation-files.html).  This file seems to differ in a few fips codes from the source for the 02_20_uszip file which Jingjing constructed.  There are at least 10 additional fips in 03_20_msa that are not in the uszip file, and one of the msa codes seems to be incorrect: 49020 (a google search confirms that it is incorrect in uszip and correct in the census data). 
+
+07/08/2020:
+We are reserving 00001-00099 for states codes of the form 100XX where XX is the fips code for the state.  In the case that the CBSA codes change then it should be verified that these are not used.  The current smallest CBSA is 10100.


There's a typo somewhere here; either we should reserve _1_0001-_1_0099, or the codes should be of the form _0_00XX. I vote for the former, since the existing API validation code enforces MSAs to be in the range 10000-99999.

I was inconsistent here, and the code actually uses 000XX. Will more everything to 100XX convention.

krivard · 2020-07-10T21:08:21Z

_delphi_utils_python/data_proc/geomap/geo_data_proc.py

+import pandas as pd
+from os import path
+import json


Minor nit: alphabetize these for (our perennially losing battle for) style consistency with the rest of Delphi

krivard · 2020-07-10T21:39:23Z

_delphi_utils_python/delphi_utils/geomap.py

+       + county -> state : unweighted
+       + county -> msa : unweighted
+       + county -> megacounty
+       - zip -> * not county


(what does "* not county" mean?)

krivard · 2020-07-10T21:41:24Z

_delphi_utils_python/delphi_utils/geomap.py

+                             date_col='date',
+                             mega_col='megafips',
+                             count_cols=None):
+        """convert and aggregate from zip to fips (county)


Suggested change

"""convert and aggregate from zip to fips (county)

"""convert and aggregate from fips to fips, collapsing small-count fips into megacounties

krivard · 2020-07-10T21:43:04Z

_delphi_utils_python/delphi_utils/geomap.py

+        Args:
+            data: pd.DataFrame input data
+            thr_count: numeric, if the sum of counts exceed this, then fips is converted to mega
+            thr_win_len: int, the number of Days to use as an average


Wait does this method also convert counts to average counts? It shouldn't do that, should it?

This is old documentation that was carried over, I think if means that the average is what determines censoring

krivard · 2020-07-10T21:44:27Z

emr_hosp/delphi_emr_hosp/config.py

@@ -64,7 +64,7 @@ class Constants:
    # number of counties in usa, including megacounties
    NUM_COUNTIES = 3141 + 52
    NUM_HRRS = 308
-    NUM_MSAS = 392
+    NUM_MSAS = 392 + 52  # MSA + States


Suggested change

NUM_MSAS = 392 + 52 # MSA + States

NUM_MSAS = 392 + 52 # MSA + one mega-MSA per state

krivard · 2020-07-10T21:46:36Z

emr_hosp/delphi_emr_hosp/update_sensor.py

+            data_frame = gmpr.county_to_state(data,state_id_col=geo)
+        elif geo == "msa":
+            data_frame = gmpr.county_to_msa(data,msa_col=geo)
+        elif geo == "hrr":
            data_frame = data  # data is already adjusted in aggregation step above


(which aggregation step is that?)

krivard · 2020-08-04T18:15:59Z

Currently tracking differences to results generated by the old codebase. Mostly this seems to be due to legit changes in how we're mapping FIPS to metro areas. We'll want to make a note of these changes in the API changelog when we merge this fix.

krivard · 2020-08-05T18:19:23Z

We should investigate the differences between deploy-jhu and main and decide whether to merge to main.

jsharpna added 14 commits June 30, 2020 15:05

added cross generation, load state and zip -> fips tested

6398f0a

add GeoMapper, docstrings, tests

9318662

add zip->county mapping

a8f34b1

added mega v0.1

68fee88

debug gmpr use in hosp

d9d0db0

add gmpr to util init

655dcf6

fix bug from emr

d9a1cc4

add data processing script, source files, and derived files

1610ad1

changed filenames in geomap

a114175

Update README.md

873a3f7

added consistency checks, rest of msa

c473e8b

debug msa rest of state

79c0cd8

Merge branch 'rf_geo' of https://github.com/cmu-delphi/covidcast-indi…

c170fd2

…cators into rf_geo

Merge branch 'main' into rf_geo

647aca3

krivard requested review from amartyabasu and eujing July 9, 2020 18:37

krivard approved these changes Jul 10, 2020

View reviewed changes

jsharpna added 5 commits July 14, 2020 15:09

added jhu to fips converters

2fb7f66

add jhu uid to county

fed6bf1

refactor jhu, add fips to zip utility

e080035

finish hrr, debug jhu, needs stable comp

48de2d4

finish hrr, debug jhu, needs stable comp

885155d

jsharpna added 4 commits August 5, 2020 07:22

compared msa versions

a6aa219

mod compare rec

206bcc1

butcher compare_receiving script

8ec3b13

Merge branch 'main' into rf_geo

325f154

krivard mentioned this pull request Aug 6, 2020

pipeline for Quidel flu test #181

Open

2018 msa raw data

9854433

add mega functionality, struggling with 0.0 bug

78e145e

dshemetov mentioned this pull request Aug 12, 2020

geomap refactor refactor #215

Closed

15 tasks

fix msa mega

eee0acf

krivard merged commit 73308d9 into main Aug 18, 2020


		Authors: Jingjing Tang, James Sharpnack

		The data_proc/geomap directory contains original source data, processing scripts, and notes for processing from original source to crosswalk tables in the data directory for the delphi_utils package.

	The data_proc/geomap directory contains original source data, processing scripts, and notes for processing from original source to crosswalk tables in the data directory for the delphi_utils package.
	The `data_proc/geomap` directory contains original source data, processing scripts, and notes for generating the crosswalk tables in the data directory of the delphi_utils package.

		Requires the following source files below.

		Run the following to write the cross files in the package data dir...


		Some of the source files were constructed by hand, most notably 02_20_uszips.csv.

		The 02_20_uszips.csv file is based on the newest consensus data including 5-digit zipcode, fips code, county name, state, population, HRR, HSA (I downloaded the original file from here https://simplemaps.com/data/us-zips. This file matches best to the most recent (2020) situation in terms of the population. But there still exist some matching problems. I manually checked and corrected those lines (~20) with zip-codes.com (https://www.zip-codes.com/zip-code/58439/zip-code-58439.asp). The mapping from 5-digit zipcode to HRR is based on the file in 2017 version downloaded from https://atlasdata.dartmouth.edu/static/supp_research_data

	"""convert and aggregate from zip to fips (county)
	"""convert and aggregate from fips to fips, collapsing small-count fips into megacounties

	NUM_MSAS = 392 + 52 # MSA + States
	NUM_MSAS = 392 + 52 # MSA + one mega-MSA per state

Add geomap utilities, hosp integration #137

Add geomap utilities, hosp integration #137

Uh oh!

Conversation

jsharpna commented Jul 8, 2020

Uh oh!

krivard left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

krivard commented Aug 4, 2020

Uh oh!

krivard commented Aug 5, 2020

Uh oh!

Uh oh!