Project 3

the Air that I Breathe

Team member choice due 3/22/19 at 8:59 pm Chicago time
Project alpha version due 4/15/19 at
8:59 pm Chicago time
Project final version due 4/29/19 at
8:59 pm Chicago time



Project 3 will be the second group project and the focus here will be on more local and current data from the Array of Things in Chicago, with the idea that in the near future there will be more and more sites (and people) collecting similar data you have been using in Projects 1 and 2, and using it in combination with many other datasets to create much more local and even personal views. This project will also deal with sensors giving different values, sometimes wildly different, for the same measurement and how to make choices about what to show.

Again we will be using the main classroom wall as the target display, and again your interface should work with touch on that wall for  normal sized person.

You can keep your group from Project 2 or form new groups. Send Andy an email once you know your group for project 3 (even if it is the same as for project 2). I will create groups for people that do not form groups on their own. Again the standard group size will be 2 or 3 people per group.
 
As with Project 2 you should very quickly set up a web page for your new group project and send the URL to andy. Each Friday of the project each team member should post on the project web site an overview of what he/she did on the project that week.




General information on the Array of things:

https://aot-file-browser.plenar.io/data-sets/chicago-complete
https://medium.com/array-of-things/five-years-100-nodes-and-more-to-come-d3802653db9f

https://arrayofthings.github.io/

Information on the individual sensors:
https://github.com/waggle-sensor/sensors/blob/develop/README.md

and on the API and the library for R
The R library used to work, but now seems to be out of date. It may return or you may need to grab the json files directly and manipulate them (which is what the R library did behind the scenes)

https://arrayofthings.docs.apiary.io/#introduction/api-endpoints
https://github.com/UrbanCCD-UChicago/aot-client-r

the documentation believes it can be installed from
devtools::install_github("UbranCCD-UChicago/aot-client-r")
but that always seems to fail.

A few weeks ago it was possible to grab the zip archive from github, unzip it, and then:
install.packages("devtools")
devtools::install_local("aot-client-r-master")
library(AotClient)
but that is also not happy as of mid March 2019, possibly due to some code changes during their early march hackathon
* a temporary solution (thanks Emmanuel) is to edit the NAMESPACE file from the zip archive and comment out this line and then install with devtools: #export(ls.raw_observations)

and then you could do things like:
get a list of all the nodes
ls.nodes(filters=list())
until its fixed the same information is available at:
https://api.arrayofthings.org/api/nodes
which is where that call grabs its data from

more human readable data is available at:
https://aot-file-browser.plenar.io/data-sets/chicago-complete
if you go to the Node Information part of the page and click on the 'Show Table' button


noticing that node 053 is close to the UIC campus you can find out more about it
stat.node("053")
or directly with
https://api.arrayofthings.org/api/nodes/053span>



and then see if there is a current temperature reading at that location (from one of the many temperature sensors)
a = ls.observations()
b = subset(a, node_vsn == "053")
c = subset(b, sensor_path == "metsense.bmp180.temperature")

until the R interface comes back, the same information can be grabbed directly using:

url <- "https://api.arrayofthings.org/api/observations" or https://api.arrayofthings.org/api/observations?size=1000 if you want more than the standard 200 or filtering by timestamp if you want a specific range
s <- download.file(url, "/var/tmp/obs", quiet = FALSE) (note that you should use a unique file name to avoid crashing into another group's data)
t = fromJSON("/var/tmp/obs")
u = t$data
v = subset(u, sensor_path == "metsense.bmp180.temperature")

some of the resulting readings make sense, others seem way out of range

so maybe try a different temperature sensor with v = subset(u, sensor_path == "metsense.tmp112.temperature")

or v = subset(u, sensor_path == "metsense.tsys01.temperature")

or take a look at  the current humidity at that location
d = subset(u, sensor_path == "metsense.htu21d.humidity")

also note that the timestamp is in GMT


there are various sensors looking at the same pollutants that we have been looking at in the other projects.
ls.sensors()
chemsense.co.concentration     0-1000 ppm
chemsense.h2s.concentration     0-50 ppm
chemsense.no2.concentration     0-20 ppm
chemsense.o3.concentration     0-20 ppm
chemsense.so2.concentration     0-20 ppm

the particulate matter ones don't seem to be online yet, but may come on line soon.
alphasense.opc_n2.pm10     μg/m^3
alphasense.opc_n2.pm2_5     μg/m^3

plantower.pms7003.pm10_atm     μg/m^3
plantower.pms7003.pm25_atm     μg/m^3


-----------------------------------

One of the most popular and accurate weather APIs was from Weather Underground, which transitioned in 2018 to a pay only model, so we can't use it. Dark Sky has a similar API with 1000 calls per day for free, so we are going to integrate their API:
https://CRAN.R-project.org/package=darksky
https://github.com/hrbrmstr/darksky
https://darksky.net/dev

library(darksky)
you will need to get a free API key.

then you can get the current weather forecast for Chicago (UIC) with
get_current_forecast(41.870, -87.647)

or for Chicago on New Year's Day 2019
get_forecast_for(41.870, -87.647, "2019-01-01T12:00:00-0600", add_headers=TRUE)

-----------------------------------

OpenAQ also has an R library that allows 2000 requests every 5 minutes
https://github.com/ropensci/ropenaq
https://ropensci.org/tutorials/ropenaq_tutorial/

library("ropenaq")
which allows commands such as
aq_latest(country = "US", city = "Chicago-Naperville-Joliet")
or
aq_measurements(city = "Chicago-Naperville-Joliet", date_from = "2018-12-01", date_to = "2018-12-31", parameter = "pm25")

note that OpenAQ uses tibbles.



A major focus of this project is going to be on querying the data dynamically in real-time, and trying to integrate data from multiple sources to compare data for a wider area with the data collected at individual locations so see how much of a difference there can be in a city like Chicago.


For a C you need to create a useful, responsive interactive visualization focusing on the Array of Things data for the Chicago area


For a B you need to add in data for the Chicago area from Dark Sky and the previous Projects:


For an A you need to add in data for the Chicago area from OpenAQ:


Graduate Students need to add:


In all of these case you need to make sure that your visualizations are well constructed with good color and font choices, proper labeling, and that they effectively reveal the truth about the data to the user

Note that as part of the web page part of the grade you will need to use your interface to show your findings, so make sure that the way your interface displays information is clear.

Again, your app will be evaluated running full screen with touch interaction on the classroom wall and should not require scrolling.


For this project you should host your solution using the evl shiny server. There are a new set of logins for this project.



There are two deadlines for this project. By the first deadline you should have implemented the initial screen layout of your application and have the basic functionality allowing the user to perform an example of the various 'C' functionality on the classroom wall. This will make sure that your group is on track and that you can focus on making a good interface and set of visualizations, not just functional ones. Personally, I think you should have the entire C functionality done at that point if you are going for an A on the project as a whole. You should make this version of the interface available on your group project page.

As part of the final turn in you should create a set of web pages that describe your work on the project. This should include:

all of which should have plenty of screenshots with meaningful captions. Web pages like this can be very helpful later on in helping you build up a portfolio of your work when you start looking for a job so please put some effort into it.

Be sure to document any external libraries, tools, etc. that you make use of - give credit where credit is due for everything that you didn't create yourself.

You should also create a 2-3 minute YouTube video showing the use of your application including narration with decent audio quality. That video should be in a very obvious place on your main project web page. The easiest way to do this is to either shoot it at the classroom wall or use a screen-capture tool while interacting with your application, though you will most likely find its useful to do some editing afterwards to tighten the video up. Its also a good idea to have a video like this available as a backup during your presentation just in case of gremlins.
You may want to shoot this video on the wall itself.

The web page including screen snapshots and video need to be done by the deadline so be sure to leave enough time to get that work done.

I will be linking your web page to the course notes so please send andy and sai a nice jpg image of your visualization for the web. This should be named p3.<someone_in_your_groups_last_name>.jpg. 



When the project is done, each person in the group should also send Andy a private email with no one else CC'd ranking your coworkers on the project on a scale from 1 (low) to 5 (high) in terms of how good a coworker they were on the project. If you never want to work with them again, give them a 1. If this person would be a first choice for a partner on a future project then give them a 5. If they did what was expected but nothing particularly good or bad then give them a 3. By default your score should be 3 unless you have a particular reason to increase or decrease the number. If you are giving a score other than 3 you need to say why. Please confine your responses to 1, 2, 3, 4, 5 and no 1/3ds or .5s please. Each person's score on the project will be based on the overall score for the group modified by these rankings.



Each group will show their visualization to the class and describe its features. This allows everyone to see a variety of solutions to the problem, and a variety of implementations. Rehearse your presentation ... several times. All team members are expected to participate equally in that presentation.




Project 3 groups:

1 Masoud, Rami & Munoz, Jonathan & Kumar, Mudit & Raol, Jeet
2 Laczny, Nicole & Ptasznik, Conrad & Charlesworth, Casey-Leigh
3 Thiruvarpu Neelakantan, Sai Krishnan & Chandrasekaran, Praveen & Jayaraman, Varsha & Aleem, Abdullah
4 Mantovani, Mirko & Vasudevan, Abhishek & Khemani, Ashwani
5 Toher, William & Vo, Dylan & Bedu, Wilfried Christophe
6 Qarni, Fatima & Martinez, Emmanuel
7 Patel, Bhargav & Ilic, Aleksandar & Brahmbhatt, Rohan & Lin, Catherine
8 Baiju, Aiswarya & Baig, Mirza & Badriyah, Arsh & Abdul salam, Mohammad
9 Nunez, Jhon & Arellano, Daisy & Lazcano, Cesar & Patel, Jay
10 Kumar, Vikas & Rajesh, Yazhini & Patel, Kevalkumar
13 Tlemat, Anas & Hatten, Danielle & Hruby, Raymond


Note that group numbers are changing for project 3 and there will be new accounts on the shiny.evl.uic.edu server.

These new accounts will be in the form p3g#



current list of packages installed on the evl shiny server:

> installed.packages()[,1:2]

                Package           LibPath                        

AotClient       "AotClient"       "/usr/local/lib/R/site-library"

askpass         "askpass"         "/usr/local/lib/R/site-library"

assertthat      "assertthat"      "/usr/local/lib/R/site-library"

backports       "backports"       "/usr/local/lib/R/site-library"

base64enc       "base64enc"       "/usr/local/lib/R/site-library"

BH              "BH"              "/usr/local/lib/R/site-library"

bindr           "bindr"           "/usr/local/lib/R/site-library"

bindrcpp        "bindrcpp"        "/usr/local/lib/R/site-library"

bitops          "bitops"          "/usr/local/lib/R/site-library"

brew            "brew"            "/usr/local/lib/R/site-library"

broom           "broom"           "/usr/local/lib/R/site-library"

callr           "callr"           "/usr/local/lib/R/site-library"

cdlTools        "cdlTools"        "/usr/local/lib/R/site-library"

cellranger      "cellranger"      "/usr/local/lib/R/site-library"

classInt        "classInt"        "/usr/local/lib/R/site-library"

cli             "cli"             "/usr/local/lib/R/site-library"

clipr           "clipr"           "/usr/local/lib/R/site-library"

clisymbols      "clisymbols"      "/usr/local/lib/R/site-library"

colorspace      "colorspace"      "/usr/local/lib/R/site-library"

colourpicker    "colourpicker"    "/usr/local/lib/R/site-library"

crayon          "crayon"          "/usr/local/lib/R/site-library"

crosstalk       "crosstalk"       "/usr/local/lib/R/site-library"

crul            "crul"            "/usr/local/lib/R/site-library"

curl            "curl"            "/usr/local/lib/R/site-library"

darksky         "darksky"         "/usr/local/lib/R/site-library"

dashboardthemes "dashboardthemes" "/usr/local/lib/R/site-library"

data.table      "data.table"      "/usr/local/lib/R/site-library"

DBI             "DBI"             "/usr/local/lib/R/site-library"

dbplyr          "dbplyr"          "/usr/local/lib/R/site-library"

desc            "desc"            "/usr/local/lib/R/site-library"

devtools        "devtools"        "/usr/local/lib/R/site-library"

dichromat       "dichromat"       "/usr/local/lib/R/site-library"

digest          "digest"          "/usr/local/lib/R/site-library"

dplyr           "dplyr"           "/usr/local/lib/R/site-library"

DT              "DT"              "/usr/local/lib/R/site-library"

dygraphs        "dygraphs"        "/usr/local/lib/R/site-library"

e1071           "e1071"           "/usr/local/lib/R/site-library"

ellipsis        "ellipsis"        "/usr/local/lib/R/site-library"

evaluate        "evaluate"        "/usr/local/lib/R/site-library"

fansi           "fansi"           "/usr/local/lib/R/site-library"

fasttime        "fasttime"        "/usr/local/lib/R/site-library"

feather         "feather"         "/usr/local/lib/R/site-library"

forcats         "forcats"         "/usr/local/lib/R/site-library"

fs              "fs"              "/usr/local/lib/R/site-library"

fst             "fst"             "/usr/local/lib/R/site-library"

future          "future"          "/usr/local/lib/R/site-library"

gdtools         "gdtools"         "/usr/local/lib/R/site-library"

generics        "generics"        "/usr/local/lib/R/site-library"

geojson         "geojson"         "/usr/local/lib/R/site-library"

geojsonio       "geojsonio"       "/usr/local/lib/R/site-library"

ggmap           "ggmap"           "/usr/local/lib/R/site-library"

ggplot2         "ggplot2"         "/usr/local/lib/R/site-library"

ggrepel         "ggrepel"         "/usr/local/lib/R/site-library"

ggridges        "ggridges"        "/usr/local/lib/R/site-library"

ggthemes        "ggthemes"        "/usr/local/lib/R/site-library"

ggvis           "ggvis"           "/usr/local/lib/R/site-library"

gh              "gh"              "/usr/local/lib/R/site-library"

git2r           "git2r"           "/usr/local/lib/R/site-library"

globals         "globals"         "/usr/local/lib/R/site-library"

glue            "glue"            "/usr/local/lib/R/site-library"

gmailr          "gmailr"          "/usr/local/lib/R/site-library"

gridExtra       "gridExtra"       "/usr/local/lib/R/site-library"

gtable          "gtable"          "/usr/local/lib/R/site-library"

haven           "haven"           "/usr/local/lib/R/site-library"

hexbin          "hexbin"          "/usr/local/lib/R/site-library"

highr           "highr"           "/usr/local/lib/R/site-library"

hms             "hms"             "/usr/local/lib/R/site-library"

htmltools       "htmltools"       "/usr/local/lib/R/site-library"

htmlwidgets     "htmlwidgets"     "/usr/local/lib/R/site-library"

httpcode        "httpcode"        "/usr/local/lib/R/site-library"

httpuv          "httpuv"          "/usr/local/lib/R/site-library"

httr            "httr"            "/usr/local/lib/R/site-library"

ini             "ini"             "/usr/local/lib/R/site-library"

jpeg            "jpeg"            "/usr/local/lib/R/site-library"

jqr             "jqr"             "/usr/local/lib/R/site-library"

jsonlite        "jsonlite"        "/usr/local/lib/R/site-library"

knitr           "knitr"           "/usr/local/lib/R/site-library"

labeling        "labeling"        "/usr/local/lib/R/site-library"

later           "later"           "/usr/local/lib/R/site-library"

lazyeval        "lazyeval"        "/usr/local/lib/R/site-library"

leaflet         "leaflet"         "/usr/local/lib/R/site-library"

listenv         "listenv"         "/usr/local/lib/R/site-library"

lubridate       "lubridate"       "/usr/local/lib/R/site-library"

lwgeom          "lwgeom"          "/usr/local/lib/R/site-library"

magrittr        "magrittr"        "/usr/local/lib/R/site-library"

mapdata         "mapdata"         "/usr/local/lib/R/site-library"

maps            "maps"            "/usr/local/lib/R/site-library"

maptools        "maptools"        "/usr/local/lib/R/site-library"

mapview         "mapview"         "/usr/local/lib/R/site-library"

markdown        "markdown"        "/usr/local/lib/R/site-library"

memoise         "memoise"         "/usr/local/lib/R/site-library"

mime            "mime"            "/usr/local/lib/R/site-library"

miniUI          "miniUI"          "/usr/local/lib/R/site-library"

modelr          "modelr"          "/usr/local/lib/R/site-library"

munsell         "munsell"         "/usr/local/lib/R/site-library"

NISTunits       "NISTunits"       "/usr/local/lib/R/site-library"

NLP             "NLP"             "/usr/local/lib/R/site-library"

openssl         "openssl"         "/usr/local/lib/R/site-library"

pillar          "pillar"          "/usr/local/lib/R/site-library"

pkgbuild        "pkgbuild"        "/usr/local/lib/R/site-library"

pkgconfig       "pkgconfig"       "/usr/local/lib/R/site-library"

pkgload         "pkgload"         "/usr/local/lib/R/site-library"

plogr           "plogr"           "/usr/local/lib/R/site-library"

plotly          "plotly"          "/usr/local/lib/R/site-library"

plyr            "plyr"            "/usr/local/lib/R/site-library"

png             "png"             "/usr/local/lib/R/site-library"

prettyunits     "prettyunits"     "/usr/local/lib/R/site-library"

processx        "processx"        "/usr/local/lib/R/site-library"

progress        "progress"        "/usr/local/lib/R/site-library"

promises        "promises"        "/usr/local/lib/R/site-library"

protolite       "protolite"       "/usr/local/lib/R/site-library"

ps              "ps"              "/usr/local/lib/R/site-library"

purrr           "purrr"           "/usr/local/lib/R/site-library"

R6              "R6"              "/usr/local/lib/R/site-library"

raster          "raster"          "/usr/local/lib/R/site-library"

rcmdcheck       "rcmdcheck"       "/usr/local/lib/R/site-library"

RColorBrewer    "RColorBrewer"    "/usr/local/lib/R/site-library"

Rcpp            "Rcpp"            "/usr/local/lib/R/site-library"

readr           "readr"           "/usr/local/lib/R/site-library"

readxl          "readxl"          "/usr/local/lib/R/site-library"

rematch         "rematch"         "/usr/local/lib/R/site-library"

remotes         "remotes"         "/usr/local/lib/R/site-library"

reprex          "reprex"          "/usr/local/lib/R/site-library"

reshape         "reshape"         "/usr/local/lib/R/site-library"

reshape2        "reshape2"        "/usr/local/lib/R/site-library"

rgdal           "rgdal"           "/usr/local/lib/R/site-library"

rgeos           "rgeos"           "/usr/local/lib/R/site-library"

RgoogleMaps     "RgoogleMaps"     "/usr/local/lib/R/site-library"

rjson           "rjson"           "/usr/local/lib/R/site-library"

rlang           "rlang"           "/usr/local/lib/R/site-library"

rmarkdown       "rmarkdown"       "/usr/local/lib/R/site-library"

ropenaq         "ropenaq"         "/usr/local/lib/R/site-library"

rprojroot       "rprojroot"       "/usr/local/lib/R/site-library"

rstudioapi      "rstudioapi"      "/usr/local/lib/R/site-library"

rvest           "rvest"           "/usr/local/lib/R/site-library"

satellite       "satellite"       "/usr/local/lib/R/site-library"

scales          "scales"          "/usr/local/lib/R/site-library"

selectr         "selectr"         "/usr/local/lib/R/site-library"

sessioninfo     "sessioninfo"     "/usr/local/lib/R/site-library"

sf              "sf"              "/usr/local/lib/R/site-library"

shiny           "shiny"           "/usr/local/lib/R/site-library"

shinyalert      "shinyalert"      "/usr/local/lib/R/site-library"

shinyBS         "shinyBS"         "/usr/local/lib/R/site-library"

shinycssloaders "shinycssloaders" "/usr/local/lib/R/site-library"

shinydashboard  "shinydashboard"  "/usr/local/lib/R/site-library"

shinyjs         "shinyjs"         "/usr/local/lib/R/site-library"

shinythemes     "shinythemes"     "/usr/local/lib/R/site-library"

shinyWidgets    "shinyWidgets"    "/usr/local/lib/R/site-library"

slam            "slam"            "/usr/local/lib/R/site-library"

sourcetools     "sourcetools"     "/usr/local/lib/R/site-library"

sp              "sp"              "/usr/local/lib/R/site-library"

splitstackshape "splitstackshape" "/usr/local/lib/R/site-library"

stringi         "stringi"         "/usr/local/lib/R/site-library"

stringr         "stringr"         "/usr/local/lib/R/site-library"

svglite         "svglite"         "/usr/local/lib/R/site-library"

sys             "sys"             "/usr/local/lib/R/site-library"

tibble          "tibble"          "/usr/local/lib/R/site-library"

tidyr           "tidyr"           "/usr/local/lib/R/site-library"

tidyselect      "tidyselect"      "/usr/local/lib/R/site-library"

tidyverse       "tidyverse"       "/usr/local/lib/R/site-library"

tinytex         "tinytex"         "/usr/local/lib/R/site-library"

tm              "tm"              "/usr/local/lib/R/site-library"

tmap            "tmap"            "/usr/local/lib/R/site-library"

tmaptools       "tmaptools"       "/usr/local/lib/R/site-library"

triebeard       "triebeard"       "/usr/local/lib/R/site-library"

txtplot         "txtplot"         "/usr/local/lib/R/site-library"

units           "units"           "/usr/local/lib/R/site-library"

urltools        "urltools"        "/usr/local/lib/R/site-library"

usethis         "usethis"         "/usr/local/lib/R/site-library"

utf8            "utf8"            "/usr/local/lib/R/site-library"

uuid            "uuid"            "/usr/local/lib/R/site-library"

V8              "V8"              "/usr/local/lib/R/site-library"

viridis         "viridis"         "/usr/local/lib/R/site-library"

viridisLite     "viridisLite"     "/usr/local/lib/R/site-library"

webshot         "webshot"         "/usr/local/lib/R/site-library"

whisker         "whisker"         "/usr/local/lib/R/site-library"

withr           "withr"           "/usr/local/lib/R/site-library"

wordcloud2      "wordcloud2"      "/usr/local/lib/R/site-library"

xfun            "xfun"            "/usr/local/lib/R/site-library"

XML             "XML"             "/usr/local/lib/R/site-library"

xml2            "xml2"            "/usr/local/lib/R/site-library"

xopen           "xopen"           "/usr/local/lib/R/site-library"

xtable          "xtable"          "/usr/local/lib/R/site-library"

xts             "xts"             "/usr/local/lib/R/site-library"

yaml            "yaml"            "/usr/local/lib/R/site-library"

zoo             "zoo"             "/usr/local/lib/R/site-library"

base            "base"            "/usr/lib/R/library"           

boot            "boot"            "/usr/lib/R/library"           

class           "class"           "/usr/lib/R/library"           

cluster         "cluster"         "/usr/lib/R/library"           

codetools       "codetools"       "/usr/lib/R/library"           

compiler        "compiler"        "/usr/lib/R/library"           

datasets        "datasets"        "/usr/lib/R/library"           

foreign         "foreign"         "/usr/lib/R/library"           

graphics        "graphics"        "/usr/lib/R/library"           

grDevices       "grDevices"       "/usr/lib/R/library"           

grid            "grid"            "/usr/lib/R/library"           

KernSmooth      "KernSmooth"      "/usr/lib/R/library"           

lattice         "lattice"         "/usr/lib/R/library"           

MASS            "MASS"            "/usr/lib/R/library"           

Matrix          "Matrix"          "/usr/lib/R/library"           

methods         "methods"         "/usr/lib/R/library"           

mgcv            "mgcv"            "/usr/lib/R/library"           

nlme            "nlme"            "/usr/lib/R/library"           

nnet            "nnet"            "/usr/lib/R/library"           

parallel        "parallel"        "/usr/lib/R/library"           

rpart           "rpart"           "/usr/lib/R/library"           

spatial         "spatial"         "/usr/lib/R/library"           

splines         "splines"         "/usr/lib/R/library"           

stats           "stats"           "/usr/lib/R/library"           

stats4          "stats4"          "/usr/lib/R/library"           

survival        "survival"        "/usr/lib/R/library"           

tcltk           "tcltk"           "/usr/lib/R/library"           

tools           "tools"           "/usr/lib/R/library"           

utils           "utils"           "/usr/lib/R/library"    




last revision 4/29/19 (updated group listing)