Processing: How to Add CSV Data to Pretty Maps

header

The internet has a shit-ton of map data. I didn’t realize it until I started skimming through HTML on sites like Craigslist and Yelp. Yes, by the way, I am that weirdo who sits in coffee shops on Saturday mornings muttering at tags I don’t understand.

The point is, I want to be able to play with that data in Processing. The first step is just getting that information into the sketch and mapping it properly. That’s what I’m doing today… I acknowledge that I get fixated on projects with very few practical applications. It’s possible I need to get out more.

Suggested Materials:

Google Earth Pro:
Required for the background image and to double check coordinates. ArcGIS can be used too, but if I could afford a subscription, I wouldn’t be doing this in Processing.
Processing:
Required to actually pull all the information together and display it.
Airbnb Scraper:
Required for the .csv file of latitude and longitude. I’m working on scraping more data off other websites, but since I’ve already got the Airbnb script, I’m taking advantage.

Step 1: Import the Image

proccoors

To get the background image, I took a screenshot in Google Earth, scaled it down to a reasonable size, and saved it as SFMap.png. Then in my Processing script, I hit Sketch, Add File, and picked out the image.The script to add it looks like this.

how-to-make-an-image

Step 2: Import the Data

The .csv file is easier to handle if it has named headers. My file has a  Lat, Long and Price column.how-to-load-a-table

After the table is loaded, tablename.getRowCount() will return the number of rows. To retrieve the values, tablename.getFloat(Row#,”ColumnName”) is a safe way to return the latitude and longitude without rounding them into oblivion.

Step 3: Scale Down the Coordinates

Let’s talk about where I live for a moment. I’m in North America. Specific, right? Northern hemisphere. Western side of the world.That translates to a positive latitude and a negative longitude. For anyone else out there who was asleep during 6th grade geography, latitude corresponds to the y-axis. Longitude is the x-axis. Don’t get those confused. You know what happens to people who get those confused? They stay up until 3 in the morning screaming incoherently at Google Earth Pro because it keeps sending points to Antarctica.

latitude

We know that the image is 1103 pixels wide which represents approximately 0.238952636719006 degrees because that’s the difference between my western and eastern boundaries in Google Earth. To get the X value, we can get the difference between the western boundary (where x=0) and the longitude of the point. Scaling then just requires cross-multiplying and dividing. Keep in mind, the difference between a negative and a negative is negative (yes, to answer your question, I’ve been exceptionally dim this week and forgot to account for that the first several test runs). To get it to display on the Processing grid, you’ll have to turn the longitude array positive.

math

Almost the same process applies for the latitude except the image is 302 pixels tall and that represents 0.0516437133917975 degrees. Since Processing places y=0 at the top of the image, when we scale the length, we’ll have to get the difference between the northern boundary and the point’s latitude. Then cross multiply and divide.

Step 4: Create an Array

I know what you’re thinking. There’s no way in hell I’m going to sit here and run the math on every one of those points. This situation has LOOP written across it in giant neon bubble letters. So let’s pull each coordinate from the table, run the math, and shove the scaled values into a separate array.

Yeah. Ummm. I think there’s a more condensed way to do this, but I like the 3 step system. Declare it, create it, and then assign values.

how-to-make-an-array

Step 3 is where we throw in the loop to do the scaling before assigning the values.

Step 5: Draw Stuff

I went with a simple ellipse filled with white (255,255,255). You’ll notice below that my color parameters include a fourth value—that’s the alpha transparency. I like my dots with a little haze around them, so I added several ellipses with different transparencies. That’s completely customizable depending on how you want to color code.

The Code

PImage img;
Table table;
float[] lat;
float[] lon;

void setup() {
size(1103, 302);
img = loadImage("SFMap.png");
//Load points from the Airbnb csv
table = loadTable("Airbnb.csv", "header");
//Create arrays from longitude and latitude
lat = new float[table.getRowCount()];
lon = new float[table.getRowCount()];
//Start a for loop to scale down X and Y values into coordinate arrays
//This bit needs adjusting if you import a map at a different scale
for (int i = 0; i < table.getRowCount(); i++) {
lat[i]= ((1103*(-122.560608677635-(table.getFloat(i, "Lat"))))/0.238952636719006);
lon[i]= ((302*(37.912159028101655-(table.getFloat(i, "Long"))))/0.0516437133917975);
}
}

void draw(){
image(img, 0,0);
//Make everything borderless
noStroke();
//Run a loop the length of the lat array, starting at 0 and increasing by 1
for (int i = 0; i < lat.length; i++) {
//Make a 9 pixel circle with a 20/255 transparency value (really transparent)
fill(255,255,255,20);
ellipse(-lat[i], lon[i],9,9);
//Make a 6 pixel circle with a 80/255 transparency value (transparent-ish)
fill(255,255,255,80);
ellipse(-lat[i], lon[i],6,6);
//Make a 3 pixel circle with a 155/255 transparency value (kinda transparent)
fill(255,255,255,155);
ellipse(-lat[i], lon[i],3,3);
//Make a 1 pixel circle with a 255 transparency value (not transparent)
fill(255,255,255,255);
ellipse(-lat[i], lon[i],1,1);
}
}
Advertisements
Tagged , , , , , , ,

4 thoughts on “Processing: How to Add CSV Data to Pretty Maps

    • AT LAST. And it only took 6 short hours of Daniel Shiffman tutorials, 20 cups of coffee and 1 mild panic attack when my computer crashed and wouldn’t turn back on again. That wasn’t Processing though. Battery went out.

      Liked by 1 person

      • ARJ says:

        haha i knew shiffman was in the air. btw tag your posts meaning put a tag : processing language
        2nd a friend would like to recommend your blog (another wp blogger) but would not do so bec of ÷__#÷%^×3 words (can you assure none in your future posts?)

        Liked by 1 person

      • What you mean a tag for processing? Processing is tagged? Confusion???

        Sadly, the swearing will definitely have to stay. I make a conscious effort not to when I comment on other blogs, but it’s too much effort to self censor my own. And the rage ranting is cathartic.

        Tell ’em thanks for thinking of recommending GifGuide, but I wouldn’t suggest it to anyone who doesn’t engage in cursing on a regular basis. I actually had a great conversation about this with a blogger called ECG who gave me the BEST idea for a profanity scraper. It’s proving really difficult to debug though (I think I need to switch to Python rather than AHK). Probably focusing on that next week.

        Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: