PANNING!!

Solved the next problem in a single day! Had one of those moments when a logic error causes visibly amusing results on screen. On first attempt, I clicked, dragged, and the circle went ZOOMING out of the screen. I was able to retrieve it and could pan it around, as long as I was EXTREMELY cautious with my movements. It was pretty immediately obvious that each movement was being compared not to the previous mouse location, but to the original mouse location, so the farther away the mouse got from its mousedown location, the more extreme the movements of the circle.

Changed my logic entirely for the next go-round and set up the circles so that they all maintained the same "offset" that they had with the mouseLoc when mouseDown occurred. Worked fine and I was able to move on and implement with an array of circle objects shortly after.

Next step will be to try to bring all of the functionalities together. MIGHT try combining just zooming and panning before trying to put it together with everything else. In the meantime, I need to work on organizing my full data set for the final deployment.

ZOOMING!!

YES! While just a few lines of code, this was one of the more challenging problem-solving tasks that I've had to overcome in this project. Of course, as I mentioned in the previous post, there's always a tendency to push the hard tasks off until end, (and I suspect that the single remaining big task - panning - will prove to be even tougher). Jim called it the "smart friend" approach to programming - do the easier stuff and tell yourself, "I'll just have my smart friend do that other part." Of course, eventually you end up stepping up as the smart friend, but it keeps you from freaking out in the meantime.

Actually, all this one really took was taking pencil to paper to draw out what's actually happening when you PERCEIVE the field as zooming in or out, which is, of course, simply a repositioning of the elements in the field. Once the pictures were drawn, it all came together (still took four days of thinking). It also involves that percent increase, percent decrease logic - for example, if the market drops 33.3%, it has to rise 50% to get you back where you started.

Also implemented two buttons for zoom in and zoom out, but in a most rudimentary fashion. I'll really need to create a button class to handle these at some point.

Actually, there's one more trick to be pulled off here - the implementation of a svg (scalable vector graphic) version of the Texas map and making the map grow and shrink with zoom in and zoom out functions.

April 24 - v07

• Expanded search for maxPop and minPop functions to search across a series of columns that represent populations over a series of years by nesting two "for" statements. 
• Got it to read a progressive series of population values in successive columns. 
• Updates year and circle size on mouseup. There's a problem with text alignment in the year. It jumps left - MAYBE when you roll over a circle. 

This is getting to be a serious mental exercise at this point! I guess that's typically the case, as there's a tendency to solve the easy problems and leave the tough ones for later. As a result, each problem gets progressively more challenging!

April 23 - v06

GOOD bit of progress today!

• Went about correcting the representation of population using area by rewriting map function. DID include a minimum radius that would accommodate VERY small districts (working with down to 255 in my test data). 
• Enabled reading of dropout rate data from spreadsheet and representation as a pie slice. 
• Centered (and reduced size of) rollover text (in Circle object class - copied changes to my class library).
• Added calculation of minimum district size (employed MAX_INT).

April 22 catch up

A number of new breakthroughs since the last post. Have deployed my Circle class as an array of circles, which is perhaps the biggest hurdle to overcome to realize the complete visualization. I also diverted my attention briefly to create another object class (which, incidentally, went VERY smoothly) to create bullet graph as per Stephen Few. An exciting extension of this idea came at Stephen's workshop, when he mentioned that (1) no one commercial vendor had yet implemented a bullet graph and (2) he had posted a complete bullet graph specification on his website. I see this as an opportunity to extend my bullet graph object class to fully realize Stephen's specification and to share it with the Processing community.

But back to the Texas vis project...
I've learned some important lessons about the XlsReader library and the nature of xls spreadsheets. It took my QUITE some time to determine that CALCULATED VALUES in an excel spreadsheet are not recognized as ANY of the data primitive types (at least int, float, or string). While my code was running, I was getting a continuous stream of error messages indicating a type mismatch. I ultimately determined that it was the result of calculated results in the spreadsheet. When I replaced them with "hard-coded" values, the problem was resolved.

Next steps will be to read the dropout rate from the xls and represent it correctly. THEN, read a series of values over the course of years and move through them on mouseclicks.

Expanding the Circle object class

I've added some new methods to my Circle class - preparing it to serve as the "district" circles that will populate my Texas dropout vis. I've now implemented the arc and resizing of the arc. I've also implemented the rollover text that will give the name of the district, along with any other data that I wish to display. At the moment I seem to be having difficulty with anti-aliasing the text - look pretty pixelated, though I'm obviously controlling the font, because I CAN resize it.

Onward and upward!

Resurfacing

While there have been no posts for a number of weeks here, that doesn't translate to no activity - just a lack of diligence on the blogging front. This morning marked another good breakthrough, though, so I thought I'd try to catch up with a list of things that I've been working on.

1) in a big move toward my primary goal, I've drawn circles on a Texas map, locating them so that they are centered upon their district headquarters and sized according to the district's student population.

2) another BIG breakthrough was figuring out how to implement a library and getting the XLSReader library implemented and working. Now I can suck in all of my data from excel spreadsheets rather than formatting at CSV or TSV documents.

3) I also learned how to receive mouse input and created a little demo of this that causes an arc to resize within a circle. ArcGrowth02 is the demo.

4) Also created a bit of a button functionality - changes colors on rollover and then responds to a mouseup within. MouseInRect demonstrates this functionality.

5) But the BIGGEST breakthrough was creating and successfully implementing an object class. The class is pretty simple at the moment - just draws and resizes a circle - but it will be the basis of all of the circles that are drawn as part of the project. I'll probably also create a button class, since everybody seems to need one of those.

Well, it was a good exercise - ;-)

SO, in my Feb 18 post, I commented:

Now the question is how to convert longitude and latitude to positions on the image - not a DIFFICULT problem, but one that took some thought. I think I worked it out during a long drive this weekend. Will try to implement tomorrow. 

I spent a good hour or two earlier this AM getting it implemented. Then I moved forward in my book and about 6 pages later came upon the introduction of the map() function, which "converts numbers from one range to another," exactly my task. The author goes on to say "A lot of visualization problems revolve around mapping data from one range to another, so the map() method is used frequently."

Oh well. As I said - it was a good exercise. I'll go back and modify my code to take advantage of the map() function and make sure that I can make it work.

note - And indeed, it did work. And in the process I discovered that it can also reverse the axis in the scaling process - scaling the range (36 - 25) to (0 - 545).

Too long since I've done any coding :-)

So, back at the TX vis after an intense week on other things. Rather than start out right off including the code that converts lat and long positions into x and y on the image, I decided to do the conversion in Excel. Got that done quickly enough, then started trying to draw the dots onto the map. THIS went on for a couple of hours, with one error message after another - and all of them new to me. I moved data files back and forth from the example that I was copying. Moved map files back and forth. And FINALLY, as I worked through the two programs, side by side, line by line, I recalled a fundamental fact about computers and programming. You start counting from ZERO. So if you want to retrieve the elements in the second and third columns of a data set, you call them columns 1 and 2. Now the "out of bounds" error messages make more sense. I now have a TX map with about 350 dots drawn all over it. Need to get the lats/longs for the other 800 or so. 

Positioning locations on the map

So the problems are falling one at a time on this - (1) have learned how to draw a map (or any image, for that matter) to the window, (2) have acquired a list of all of the districts in TX (all of the schools, for that matter, which I was able to parse down to a list of districts), (3) have figured out how to convert the addresses to longitude and latitude. Now the question is how to convert longitude and latitude to positions on the image - not a DIFFICULT problem, but one that took some thought. I think I worked it out during a long drive this weekend. Will try to implement tomorrow.

Starting the Texas Ed Vis

So an early exercise in the Processing book is to display a map of the US, then draw a circle at the center of each state, reading the position data from a file. This aligns very much with my visualization project, which will involve drawing circles at the location of each TX school district.

Downloaded a TEA data file containing info on every school in the state. Extracted from it a list of every district in the state. Contained addresses (sometimes a PO box) for each district. Went looking for a way to convert addresses into longitude and latitude.

Found and tried a couple of things, but ultimately landed (surprise, surprise) at Google Code and the Google Maps API. One of the Web Services that they offer is a Geocoding API. Geocoding is the process of converting an address to latitude and longitude. Now I'm in WELL over my head, but I learn that you can send a request to the API in the form of an http (statement?). The statement that I ultimately sent looked like this:

http://maps.googleapis.com/maps/api/geocode/json?address=2211+Lawnmont+Ave,+Austin,+TX&sensor=false

and it delivered a result that included something like this:

"formatted_address" : "2211 Lawnmont Ave, Austin, TX 78756, USA",
"geometry" : {
"location" : {
"lat" : 30.3268790,
"lng" : -97.74253999999999
}

So I'd managed to execute the task for one address. Now the question is how to do it for the 1250 addresses I have for Texas school districts. Time to consult w/ David Kim...

FYI - info on Google Geocoding API is here:

http://code.google.com/apis/maps/documentation/geocoding/

First Processing Code

Got through most of the Ch 01 coding examples this eve. Changed the NEAR burnt orange background that was used in the examples to a TRUE burnt orange - RGB = 199, 91, 18.

Tested each of the export functions - for web deployment or as a standalone application. Tested the standalone app for Mac (not windows or linux). Also tested the saveFrame function and created more than 5000 identical TIFF files before I was able to stop the app. Also discovered that Processing is case sensitive!!!

I find it interesting that Processing can interact with a variety of renderers depending on need.

Visualizing Data - Ch 01

Read Chapter 1 of Visualizing Data, by Ben Fry.

"One of the most important skills in understanding data is asking good questions." p 4

Seven steps to answering a question with data:

Acquire - get the data
Parse - organize the data
Filter - remove data that is not of interest
Mine - use statistical or data mining techniques to discern patterns in the data
Represent - choose a visual model
Refine - improve upon the model to make it clearer and more engaging
Interact - provides methods to control what features are visible

Information visualization refers to the process of representing data that is primarily numeric or symbolic in nature.

Illuminating the Path (and Processing)

Downloaded and installed Processing (again, since my catastrophic HD crash of a few months ago).

Started to read Illuminating the Path: The Research and Development Agenda for Visual Analytics. The first thing that comes to my attention is that the goal of the volume is to develop an agenda that is SPECIFICALLY focused on vis analytics for the purpose of enhancing homeland security. I'm, then, forced to wonder how appropriate this particular agenda is for MY stated purpose - applying vis analytics to educational data. Whatever the case, Kelly Gaither's response, when I told her that I was considering this as one of the central texts for my study, was that this is ESSENTIAL reading. With that endorsement, I begin my review of this volume.

How about this statement of need:
"New methods are required that will allow the analyst to examine this massive, multi-dimensional, multi-source, time-varying information stream to make decisions in a time critical manner."


Here's a quote from page 5 that addresses my earlier stated concern:
"Although the agenda described herein is focused specifically on meeting homeland security challenges, the new capabilities created will have an impact on a wide variety of fields ranging from business to scientific research, in which understanding complex and dynamic information is important."

From page 9:

"interactions must be adaptable for use in platforms ranging from the large displays in emergency management control rooms to field-deployable handheld devices"

So far...

I've decided to track my activities for my independent study course for the spring of 2012 via this blog. I've been at it for a few weeks, so this will be a bit of a catch-up.

I'm going to focus on three related topics:

• Educational data mining
• Visual analytics
• Information visualization

I've identified a text related to each and have begun reading the text on EDM. Early chapters are surveying techniques of (educational) data mining. It is worth noting that the second chapter of the book (chapter 1 was the Intro) was entitled Visualization in Educational Environments. 

Chapter 3 is a useful review of the basics of statistical analysis. 

Chapter 4 describes the Pittsburgh Science of Learning Center (PSLC) DataShop. This resource provides (1) an environment for data storage and dissemination that aligns with the demands of the NSF data management plan requirement and (2) a ready repository of educational datasets. 

Chapter 5, which I'm currently reading, addresses classification as a data mining process. 

I've also started to develop a website which will serve as a repository for my learning, located here:

https://sites.google.com/a/utexas.edu/edvis/

Falling asleep, so I'll publish this one and pick it up tomorrow.