Comparing two point datasets for error checking

Let's say you have a set of points given to you that represent addresses of snake farms[insert object here].  Also you've been given the raw address data and you want to compare the quality of the geocoded point file by geocoding the points yourself using the best road data you can find.  Then you might want to know the distances between all the "identical" points to check your data for errors.

As an exercise I generated 2 random sets of points (100 points each) and arbitrarily joined them based on an ID field of 0-99.  One way to figure out the distance between points that should be the same in each file(although in this case NONE will be the same because both were supposed to be random point sets).  Generate an X and Y field for each point in each dataset... (x1,y1 and x2,y2)  Then join the two datasets and create a new field called perhaps "distance" and using the field calculator in arcgis use sqr((x1-x2)^2 + (y1-y2)^2).  Your results will be in whatever unit your coordinate system is measuring in.  I used NAD83 Stateplane Texas North Central for the example.

Another method involves using the ET Geowizards free tool "point to polyline". To do this you would take both files and copy both sets of points into a 3rd shapefile.  This method is nice because you get a line connecting points with identical addresses and could color ramp them to look for ones with large differences in distances.

2 point datasets with lines connecting related points 

Mapbox, TileMill and TileStream

Tilemill is pretty awesome.  Quickly you can prepare data as shapefiles, zip them up and then theme them using Tilemill(you need to install it on OSX or linux, I'm using Ubuntu 10.10).  Once you have it installed you can very quickly theme a map using CSS like style code, then export your map into an MBtiles format and start serving it up using Tilestream.  I built both from source after mirroring the projects from their git repositories.
This screen shot is of the first map I am working on.  It uses some of the data that Mapbox team has stored on their s3 amazon account and a shapefile of the latest TEA district boundaries.  I'm working on making it more data rich. There are ways to use your custom MBtiles data with Google Maps API and OpenStreetMap data. 
Simple "MSS" style code to generate most of the above map, there are two other stylesheets being used. One for labeling(with various definitions for different zoom levels) and one to color all the school districts contrasting colors. The colors can be defined by attribute data read from the shapefile DBF.

Map {
  background-color: #fff;
}
 #states {
  line-color: #002bff;
  line-width: 1;
}
#lakes {
  polygon-fill:@water;
  line-color:#002bff;
  line-width:0.4;
  line-opacity:0.4;
}
#world {
  ::outline {
    line-color: #000;
    line-width: 4;
  }
  line-color: #fff;
  line-width: 2;
}
#districts {
  line-color: #rgba(0,0,0,0.75);
  line-width: 1;

Development Seed has designed and released for free an iPad Mapbox App. Once you've got map style sheets down using carto in tilemill you generate the mbtiles file and copy it over to your iPad and viola instant custom map works very fast even with HUGE datasets, I tested it with a shapefile with over 300,000 polygon and it worked very well and was stable too.  This is due to the data being converted to images and sliced into tilesets, therefore it doesn't have to render the millions of points needed to make all those polygons.

Code!

I've used a few snippets of code from http://gis.utah.gov/code
The last one I used was the spatial join points and polygons without creating an extra(trash) dataset.

Fuzzy String Matching

*Update* checkout Google Refine for cleaning dirty data.

I wanted to generate numbers that represented a sort of "sameness" or "matchiness" of two strings, that's how I was thinking about it.  This led me to learning a little about fuzzy string matching using a method called Levenshtein. I didn't need to understand how to write the algorithms to do this I just wanted to use the tools.
ArcGIS has adopted Python as it's scripting language.  I downloaded compiled and installed a module called pylevenshtein which will compute the levenshtein edit distance as well as other methods of comparing two strings.  (had to install microsoft studio 2008 express it's free).  Once this is setup it's rather easy to implement the various string comparison algorithms in the field calculator.
I'm now figuring out which function to use or combination of functions to use to see if it's any better than my current method.  The Jaro Distance is looking good.

update:
I've decided that the method I was already using was best for me, perhaps I can write a script to automate some of the steps involved.
Step 1  select where old name <> new name
Step 2 export this selection
Step 3 delete all unnecessary fields(to reduce dataset size and ease the manual review steps)
Step 4 create two new txt fields 4 characters in length
Step 5 populate new txt fields with left(old name/new name, 4)
Step 6 select where left4_old=left4_new
Step 7 scan selected parcels > X acres(depending on size of dataset)
Step 8 delete the manually corrected selection of left4_old=left4_new

On spot checking this method the results are very good.  I found it difficult to interpret the various edit distance algorithms from the pylevenshtein package into something meaningful that would improve the accuracy and speed in developing change data.  The exercise was to try and find a better/faster way and I ended up going back to my original method.  Maybe it's because my method is easier to comprehend.

One goal for generating the data is to highlight ownership changes over time on a map. This could also be useful when looking at older subdivisions and gauging the regeneration rate.

NIR to fake NC

You can fake natural color imagery using near infrared imagery by playing with the 3 (or more) bands.
Here I took some 2010 NAIP imagery, clipped it to the area of interest and set the RGB bands to
Red=Band 2
Green=Band 3
Blue=Band 3

Remote Sensing

Here's a link to a tutorial with a lot of history published by someone at NASA.

Supervised Image Classification

Related to the previous post about vacant land inventory, this is an exercise in delineating land use types using image analysis.  Previously I used parcel vector data and the linked attribute data looking for parcels over a certain size that had a low or zero improvement value to locate vacant land.

First I've got a 2010 3 band CIR image to work with.  I've started with 5 basic classes(water, forested, pasture, bare earth, pavement) and defined several training data regions for each class manually using a polygon lasso tool.  Waiting for processing results....

Initial results are not great.  I seem to have classified the wooded areas very well, but the road/water/bare earth probably needs more manual definition placed with better precision.  Next to try this on a smaller dataset so it doesn't take days to process and see how good/bad the results are.