Creating Elevation Profiles from Rasters in ArcGIS

As with many things GIS, there are a couple tricks you need to know if you want to generate an elevation profile.  In my case, I'm using ArcGIS and its 3D Analyst extension to generate elevation profiles using SRTM DEM data (see http://srtm.csi.cgiar.org/SELECTION/inputCoord.asp for data download - I chose to get the data in "ArcINFO ASCII" format).

Getting the DEM into ArcGIS is a no-brainer.  Just download the data, unzip it, and use ArcCatalog to add it to map.

To use 3D Analyst to generate the profile, you have to enable the 3D Analyst Extension, and then add the 3D Analyst toolbar (right click in an empty area of your toolbar and click 3D Analyst).  Then you can draw a 3D polyline (click the button) along the path for which you want the profile.  Then, click the profile graph button () to generate the profile.  The data is saved where you've configured it under the 3D Analyst -> options menu ().

But I found an unpleasant surprise - my resulting dbf's (you can open them with excel) were reporting a stretched value - not the actual elevation value from the SRTM data.  So, here's the trick: you have to set the "Source Type" of your SRTM data to "Elevation".  To do that, right-click on your elevation raster in ArcCatalog, find the "Source Type" row, and switch it to "Elevation".  Now, re-add that DEM to your map, draw your 3D line again, generate your profile again, and voila - the dbf should now be reporting actual elevation and not a stretched value.

I didn't find this information anywhere else, so I'm hoping this might help some folks in the same situation I found myself in.

How to Keep Google from Profiling You

I just sent this email to my brother, who is interested in not being profiled by google.  I thought I would share it with you all as well.  This is a list of firefox extensions to consider if you want to keep google (and other bad boys) from following you everywhere.  Note that you don't need admin rights on your computer to install these.

To hide yourself from google, i would suggest doing a few things, which will be good for all your stuff, actually:

1. Install CS Lite Mod - https://addons.mozilla.org/en-US/firefox/addon/cslite-mod/
1. Set it to Deny Cookies Globally, and then allow cookies on a site-by-site basis.  You may occassionally have sites act shitty on you, in which case you can allow cookies for them that will persist forever, you can allow cookies per session (which allows that site to set cookies, but times them out after a while), or you can temporarily allow cookies, which turns that site back into a "no cookie site" after a while.
2. Allow google to set cookies on a per-session basis
   
2. Install Adblock Plus - https://addons.mozilla.org/en-US/firefox/addon/adblock-plus/?src=search
1. Enable the ad filters, tracking filters, and whatever else you want
2. Ad the following lines so that google won't follow you around the
   
3. Install Google Redirects Fixer - https://addons.mozilla.org/en-US/firefox/addon/google-no-tracking-url/?src=search
1. You know how when you click on a google search result, it first shows a google URL in your address bar, and then forwards you to the site?  Yeah, that's google tracking what you click.  This addon will make google search links link directly to the site you want, and not through a google redirect.
4. Install RefControl - https://addons.mozilla.org/en-US/firefox/addon/refcontrol/
1. When you click on a link, your browser sends a query to the site at that link location to send you the page.  Along with the query, it tells that site where you're coming from.  So, if you click on a link that sends you to google from a lifehacker page, google will know you're coming from lifehacker.  RefControl will allow you to stop your browser from sending that information, and it will allow you to turn it back on on a site-by-site basis (if stripping that ref info breaks a site for example - but it usually never does).
5. Install DoNotTrackMe - https://addons.mozilla.org/en-US/firefox/addon/donottrackplus/
1. I don't use this myself, but I probably should.  As I understand it, it keeps your browser from alerting tracking companies that you're on a particular page.  I don't use it because I have CSFire installed (see below).  But this might be better.
   
6. If you really want to fuck with google, install TrackMeNot - https://addons.mozilla.org/en-US/firefox/addon/trackmenot/
1. It will send random search strings to google, thereby obfuscating their user profile for you
7. You might think about installing NoScript - https://addons.mozilla.org/en-US/firefox/addon/noscript/
1. This will block lots of nefarious tracking shite.  But it will also break lots of sites.  So, I don't use it.  But you might think about it.
8. I use a thing called CsFire - https://addons.mozilla.org/en-US/firefox/addon/csfire/
1. It's kind of a pain, so i don't really recommend you use it.  But what it does is strip queries to third party content of cookies and the like.  For example, if a lifehacker page is showing an image from google images, CSFire will allow you to grab the image, but it won't allow google to see your google-related cookies, etc.  So, it will be harder for google to know that you're visiting the lifehacker page.

Ok, that's a lot.  But do it!

Feel free to (in fact, please do) share this with your colleagues or anyone else who wants to avoid the REAL big brother these days.  (ok, that ended up sounding a little militia-like paranoid, but you know what I mean).

resolving factor frustration in R

Factors in R have two components: their "index" (my term), which is just a vector of integers, and their "level", which is a vector of characters (or integers, though it's most helpful to think of them as just characters for now).  Matloff explains this well in his book The Art of R Programming, but I'll give my own quick example here.

    > f = factor(c(10,11,12,11))
    > f
    [1] 10 11 12 11
    Levels: 10 11 12

    > levels(f)
    [1] "10" "11" "12"

    > unclass(f)
    [1] 1 2 3 2
    attr(,"levels")
    [1] "10" "11" "12"

    > attributes(f)
    $levels
    [1] "10" "11" "12"

    $class
    [1] "factor"

You can see that R's internal representation of factors is just as I explained above.  That is, there is an integer index that is essentially a lookup table with integer indicies that point to the "labels" character vector.  The "labels" character vector attribute is the factor's second component.

While perhaps a bit confusing, this is all relatively straightforward.  Where factors can get really frustrating, however, is when different functions use different parts of the factor; some use the de-referenced "levels" of the factor, whereas others use the index values.  This problem is most evident in the as.character() and as.numeric() functions:

    > as.character(f)
    [1] "10" "11" "12" "11"

    > as.numeric(f)
    [1] 1 2 3 2

What's happening here?  as.character() is dereferencing the index and returning the character levels.  On the other hand, as.numeric() is returning the /index vector/ part of the factor.  This is probably not what you'd expect if you have numeric factors, such as we have above.  When you want to turn your factor into numbers, what you probably want to do is:

    > as.numeric(as.character(f))
    [1] 10 11 12 11

I've been using R for years now, and I finally took the time to look into this and resolve it.  I'm sure I'll run into more unexpected behaviors in the future, but I'm glad to have at least solved this aspect of factor behavior in R.  Do you have any other examples?  Feel free to share them in the comments!

I recently had to migrate to the 64-bit version of R due to the 4 GB memory limit imposed by the 32-bit version.  In doing so, the package I was using for importing Excel files (xlsReadWrite) was rendered incompatible.  Thus, I've moved over to XLConnect: http://cran.r-project.org/package=XLConnect

XLConnect looks very nice: it is fully featured, and is implemented as objects.  It uses a Java library to do the Excel imports, apparently.  This would seem like a detail, but it ends up adding a very important wrinkle.  According to the developer, Java's "ToString" function adds a ".0" to any values that look numeric.  So, even if you tell XLConnect's readWorksheet function to read the column in as a character data type (using the colTypes argument), it will add a ".0" to every value that looks to Java like a number.

Here's an example.  Let's make an excel document called XLConnect_test.xlsx:



Now, let's try to import it via XLConnect:

> require(XLConnect)
> xlc_df = readWorksheetFromFile(file="XLConnect_test.xlsx", sheet=1)
> xlc_df

    Col1 Col2
1    ABC  1.0
2  123.0  2.0
3 123ABC  ABC

> xlc_df = readWorksheetFromFile(file="XLConnect_test.xlsx", sheet=1, colTypes=c("character", "character"))
> xlc_df

    Col1 Col2
1    ABC  1.0
2  123.0  2.0
3 123ABC  ABC

Notice how XLConnect adds ".0" to each value, even though those columns are supposed to be character strings?  That's not good.  The nicely-communicative developer says that they're working on this for a future version.  In the meantime, here's the hack I've written so the rest of us can move forward:

xlc_dot_zero_hack <- function(df) {
    # looks for all numerics that had a ".0" added to them by XLConnect, and removes the ".0"
        return(
            as.data.frame( lapply( df, FUN = function(x) sub("^(\\d*)\\.0$", "\\1", x, perl=TRUE) ) )
        )
}

Just import your Excel data, and then run the function above on your imported dataframe.  Don't forget to reassign the result to a dataframe.  In our example, we would write:

> xlc_df = xlc_dot_zero_hack(xlc_df)
> xlc_df

    Col1 Col2
1    ABC    1
2    123    2
3 123ABC  ABC

Voilà!  Note that if your data has any strings that *actually* have a ".0" at the end, this hack will have unexpected results.  Specifically, it will remove *all* ".0"s that come after a string of digits.

I hope this might help others out there with the same conundrum!

Wednesday, September 5, 2012

Cross-References don't show up in Word's dialog box?

I was having an issue with a manuscript I'm putting together recently.  I was passing it back and forth with my advisor, and I was using Track Changes.  As I added some figures with captions, I then wanted to refer to those figures in the text via a cross-reference.

When I clicked on Insert -> Cross-Reference in Word 2010, the recently-inserted captions weren't in the list!



Ugh, *another* Word screwup with complex documents.

Well, after a lost day of screwing around with the problem (and sleeping on it), I came across the solution.

The problem seems to be that Word doesn't like you to insert your figures and captions as a "tracked change".  I went back and "accepted" all my insertions of figures and captions.  And, voila!  The figures now show up in the cross-reference dialog box.  Now, whenever I insert a new figure, I'll either turn off track changes for the moment, or just go back and accept the insertion.

Hope this helped somebody else!

pushd / popd for R

Hi Folks,

I haven't gotten around to putting my R code in nice objects yet. So, my code is procedural, and it's nasty. It sources one script to the next, processing data, analyzing data, etc. As the scripts get run around the codebase, they often change their working directory. So, I wanted access to a bash-equivalent pushd and popd. I'll run these at the beginning and end of each script.

These are not battle-tested yet, and there's really no error-checking or redundancy in them. So, use with care, and they'll probably evolve over time. Feel free to add to them in the comments!

Enjoy!

#  push getwd() strings into a FIFO vector

dir_fifo = c()

pushd <- function(cd) {

    # usage: pushd("directory to change to")
    dir_fifo = append(dir_fifo, getwd(), 0)
    assign("dir_fifo", dir_fifo, envir = .GlobalEnv)
    setwd(cd)
}

popd <- function() {

    # usage: popd()
    setwd(dir_fifo[1])
    dir_fifo = dir_fifo[-1]
    assign("dir_fifo", dir_fifo, envir = .GlobalEnv)
}

Do you miss seeing your Lenovo drivers ordered by release date?

IBM used to have a page where you could view all the drivers for your
system ordered by their release date. Since the IBM/Lenovo "System
Update" app has always been pretty unreliable, seeing the driver
releases ordered by date was always (for me, anyway) the best way to
make sure my system was up to date.

Lenovo, in all their wisdom, got rid of the ability to order drivers by
release dates. But, thankfully, there's a workaround!

Here's what you do. Go to the Lenovo support site and enter your system
type here: http://support.lenovo.com/en_US/downloads/default.page .
Then, click on the "Subscribe to driver updates" link in the upper-right
corner. That will take you to an RSS page. Don't subscribe to it if
you don't want to! But from here, you'll be able to see all the driver
releases ordered by date. Voilá!