Rewrite Spark merge in new SQL interface

Since we have structured data that's right down PySpark SQL's alley, we should make use of its power and rewrite the merging using their DataFrame.