Invalid GeoJSON

[kdeloach]

Some points result in a shapefile that yields invalid JSON when converted by ogr2ogr.

For the following request:

curl http://localhost:5001/rwd/39.94834695071714/-75.15013754367828

The result GeoJSON:

> cat f9c3a5a8325c4700b0b4e98c30d29b17.json
{
"type": "FeatureCollection",

"features": [
{ "type": "Feature", "properties": { "GRIDCODE": 1, "Area": 0.024903, "BasinLen": 0.243626, "Strord": 1.000000, "StrLen": 0.000000, "DrnDen": 0.000000, "AvgOLF": inf, "BR": 2.728631, "RR": 0.011200, "Avgslp": 0.024175 }, "geometry": { "type": "Polygon", "coordinates": [ [ [ -75.150740740744652, 39.949814814794564 ], [ -75.150185185186714, 39.949814814794564 ], [ -75.150185185186714, 39.949629629608701 ], [ -75.149814814814761, 39.949537037015766 ], [ -75.149907407407753, 39.948981481458155 ], [ -75.150092592593722, 39.948796296272285 ], [ -75.150000000000745, 39.948425925900544 ], [ -75.150370370372698, 39.948333333307609 ], [ -75.15064814815166, 39.947962962935868 ], [ -75.151203703709598, 39.947962962935868 ], [ -75.151851851860513, 39.948611111086414 ], [ -75.151851851860513, 39.94925925923696 ], [ -75.151018518523614, 39.949537037015766 ], [ -75.150740740744652, 39.949814814794564 ] ] ] } }
]
}

Part of the result is invalid JSON ("AvgOLF": inf). Because of this, a valid JSON object cannot be decoded.

> cat f9c3a5a8325c4700b0b4e98c30d29b17.json | python -m json.tool
No JSON object could be decoded

The following command can be executed against shapefiles generated by RWD to reproduce this issue:

ogr2ogr output.json New_Point_Watershed.shp -f GeoJSON

[mmcfarland]

In the shapefile (as viewed through QGIS), what is the actual value for AvgOLF?

[mmcfarland]

Just checked and inf is the actual value, it's just not encoding the string properly.

[mmcfarland]

According to the details on the ogr GeoJSON spec, it should encode strings correctly:

Schema detection will recognized fields of type String, Integer, Real, StringList, IntegerList and RealList. Starting with GDAL 2.0, Integer(Boolean), Date, Time and DateTime fields are also recognized.

It also says we can turn off attribute generation at all by using an environment variable:

It is possible to tell the driver to not to process attributes by setting environment variable ATTRIBUTES_SKIP=YES. Default behavior is to preserve all attributes (as an union, see previous paragraph), what is equal to setting ATTRIBUTES_SKIP=NO.

However, we need the attributes for the input point shapefile, thought not the watershed. Too bad this feature is not part of the command input, which would let us move past this for now.

[mmcfarland]

Sorry, just clicked what the slack chatter was about regarding inf. Inf is also a phrase (D-Infinity) for flow direction, and though it was a string of the method used, but doing an ogrinfo -al on the shapefile shows that it's a Real number.

Area: Real (24.15)
BasinLen: Real (24.15)
Strord: Real (24.15)
StrLen: Real (24.15)
DrnDen: Real (24.15)
AvgOLF: Real (24.15)
BR: Real (24.15)
RR: Real (24.15)
Avgslp: Real (24.15)
OGRFeature(New_Point_Watershed):0
  GRIDCODE (Integer) = 1
  Area (Real) = 0.023124530000000
  BasinLen (Real) = 0.352130615234375
  Strord (Real) = 1.000000000000000
  StrLen (Real) = 0.000000000000000
  DrnDen (Real) = 0.000000000000000
  AvgOLF (Real) = inf
  BR (Real) = 7.183765411376953
  RR (Real) = 0.020400853827596
  Avgslp (Real) = 0.031426668167114

[kdeloach]

Here's the output from my test shapefile:

root@9b628314dc31:/tmp/tmprerrE0# ogrinfo -al New_Point_Watershed.shp
INFO: Open of `New_Point_Watershed.shp'
      using driver `ESRI Shapefile' successful.

Layer name: New_Point_Watershed
Geometry: Polygon
Feature Count: 1
Extent: (-75.151852, 39.947963) - (-75.149815, 39.949815)
Layer SRS WKT:
GEOGCS["GCS_North_American_1983",
    DATUM["North_American_Datum_1983",
        SPHEROID["GRS_1980",6378137,298.257222101]],
    PRIMEM["Greenwich",0],
    UNIT["Degree",0.017453292519943295]]
GRIDCODE: Integer (10.0)
Area: Real (24.15)
BasinLen: Real (24.15)
Strord: Real (24.15)
StrLen: Real (24.15)
DrnDen: Real (24.15)
AvgOLF: Real (24.15)
BR: Real (24.15)
RR: Real (24.15)
Avgslp: Real (24.15)
OGRFeature(New_Point_Watershed):0
  GRIDCODE (Integer) = 1
  Area (Real) = 0.024903340000000
  BasinLen (Real) = 0.243626403808594
  Strord (Real) = 1.000000000000000
  StrLen (Real) = 0.000000000000000
  DrnDen (Real) = 0.000000000000000
  AvgOLF (Real) = inf
  BR (Real) = 2.728631496429443
  RR (Real) = 0.011200064793229
  Avgslp (Real) = 0.024175461381674
  POLYGON ((-75.150740740744652 39.949814814794564,-75.150185185186714 39.949814814794564,-75.150185185186714 39.949629629608701,-75.149907407407753 39.949629629608701,-75.149907407407753 39.949537037015766,-75.149814814814761 39.949537037015766,-75.149814814814761 39.949444444422831,-75.149907407407753 39.949444444422831,-75.149907407407753 39.948981481458155,-75.150000000000745 39.948981481458155,-75.150000000000745 39.948796296272285,-75.150092592593722 39.948796296272285,-75.150092592593722 39.94870370367935,-75.150000000000745 39.94870370367935,-75.150000000000745 39.948425925900544,-75.150092592593722 39.948425925900544,-75.150092592593722 39.948333333307609,-75.150185185186714 39.948333333307609,-75.150370370372698 39.948333333307609,-75.150370370372698 39.948240740714674,-75.15046296296569 39.948240740714674,-75.15046296296569 39.948148148121739,-75.150555555558668 39.948148148121739,-75.150555555558668 39.948055555528803,-75.15064814815166 39.948055555528803,-75.15064814815166 39.947962962935868,-75.151203703709598 39.947962962935868,-75.151203703709598 39.948055555528803,-75.15129629630259 39.948055555528803,-75.15129629630259 39.948148148121739,-75.151388888895568 39.948148148121739,-75.151388888895568 39.948240740714674,-75.15148148148856 39.948240740714674,-75.15148148148856 39.948333333307609,-75.151574074081552 39.948333333307609,-75.151574074081552 39.948425925900544,-75.151666666674544 39.948425925900544,-75.151666666674544 39.948518518493479,-75.151759259267536 39.948518518493479,-75.151759259267536 39.948611111086414,-75.151851851860513 39.948611111086414,-75.151851851860513 39.94925925923696,-75.151759259267536 39.94925925923696,-75.151666666674544 39.94925925923696,-75.151666666674544 39.949351851829896,-75.151574074081552 39.949351851829896,-75.151574074081552 39.949444444422831,-75.15129629630259 39.949444444422831,-75.151203703709598 39.949444444422831,-75.151203703709598 39.949537037015766,-75.151111111116606 39.949537037015766,-75.151018518523614 39.949537037015766,-75.151018518523614 39.949629629608701,-75.150925925930622 39.949629629608701,-75.150833333337644 39.949629629608701,-75.150833333337644 39.949722222201629,-75.150740740744652 39.949722222201629,-75.150740740744652 39.949814814794564))

[mmcfarland]

A new upstream commit to the rwd python file should prevent inf values for AvgOLF:

nazmussazib@ba6135b

[kdeloach]

Great! I'll add it now.

[mmcfarland]

Just note that that commit was added to upstream master 😃

[kdeloach]

Fixed by #28

[kdeloach]

I encountered this issue again today. If the result contains "Infinity" then JSON.parse raises a SyntaxError exception.

Here's an example where AvgOLF_km value is "Infinity". To resolve this, "null" should be used instead of "inf" or "Infinity".

{"watershed":{  
   "crs":{  
      "type":"name",
      "properties":{  
         "name":"urn:ogc:def:crs:OGC:1.3:CRS84"
      }
   },
   "type":"FeatureCollection",
   "features":[  
      {  
         "geometry":{  
            "type":"Polygon",
            "coordinates":[  
               [  
                  [  
                     -77.41784368386455,
                     40.92746528366919
                  ],
                  [  
                     -77.4167867534997,
                     40.927309126316004
                  ],
                  [  
                     -77.41578950278605,
                     40.922792034707044
                  ],
                  [  
                     -77.41423410368498,
                     40.92037731437197
                  ],
                  [  
                     -77.4240873598042,
                     40.9177365216759
                  ],
                  [  
                     -77.42492429113086,
                     40.914582919550014
                  ],
                  [  
                     -77.42632973029681,
                     40.91342500457668
                  ],
                  [  
                     -77.42541246668975,
                     40.91274331939958
                  ],
                  [  
                     -77.42568785200612,
                     40.91032609227276
                  ],
                  [  
                     -77.42793349458836,
                     40.907380594909434
                  ],
                  [  
                     -77.42567361028837,
                     40.90486199737555
                  ],
                  [  
                     -77.42622785372127,
                     40.90139359231112
                  ],
                  [  
                     -77.42861984741049,
                     40.90065453823096
                  ],
                  [  
                     -77.43298550831031,
                     40.900753106067185
                  ],
                  [  
                     -77.43397982428138,
                     40.903903964115806
                  ],
                  [  
                     -77.4355317842725,
                     40.904952390572454
                  ],
                  [  
                     -77.43778443439614,
                     40.90473876980789
                  ],
                  [  
                     -77.43863203202767,
                     40.90568316731633
                  ],
                  [  
                     -77.43800487654532,
                     40.908048419606594
                  ],
                  [  
                     -77.43899188374998,
                     40.90846720505638
                  ],
                  [  
                     -77.43949113154359,
                     40.91072563335188
                  ],
                  [  
                     -77.44175169354772,
                     40.91324399732332
                  ],
                  [  
                     -77.43893746400099,
                     40.914194121596765
                  ],
                  [  
                     -77.43865870446548,
                     40.91524535239801
                  ],
                  [  
                     -77.44063678766132,
                     40.917448943585164
                  ],
                  [  
                     -77.44035804052521,
                     40.9185001808101
                  ],
                  [  
                     -77.44170136275717,
                     40.92033697290889
                  ],
                  [  
                     -77.41784368386455,
                     40.92746528366919
                  ]
               ]
            ]
         },
         "type":"Feature",
         "properties":{  
            "Strord":1.0,
            "BasinL_km":3.94404248046875,
            "DrnDen_kmi":0.0,
            "StrLen_km":0.0,
            "AvgOLF_km":Infinity,
            "Area_km2":3.3741,
            "GRIDCODE":1
         }
      }
   ]
},
"input_pt":{  
   "crs":{  
      "type":"name",
      "properties":{  
         "name":"urn:ogc:def:crs:OGC:1.3:CRS84"
      }
   },
   "type":"FeatureCollection",
   "features":[  
      {  
         "geometry":{  
            "type":"Point",
            "coordinates":[  
               -77.43404766042885,
               40.90295815186594
            ]
         },
         "type":"Feature",
         "properties":{  
            "Lat":40.913513,
            "Lon":-77.431641,
            "Dist_moved":46,
            "DistStr_m":1640.954345703125,
            "ID":1
         }
      }
   ]
}
}

[dtarb]

This is a problem in RWD that occurs due to an untrapped divide by 0 when clicking on a grid cell that is the upstream end of a stream (or on a point that snaps down to that point).

The most recent pull request will not generate this as this part of the code is deactivated (commented out) as it does not report this attribute. If we decide we want to bring these watershed attributes back then this will need to be corrected.

The longer hydrologic explanation (which you may not want to hear) is that at the upstream point on a stream the length of streams upstream is zero, and drainage density is 0 so overland flow length which is calculated as 1/(2 * drainage density) is infinity. Under these conditions this formula becomes a poor estimate of average overland flow length. I would recommend changing this to actual average overland flow length which can be computed from the DEM if we bring this back, but this will require additional preprocessing of the data.

As a heads up this may be a problem in DRB implementation too (though if it is not causing trouble you have likely trapped for it already).