one more pass at branch/fork unification?

[emiliom]

We've done a pretty significant amount of code reorganization and cleanup since mid November, and that's all now pretty stable in the master branch. But there are two possible additional loose ends that we should tackle for merging into master:

1. @denvaar, your commits to the setup branch since late November or so. It looks like you've fixed several miscellaneous bugs. It'd be great to merge those into master, the delete the setup branch.
2. @cdesyoun and @valentinedwv, is the ODM2REST API using the current odm2api master, or does it rely on your own fork? If it's using your fork, let's try to identify bug fixes and feature additions you've made, and merge them into master. If you are using master, that'll be great to confirm, too! I suspect you must be using your own fork, because master doesn't seem to have any methods for Measurements result types, but ODM2REST API does handle that result type in my "Marchantaria" use case.

Thanks!

[denvaar]

@emiliom merging the setup branch into master is something I will be doing within the next week or two. I have made a lot of additions and a few changes, so I am trying to double check with @sreeder about some things.

[emiliom]

Thanks, @denvaar!

[cdesyoun]

Emilio, for odm2 rest api, you have put some comments here, ODM2/ODM2RESTfulWebServices#1. This work is still under development server to get your review. If you think it is OK, I will update current version on github, called the production version from the development version.
Also, I used odm2api package integrating with odm2 api and tested it on production version, http://sis-devel.cloudapp.net/docs/. After changing it, the value for "FeatureGeometry" in "SamplingFeatures" is not converted to human-readable text, for example, "0101000020E6100000DAE6C6F484014EC0D5CF9B8A54580AC0" instead of "POINT(-55.51212 -1.94778)". My development server (http://sis-devel.cloudapp.net:9090/docs/) is still using my own fork. It shows readable text for feature geometry. The reason is used for geoalchemy 1, not for geoalchemy 2?

[emiliom]

Thanks, Choonhan. Yeah, I'm so sorry that I never followed up with comments and reviews on your latest version of the ODM2REST API! This has been on hold for so long, that I think you should go ahead and push to production. It's all still experimental anyway. I do want to get to it sooner rather than later, but right now I can't say when.

Regarding this comment:

After changing it, the value for "FeatureGeometry" in "SamplingFeatures" is not converted to human-readable text, for example, "0101000020E6100000DAE6C6F484014EC0D5CF9B8A54580AC0" instead of "POINT(-55.51212 -1.94778)". My development server (http://sis-devel.cloudapp.net:9090/docs/) is still using my own fork. It shows readable text for feature geometry. The reason is used for geoalchemy 1, not for geoalchemy 2?

Thanks for those details. That's exactly the kind of information we need to get to the point where the odm2api package is working for everyone. Let's try to identify the source of the problem next week. I will work with you to try to fix it.

Is that the only change you've made in your fork?? If so, we're in great shape!

[cdesyoun]

Thanks, Emilio. I will update it integrating with odm2api on github.
I think Stephanie has updated most of my issues in ODM2 rest api.
When comparing it with newly version, customized Geometry class below in odm2 api was used.

from apiCustomType import Geometry

this import (https://github.com/ODM2/ODM2PythonAPI/blob/master/odm2api/ODM2/apiCustomType.py) tells how to compile and save the geometry data.

[emiliom]

(note: I'm editing/correcting what I wrote here a minute ago...)

@cdesyoun, are you saying your issue is fixed in the current master branch?

To ensure you've installed the correct geoalchemy1 package (our ODM2 fork), please see the new installation instructions on README.md. If you don't want to use conda to install the dependencies, try installing odm2api with pip install as described in README.md. Some of the dependencies (likely shapely) might give you trouble, though.

[cdesyoun]

Emilio, I installed geoalchemy and odm2api packages on github into my virtual environment for python.
Installation was successful. In current odm2 "models.py" in master branch,

# from apiCustomType import Geometry
from geoalchemy import GeometryDDL, GeometryColumn
from geoalchemy.geometry import Geometry

it was changed, commenting out the customized Geometry class. when reading the value for "FeatureGeometry" column, this current Geometry class shows this value, "0101000020E6100000DAE6C6F484014EC0D5CF9B8A54580AC0". But, the previous Geometry class in apiCustomType.py shows this value, "POINT(-55.51212 -1.94778)" as the readable text. Can we import this customized Geometry class back below?

from apiCustomType import Geometry
from geoalchemy import GeometryDDL, GeometryColumn
#from geoalchemy.geometry import Geometry

[emiliom]

@cdesyoun, I'm trying to understand the issue clearly and track what changes have been made recently.

apiCustomType.py was deprecated when we switched to using geoalchemy1 (see issue#13); it relied on geoalchemy2. From those same comments, it's unclear if there was functionality in apiCustomType.py that should be brought back, but w/o using geoalchemy2. This will take more research while Stephanie is away.

At the same time, in the last couple of days @denvaar made some changes to models.py, then reverted some of them (for good reasons) while making other changes. When exactly did you install the odm2api package that you ran those FeatureGeometry column tests on? What database type (PostgreSQL? SQLite?) were you testing against?

Also bear in mind that it's not obvious that FeatureGeometry should return a WKT by default, even if that was the previous behavior. For complex geometries (polygons, long lines), that may not be ideal. However, it should definitely provide easy access to the WKT!

This will require a good amount of additional research. I've started poking, but I'll try to do this next week; the more you can help me with testing, the better.

[emiliom]

This may be somewhat of a tangent (though related to the broad code clean-up topic) ... As I mentioned in my previous comment (referencing an earlier issue discussion with @sreeder), ODM2/apiCustomType.py has been deprecated since we decided to switch from geoalchemy2 to geoalchemy1. To double check, I did a search for usage across the odm2api, and the only place where it's imported is in ODM2/models_sqlite.py; which in turn seems like a deprecated or test version. models_sqlite.py is not called from anywhere in odm2api.

I think we should remove both apiCustomType.py and models_sqlite.py from the master branch, if @denvaar can confirm that they're not being used anymore. To be cautious and for future reference, I would create a new "backup" branch to store a master snapshot that includes these two files, then delete them from master.

[cdesyoun]

Emilio, this issue was that currently, the return value of "FeatureGeometry" is wkb, not wkt when querying "SamplingFeature". for example,

<SamplingFeatures('guaya', 'Guayaramerin', 'Station ID 16001 - PHICAB-GM?', '120.0', '0101000020E6100000950ED6FF395650C0FCDEA63FFBB125C0')>

When using "apiCustomType.py", the return value was wkt.
I used "shapely" package to convert this wkb to wkt so that I got this error, "pg_functions has no attribute 'read'" or something like that. And if it is not big issue, I can use this wkb string to return on odm2 rest api.

And I uploaded this models_sqlite.py, https://github.com/ODM2/ODM2PythonAPI/blob/master/odm2api/ODM2/models_sqlite.py as test example, creating schema into sqlite via sqlalchemy interface. You can remove this file.
And I mentioned the issue, #25, mysql and sqlite do not need schema tag.

[denvaar]

@emiliom I merged the setup branch into master this morning and deleted the branch. There were no merge conflicts (surprising) so hopefully the merge won't have any negative repercussions. Most of my work on the setup branch was adding new things rather than changing things.

@cdesyoun As for the sampling features, when I reverted the changes I made to SamplingFeature's __repr__ method, I did add an if else block to check for wkt or wkb. If that's causing a problem, I can get rid of that too.

[cdesyoun]

@emiliom and @denvaar, I got a readable text from WKB in FeatureGeometry using these codes below:

      if value.FeatureGeometry:
                binary = binascii.unhexlify(str(value.FeatureGeometry))
                point = wkb.loads(binary)
                point = '{0}'.format(wkt.dumps(point))
                samplingfeature['FeatureGeometry'] = point
      else:
                samplingfeature['FeatureGeometry'] = None

The result is below:

"FeatureGeometry": "POINT (-55.5121200000000030 -1.9477800000000001)"

so, there is no issue in odm2 REST api currently.

[cdesyoun]

@emiliom and @valentinedwv, I updated ODM2 REST API that integrates with odm2api with Emilio's review comments. Test site is http://sis-devel.cloudapp.net/docs/.

[emiliom]

@emiliom I merged the setup branch into master this morning and deleted the branch. There were no merge conflicts (surprising) so hopefully the merge won't have any negative repercussions.

Thanks, @denvaar!! Terrific. I didn't get a chance to test it today, but hopefully tomorrow.

I got a readable text from WKB in FeatureGeometry using these codes below ... so, there is no issue in odm2 REST api currently.

Thanks for that testing, @cdesyoun. Personally, I think odm2api should provide access to the WKT text representation much more easily, and not expect users to do all the transformations you've implemented. Independent of whether or not WKT is the default FeatureGeometry value type, WKT should be accessible easily via a method. Incidentally, I also intend to implement the Python Geo Interface on this entity; but that'll be later on.

BTW, you have hard-wired a point-type in your conversion. Most SamplingFeatures you'll encounter will be points, but we can't assume that in general.