Python 3.3 Compatibility!

[bardiharborow]

The title says it all.

[fcracker79]

Are you planning to do the migration?
Alternatively, I might evaluate contributing if you have not planned to do that.

[vbuterin]

I probably won't have time myself. Would very much appreciate it if you
could help out.

[fcracker79]

Hi there,

I have almost done. After porting, I have 3 errors and 1 failure from tests (2 errors are related to electrum popen, so I think it is something stupid...).

However, due to the distinction between strings and bytes introduced by python 3, it seems to be difficult to merge the two versions to be compliant both for versions 2 and 3. I could esplicitly skip individual changes if python version is 3, but the overall result would be far from maintainable.

Any suggestions? Please consider that I am a python newbie.

[vbuterin]

Hmm. I recall that was actually the big issue that I ran into when trying
to make it py3 compliant. What are the functions that are hard to make
compatible with both 2 and 3 again? Does it have to do with the methods
that accept both binary and hex input? WIll think if I can come up with a
solution.

[fcracker79]

Basically, most of the code does not care about the fact that an object is a string or a 'bytes'. In addition to that, I have noticed subtle different behaviours in binascii.hexlify:

python -c 'import binascii; print(binascii.hexlify(b"hello"))' returns the string "68656c6c6f"

whereas

python3 -c 'import binascii; print(binascii.hexlify(b"hello"))' returns the bytes b"68656c6c6f"

I have found most of the problems with serialize / deserialize (I still have some issues but I can see the end), which work both with strings and bytes (they are also recursive, which made it hard to make it work), so I can not say if a string comes from the fact that is a "binary" string or it is a "real" string. Many parts of the code speculate about the type of the object: this is a problem because isinstance(o, str) is False for bytes, or it checks if it checks if the string is alphanumeric using regexp, which crashes with bytes.

All in all, I have made changes so that there could be no "binary" strings but only "real" strings or "bytes", then I have hammered all the points where a "binary" string would have worked, but this has produced scattered changes.

Maybe I could finish with the porting, then we see the differences and we might be lucky enough to isolate the -hopefully- small pieces of code that need to be different for the two versions.

Do you think that the tests included in the project have a good code coverage? I have based my work on their outcome...

[fcracker79]

Meanwhile, I have completed the tests. I as mentioned above, I have two failing tests due to popen invocation, the other ones are clear.

[fcracker79]

Hi there,

the porting is complete. Now it would be important to merge the two version.

I will work on it, but any suggestion / help would be really appreciated.

[vbuterin]

I would say the best approach would be to have all of the methods that
currently support both binary and hex switch to supporting only either (i)
hex as string, or (ii) binary as bytes (base58 also counts as a string).
encode with base 256 and changebase with base 256 would also output bytes.
Ideally, in python 2, it would work exactly as before.

[fcracker79]

Pull requested.

[fcracker79]

Hi vbuterin,

any news about my pull request?

Regards

[vbuterin]

Just integrated. All tests pass on python2, python3 and pypy. Thanks guys!

[fcracker79]

Great! Now I would say that this issue can be closed.
Can I reach you somewhere to discuss with you about further development I would like do implement?

[vbuterin]

Sure. Email at [email protected] would be best.

[rubensayshi]

this can be closed?

[MiWCryptoCurrency]

I tried to get the ethereum genesis block creator [https://raw.githubusercontent.com/ethereum/genesis_block_generator/master/mk_genesis_block.py] to work with python3.
This script uses this library to generate the genesis block, from the blockchain API.

it kept throwing Exceptions because the calls to json.load() in bci.py were expecting strings, not bytes.
I modified every call to json.load(X) to json.load(X.decode("utf-8")) and it appeared to work correctly and produced parsed objects.
This change would break compat with 2.7, but might be able to rewrite with an include to future for dual compat.

[PeerLabs]

Python 3.4.3 ?

I want to use this library with a python app I'm developing.

So i tested this script in python2.7 ... all sweet

however I'm developing on python3 (3.4.3) ... and well I ran into this...

addr = pubtoaddr(pub)
addr
'1CQLd3bhw4EzaURHbKCwM5YZbUQfA4ReY6'
h = history(addr)
Traceback (most recent call last):
File "../pybitcointools/bitcoin/bci.py", line 169, in history
jsonobj = json.loads(data)
File "/usr/local/Cellar/python3/3.4.3_2/Frameworks/Python.framework/Versions/3.4/lib/python3.4/json/init.py", line 312, in loads
s.class.name))
TypeError: the JSON object must be str, not 'bytes'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "", line 1, in
File "/Volumes/DataDrive/ProductionDepot/piBlock/Dev/Frameworks/pybitcointools/bitcoin/bci.py", line 171, in history
raise Exception("Failed to decode data: "+data)
TypeError: Can't convert 'bytes' object to str implicitly

type(addr)
<class 'str'>

So couple questions... Should I 'back-port' my entire app to python2.7 so that i can use this library... or... Is this something people are working on to resolve?

Thanks

[BobbyBooShay]

Hm, what a coincidence! I was just now running in this same issue at nearly the exact same time! If anyone has a fix for Python 3.4 that'd be great! I'm just an amateur at best myself, so I've been tinkering around and wondering how to get it to work.

I'm just glad that it's not just me me, but rather a compatibility issue.

[MiWCryptoCurrency]

PeerLabs - no, py3 is the future. the future is now. always look forward! ;-)

BobbyBooShay - i did a PR that worked for py3 for me here, #108 but simcity4242 is better
#90 either should allow you to use the blockchain.info HTTP API lib with py3

[BobbyBooShay]

Thanks so much MiWCryptoCurrency. Your fix was pretty straight forward and easy to implement. I see simcity4242's is a bit more elegant and savvy. I'm still quite new and unfamiliar with Git and haven't learned how I would go about merging someone else's pull request or have git control the pybitcoin module.

Is there a quick primer or somewhere I should look to become more familiar with ways to utilize git when you are dealing with packages and modules such as these on your project? Almost all Bitcoin related projects are still being developed and have new features or fixes such as this one coming daily and I'm sure there is a better method rather than going in and just updating / changing things manually.

[MiWCryptoCurrency]

https://stackoverflow.com/questions/6022302/pull-requests-from-other-forks-into-my-fork has a rich set of answers on applying PR to a fork. Start by cloning this repo and applying simcity's PR.

It could work something like this if you wanted to have a git repo as a library in python.
Python is nice because it doesn't need compilation before execution, and will generate new bytecode at runtime if the .py is newer.
To replace your local copy running within python, assuming you installed with pip,
go into your Lib/site-packages, remove the existing pybitcointools directory and and clone your repo to 'pybitcointools-my-git'.
Symlink (Windows: mklink / POSIX: ln) the 'pybitcointools-my-git/bitcoin' dir to 'pybitcointools'.
Apply PR's to your fork and/or merge with the main project, update the local libraries from your fork.

There is probably a better way to do this with virtualenv so you can have pip version and git, someone else may wish to comment.

[gustavklopp]

No news about Python3 compatibility?

[gdassori]

The library is python3.4 compliant on this fork.

[andzsy]

Still getting this message:

bip32_master_key("xxxxx xxx x xxx xxxx xxxx xxx")
Traceback (most recent call last):
File "<pyshell#1>", line 1, in
bip32_master_key("scout retire curtain item key render ill require book nuclear mammal train")
File "C:\Python35\lib\site-packages\bitcoin-1.0.0_segwit_conio-py3.5.egg\bitcoin\deterministic.py", line 135, in bip32_master_key
I = hmac.new(from_string_to_bytes("Bitcoin seed"), seed, hashlib.sha512).digest()
File "C:\Python35\lib\hmac.py", line 144, in new
return HMAC(key, msg, digestmod)
File "C:\Python35\lib\hmac.py", line 84, in init
self.update(msg)
File "C:\Python35\lib\hmac.py", line 93, in update
self.inner.update(msg)
TypeError: Unicode-objects must be encoded before hashing