Fix httpretrieve #151
Open
Fix httpretrieve #151
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Thanks @smmiller for suggesting this and providing an initial fix. My modifications make `httpretrieve_open` work as its docstring says, i.e. the *caller* must take care of urlencoding the `querydata` and/or `postdata` arguments. The `_httpretrieve_build_request` function that is called internally only adds the `Content-encoding` HTTP header automatically, but does not attempt to convert/modify the actual data. This behavior essentially mimics what Python's `urllib` does.
If there is no data to be POSTed or no query, then there need not be an encoding. If there is data/a query, its default encoding is set to `application/x-www-form-urlencoded` unless an encoding was specified by the caller already.
|
Attempting to review this pull request and some of the httpretrieve unit tests complain about a NameError: $ python utf.py -a
Testing module: seattlelib
Running: ut_seattlelib_advertisepipe_works.r2py [ PASS ]
Running: ut_seattlelib_centralizeadvertise.py [ PASS ]
Running: ut_seattlelib_centralizeadvertise_v2.py [ PASS ]
Running: ut_seattlelib_dylink_catch_nestedimport_exception.r2py [ PASS ]
Running: ut_seattlelib_dylinkcircularimport1.r2py [ PASS ]
Running: ut_seattlelib_dylinkcircularimport2.r2py [ PASS ]
Running: ut_seattlelib_dylinkcircularimportsymbols1.r2py [ PASS ]
Running: ut_seattlelib_dylinkcircularimportsymbols2.r2py [ PASS ]
Running: ut_seattlelib_dylinkdispatch.r2py [ PASS ]
Running: ut_seattlelib_dylinkimportmodule.r2py [ PASS ]
Running: ut_seattlelib_dylinkimportmodulesymbols.r2py [ PASS ]
Running: ut_seattlelib_dylinkoverwritesymbols.r2py [ PASS ]
Running: ut_seattlelib_dylinkpythonfromximport.py [ PASS ]
Running: ut_seattlelib_dylinkpythonimport.py [ PASS ]
Running: ut_seattlelib_dylinkrepyportabilityimportmodule.py [ PASS ]
Running: ut_seattlelib_dylinkrepyportabilityimportmodulesymbols.py [ FAIL ]
--------------------------------------------------------------------------------
Standard out :
..............................Produced..............................
Unexpected contents in variable: local override (expected examplelib-repy.r2py's foo)
..............................Expected..............................
None
--------------------------------------------------------------------------------
Running: ut_seattlelib_dylinkrespectsfileextensions.r2py [ PASS ]
Running: ut_seattlelib_dylinkselfimport.r2py [ PASS ]
Running: ut_seattlelib_dylinksimple.r2py [ PASS ]
Running: ut_seattlelib_dylinksimplepython.py [ PASS ]
Running: ut_seattlelib_dylinktestlib1.r2py [ PASS ]
Running: ut_seattlelib_dylinktestlib2.r2py [ PASS ]
Running: ut_seattlelib_httpretrieve_content.py [ FAIL ]
--------------------------------------------------------------------------------
Standard error :
..............................Produced..............................
Traceback (most recent call last):
File "ut_seattlelib_httpretrieve_content.py", line 8, in <module>
dy_import_module_symbols('httpretrieve.r2py')
NameError: name 'dy_import_module_symbols' is not defined
..............................Expected..............................
None
--------------------------------------------------------------------------------
Running: ut_seattlelib_httpretrieve_content_timeout.py [ FAIL ]
--------------------------------------------------------------------------------
Standard error :
..............................Produced..............................
Traceback (most recent call last):
File "ut_seattlelib_httpretrieve_content_timeout.py", line 8, in <module>
dy_import_module_symbols('httpretrieve.r2py')
NameError: name 'dy_import_module_symbols' is not defined
..............................Expected..............................
None
--------------------------------------------------------------------------------
Running: ut_seattlelib_httpretrieve_filelikeobj_closed.py [ FAIL ]
--------------------------------------------------------------------------------
Standard error :
..............................Produced..............................
Traceback (most recent call last):
File "ut_seattlelib_httpretrieve_filelikeobj_closed.py", line 9, in <module>
dy_import_module_symbols('httpretrieve.r2py')
NameError: name 'dy_import_module_symbols' is not defined
..............................Expected..............................
None
--------------------------------------------------------------------------------
Running: ut_seattlelib_httpretrieve_header_timeout.py [ FAIL ]
--------------------------------------------------------------------------------
Standard error :
..............................Produced..............................
Traceback (most recent call last):
File "ut_seattlelib_httpretrieve_header_timeout.py", line 7, in <module>
dy_import_module_symbols('httpretrieve.r2py')
NameError: name 'dy_import_module_symbols' is not defined
..............................Expected..............................
None
--------------------------------------------------------------------------------
Running: ut_seattlelib_httpretrieve_post.py [ FAIL ]
--------------------------------------------------------------------------------
Standard error :
..............................Produced..............................
Traceback (most recent call last):
File "ut_seattlelib_httpretrieve_post.py", line 8, in <module>
dy_import_module_symbols('httpretrieve.r2py')
NameError: name 'dy_import_module_symbols' is not defined
..............................Expected..............................
None
--------------------------------------------------------------------------------
Running: ut_seattlelib_httpretrieve_read_done.py [ FAIL ]
--------------------------------------------------------------------------------
Standard error :
..............................Produced..............................
Traceback (most recent call last):
File "ut_seattlelib_httpretrieve_read_done.py", line 9, in <module>
dy_import_module_symbols('httpretrieve.r2py')
NameError: name 'dy_import_module_symbols' is not defined
..............................Expected..............................
None
--------------------------------------------------------------------------------
Running: ut_seattlelib_httpretrieve_read_limit.py [ FAIL ]
--------------------------------------------------------------------------------
Standard error :
..............................Produced..............................
Traceback (most recent call last):
File "ut_seattlelib_httpretrieve_read_limit.py", line 9, in <module>
dy_import_module_symbols('httpretrieve.r2py')
NameError: name 'dy_import_module_symbols' is not defined
..............................Expected..............................
None
--------------------------------------------------------------------------------
Running: ut_seattlelib_httpretrieve_redirect_notgiven.py [ FAIL ]
--------------------------------------------------------------------------------
Standard error :
..............................Produced..............................
Traceback (most recent call last):
File "ut_seattlelib_httpretrieve_redirect_notgiven.py", line 9, in <module>
dy_import_module_symbols('httpretrieve.r2py')
NameError: name 'dy_import_module_symbols' is not defined
..............................Expected..............................
None
--------------------------------------------------------------------------------
Running: ut_seattlelib_ntptime.py [ PASS ]
Running: ut_seattlelib_ntptime_resolve_servers.r2py [ PASS ]
Running: ut_seattlelib_parallelize_finishes.py [ PASS ]
Running: ut_seattlelib_session.r2py [ PASS ]
Running: ut_seattlelib_sockettimeout.py [ PASS ]
Running: ut_seattlelib_tcptime.py [ PASS ]
Running: ut_seattlelib_time_interface_updatetime_before_registermethod.r2py [ PASS ]
Running: ut_seattlelib_udpcentralizeadvertise.py [ PASS ]
Testing module: librepy
Running: ut_librepy_file_dup.r2py [ PASS ]
Running: ut_librepy_file_iter.r2py [ PASS ]
Running: ut_librepy_file_open.r2py [ PASS ]
Running: ut_librepy_file_read.r2py [ PASS ]
Running: ut_librepy_file_readline.r2py [ PASS ]
Running: ut_librepy_file_seek.r2py [ PASS ]
Running: ut_librepy_file_size.r2py [ PASS ]
Running: ut_librepy_file_write.r2py [ PASS ]
Running: ut_librepy_lock.r2py [ PASS ]
Running: ut_librepy_openconn-args.r2py [ PASS ]
Running: ut_librepy_openconn-basic.r2py [ PASS ]
Running: ut_librepy_openconn-cleanup.r2py [ FAIL ]
--------------------------------------------------------------------------------
Standard error :
..............................Produced..............................
---
Uncaught exception!
---
Following is a full traceback, and a user traceback.
The user traceback excludes non-user modules. The most recent call is displayed last.
Full debugging traceback:
"repy.py", line 154, in execute_namespace_until_completion
"/home/vlad/projects/seattletestbed/seattlelib_v2/RUNNABLE/virtual_namespace.py", line 117, in evaluate
"/home/vlad/projects/seattletestbed/seattlelib_v2/RUNNABLE/safe.py", line 588, in safe_run
"dylink.r2py", line 546, in <module>
"dylink.r2py", line 407, in dylink_dispatch
"dylink.r2py", line 520, in evaluate
"/home/vlad/projects/seattletestbed/seattlelib_v2/RUNNABLE/virtual_namespace.py", line 117, in evaluate
"/home/vlad/projects/seattletestbed/seattlelib_v2/RUNNABLE/safe.py", line 588, in safe_run
"librepy.r2py", line 153, in <module>
"dylink.r2py", line 446, in _dylink_dispatch
"dylink.r2py", line 407, in dylink_dispatch
"dylink.r2py", line 520, in evaluate
"/home/vlad/projects/seattletestbed/seattlelib_v2/RUNNABLE/virtual_namespace.py", line 117, in evaluate
"/home/vlad/projects/seattletestbed/seattlelib_v2/RUNNABLE/safe.py", line 588, in safe_run
"ut_librepy_openconn-cleanup.r2py", line 21, in <module>
"librepysocket.r2py", line 569, in openconn
"/home/vlad/projects/seattletestbed/seattlelib_v2/RUNNABLE/namespace.py", line 1207, in wrapped_function
"/home/vlad/projects/seattletestbed/seattlelib_v2/RUNNABLE/emulcomm.py", line 1325, in openconnection
"/home/vlad/projects/seattletestbed/seattlelib_v2/RUNNABLE/emulcomm.py", line 1060, in _conn_alreadyexists_check
User traceback:
"dylink.r2py", line 546, in <module>
"dylink.r2py", line 407, in dylink_dispatch
"dylink.r2py", line 520, in evaluate
"librepy.r2py", line 153, in <module>
"dylink.r2py", line 446, in _dylink_dispatch
"dylink.r2py", line 407, in dylink_dispatch
"dylink.r2py", line 520, in evaluate
"ut_librepy_openconn-cleanup.r2py", line 21, in <module>
"librepysocket.r2py", line 569, in openconn
Exception (with class 'exception_hierarchy.AddressBindingError'): Cannot bind to the specified local ip, invalid!
---
..............................Expected..............................
None
--------------------------------------------------------------------------------
Running: ut_librepy_random.r2py [ PASS ]
Running: ut_librepy_recvmess-args.r2py [ PASS ]
Running: ut_librepy_recvmess-basic.r2py [ PASS ]
Running: ut_librepy_recvmess-basictheadpool.r2py [ PASS ]
Running: ut_librepy_recvmess-dup.r2py [ PASS ]
Running: ut_librepy_recvmess-reuse.r2py [ FAIL ]
--------------------------------------------------------------------------------
Standard out :
..............................Produced..............................
Unexpected error while listening on a network socket!
Listening for: UDP on IP: 128.238.64.148 Port: 12345
Debug string:
---
Following is a full traceback, and a user traceback.
The user traceback excludes non-user modules. The most recent call is displayed last.
Full debugging traceback:
"librepysocket.r2py", line 420, in accept_waiting
"/home/vlad/projects/seattletestbed/seattlelib_v2/RUNNABLE/namespace.py", line 945, in __do_func_call
"/home/vlad/projects/seattletestbed/seattlelib_v2/RUNNABLE/namespace.py", line 1207, in wrapped_function
"/home/vlad/projects/seattletestbed/seattlelib_v2/RUNNABLE/emulcomm.py", line 1988, in getmessage
User traceback:
"librepysocket.r2py", line 420, in accept_waiting
Exception (with class 'exception_hierarchy.SocketClosedLocal'): The socket has been closed!
---
..............................Expected..............................
None
--------------------------------------------------------------------------------
Running: ut_librepy_recvmess-stop.r2py [ PASS ]
Running: ut_librepy_rlock.r2py [ PASS ]
Running: ut_librepy_runloop_basic.r2py [ PASS ]
Running: ut_librepy_runloop_cancel.r2py [ PASS ]
Running: ut_librepy_runloop_schedule.r2py [ PASS ]
Running: ut_librepy_runloop_terminate.r2py [ PASS ]
Running: ut_librepy_sendmess-args.r2py [ PASS ]
Running: ut_librepy_sendmess-basic.r2py [ PASS ]
Running: ut_librepy_socket-blockingrecv.r2py [ PASS ]
Running: ut_librepy_socket-blockingsend.r2py [ FAIL ]
--------------------------------------------------------------------------------
Standard error :
..............................Produced..............................
---
Uncaught exception!
---
Following is a full traceback, and a user traceback.
The user traceback excludes non-user modules. The most recent call is displayed last.
Full debugging traceback:
"repy.py", line 154, in execute_namespace_until_completion
"/home/vlad/projects/seattletestbed/seattlelib_v2/RUNNABLE/virtual_namespace.py", line 117, in evaluate
"/home/vlad/projects/seattletestbed/seattlelib_v2/RUNNABLE/safe.py", line 588, in safe_run
"dylink.r2py", line 546, in <module>
"dylink.r2py", line 407, in dylink_dispatch
"dylink.r2py", line 520, in evaluate
"/home/vlad/projects/seattletestbed/seattlelib_v2/RUNNABLE/virtual_namespace.py", line 117, in evaluate
"/home/vlad/projects/seattletestbed/seattlelib_v2/RUNNABLE/safe.py", line 588, in safe_run
"librepy.r2py", line 153, in <module>
"dylink.r2py", line 446, in _dylink_dispatch
"dylink.r2py", line 407, in dylink_dispatch
"dylink.r2py", line 520, in evaluate
"/home/vlad/projects/seattletestbed/seattlelib_v2/RUNNABLE/virtual_namespace.py", line 117, in evaluate
"/home/vlad/projects/seattletestbed/seattlelib_v2/RUNNABLE/safe.py", line 588, in safe_run
"ut_librepy_socket-blockingsend.r2py", line 48, in <module>
"librepysocket.r2py", line 1109, in send
"librepysocket.r2py", line 1068, in _send
"/home/vlad/projects/seattletestbed/seattlelib_v2/RUNNABLE/namespace.py", line 945, in __do_func_call
"/home/vlad/projects/seattletestbed/seattlelib_v2/RUNNABLE/namespace.py", line 1207, in wrapped_function
"/home/vlad/projects/seattletestbed/seattlelib_v2/RUNNABLE/emulcomm.py", line 1852, in send
User traceback:
"dylink.r2py", line 546, in <module>
"dylink.r2py", line 407, in dylink_dispatch
"dylink.r2py", line 520, in evaluate
"librepy.r2py", line 153, in <module>
"dylink.r2py", line 446, in _dylink_dispatch
"dylink.r2py", line 407, in dylink_dispatch
"dylink.r2py", line 520, in evaluate
"ut_librepy_socket-blockingsend.r2py", line 48, in <module>
"librepysocket.r2py", line 1109, in send
"librepysocket.r2py", line 1068, in _send
Exception (with class 'exception_hierarchy.SocketClosedRemote'): The socket has been closed remotely!
---
..............................Expected..............................
None
--------------------------------------------------------------------------------
Running: ut_librepy_socket-largerecv.r2py [ PASS ]
Running: ut_librepy_socket-nonblockrecv.r2py [ PASS ]
Running: ut_librepy_socket-nonblocksend.r2py [ PASS ]
Running: ut_librepy_socket-sendrecvall.r2py [ PASS ]
Running: ut_librepy_socket-sockpeername.r2py [ PASS ]
Running: ut_librepy_socket-timeoutsend.r2py [ PASS ]
Running: ut_librepy_socket-timoutrecv.r2py [ FAIL ]
--------------------------------------------------------------------------------
Standard error :
..............................Produced..............................
---
Uncaught exception!
---
Following is a full traceback, and a user traceback.
The user traceback excludes non-user modules. The most recent call is displayed last.
Full debugging traceback:
"repy.py", line 154, in execute_namespace_until_completion
"/home/vlad/projects/seattletestbed/seattlelib_v2/RUNNABLE/virtual_namespace.py", line 117, in evaluate
"/home/vlad/projects/seattletestbed/seattlelib_v2/RUNNABLE/safe.py", line 588, in safe_run
"dylink.r2py", line 546, in <module>
"dylink.r2py", line 407, in dylink_dispatch
"dylink.r2py", line 520, in evaluate
"/home/vlad/projects/seattletestbed/seattlelib_v2/RUNNABLE/virtual_namespace.py", line 117, in evaluate
"/home/vlad/projects/seattletestbed/seattlelib_v2/RUNNABLE/safe.py", line 588, in safe_run
"librepy.r2py", line 153, in <module>
"dylink.r2py", line 446, in _dylink_dispatch
"dylink.r2py", line 407, in dylink_dispatch
"dylink.r2py", line 520, in evaluate
"/home/vlad/projects/seattletestbed/seattlelib_v2/RUNNABLE/virtual_namespace.py", line 117, in evaluate
"/home/vlad/projects/seattletestbed/seattlelib_v2/RUNNABLE/safe.py", line 588, in safe_run
"ut_librepy_socket-timoutrecv.r2py", line 51, in <module>
"librepysocket.r2py", line 1109, in send
"librepysocket.r2py", line 1068, in _send
"/home/vlad/projects/seattletestbed/seattlelib_v2/RUNNABLE/namespace.py", line 945, in __do_func_call
"/home/vlad/projects/seattletestbed/seattlelib_v2/RUNNABLE/namespace.py", line 1207, in wrapped_function
"/home/vlad/projects/seattletestbed/seattlelib_v2/RUNNABLE/emulcomm.py", line 1823, in send
User traceback:
"dylink.r2py", line 546, in <module>
"dylink.r2py", line 407, in dylink_dispatch
"dylink.r2py", line 520, in evaluate
"librepy.r2py", line 153, in <module>
"dylink.r2py", line 446, in _dylink_dispatch
"dylink.r2py", line 407, in dylink_dispatch
"dylink.r2py", line 520, in evaluate
"ut_librepy_socket-timoutrecv.r2py", line 51, in <module>
"librepysocket.r2py", line 1109, in send
"librepysocket.r2py", line 1068, in _send
Exception (with class 'exception_hierarchy.SocketClosedRemote'): The socket has been closed by the remote end!
---
..............................Expected..............................
None
--------------------------------------------------------------------------------
Standard out :
..............................Produced..............................
Unexpected error while listening on a network socket!
Listening for: TCP on IP: 128.238.64.148 Port: 12345
Debug string:
Uncaught exception invoking callback for TCP listener!
Socket will be closed automatically.
---
Following is a full traceback, and a user traceback.
The user traceback excludes non-user modules. The most recent call is displayed last.
Full debugging traceback:
"librepysocket.r2py", line 725, in callback
"ut_librepy_socket-timoutrecv.r2py", line 27, in incoming
"librepysocket.r2py", line 1046, in recv
"librepysocket.r2py", line 1022, in _recv
User traceback:
"librepysocket.r2py", line 725, in callback
"ut_librepy_socket-timoutrecv.r2py", line 27, in incoming
"librepysocket.r2py", line 1046, in recv
"librepysocket.r2py", line 1022, in _recv
Exception (with class 'exception_hierarchy.TimeoutError'): recv() timed out!
---
..............................Expected..............................
None
--------------------------------------------------------------------------------
Running: ut_librepy_subclassthread.r2py [ PASS ]
Running: ut_librepy_subclasstimer.r2py [ PASS ]
Running: ut_librepy_thread.r2py [ PASS ]
Running: ut_librepy_threadpool_destroy.r2py [ PASS ]
Running: ut_librepy_threadpool_downscaling.r2py [ PASS ]
Running: ut_librepy_threadpool_maxthreads.r2py [ PASS ]
Running: ut_librepy_threadpool_minthreads.r2py [ PASS ]
Running: ut_librepy_threadpool_upscaling.r2py [ PASS ]
Running: ut_librepy_timer.r2py [ PASS ]
Running: ut_librepy_timerjoin.r2py [ PASS ]
Running: ut_librepy_waitforconn-args.r2py [ PASS ]
Running: ut_librepy_waitforconn-basic.r2py [ PASS ]
Running: ut_librepy_waitforconn-basicthreadpool.r2py [ PASS ]
Running: ut_librepy_waitforconn-dup.r2py [ PASS ]
Running: ut_librepy_waitforconn-reuse.r2py [ PASS ]
Running: ut_librepy_waitforconn-reusethreadpool.r2py [ PASS ]
Running: ut_librepy_waitforconn-stop.r2py [ PASS ]
$ |
|
Thanks for reviewing! The unit tests seem to lack a Otherwise, |
|
Adding pragmas to these unit tests will require converting them to r2py modules, correct? $ python utf.py -f ut_seattlelib_httpretrieve_content.py
Testing module: seattlelib
Running: ut_seattlelib_httpretrieve_content.py [ FAIL ]
--------------------------------------------------------------------------------
Standard error :
..............................Produced..............................
---
Uncaught exception!
---
Following is a full traceback, and a user traceback.
The user traceback excludes non-user modules. The most recent call is displayed last.
Full debugging traceback:
"repy.py", line 154, in execute_namespace_until_completion
"/home/vlad/projects/seattlelib_v2/RUNNABLE/virtual_namespace.py", line 117, in evaluate
"/home/vlad/projects/seattlelib_v2/RUNNABLE/safe.py", line 588, in safe_run
"dylink.r2py", line 546, in <module>
"dylink.r2py", line 397, in dylink_dispatch
"dylink.r2py", line 155, in _dy_module_code
User traceback:
"dylink.r2py", line 546, in <module>
"dylink.r2py", line 397, in dylink_dispatch
"dylink.r2py", line 155, in _dy_module_code
Exception (with type 'exceptions.Exception'): Failed to initialize the module (ut_seattlelib_httpretrieve_content.py)! Got the following exception: 'CodeUnsafeError('Code failed safety check! Error: ("<class \'exception_hierarchy.CheckNodeException\'> (43, \'Printnl\')",)',)'
---
..............................Expected..............................
None
-------------------------------------------------------------------------------- |
|
Yep, there's a few |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR fixes a couple of issues with
httpretrieve.r2py. Thanks go to @smmiller, @DennisMirante, and @yyzhuang for reporting / fixing / feedback.