error when torrent opened

[tejt99]

Hi thanks for all your help with #3 that command now works without errors, however when I run:

python3 -m pythorrent --file "FOOL'S GOLD _DAY OFF FIRE_.torrent" --path . --log=info
I get a large error:

Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/Users/theoturner/Downloads/python-pythorrent-master/pythorrent/main.py", line 48, in
client.run()
File "/Users/theoturner/Downloads/python-pythorrent-master/pythorrent/torrent.py", line 343, in run
for peer in self.peers.values()[:self.MAX_PEERS]:
File "/Users/theoturner/Downloads/python-pythorrent-master/pythorrent/torrent.py", line 293, in peers
for peer_store in self.peer_stores.values():
File "/Users/theoturner/Downloads/python-pythorrent-master/pythorrent/torrent.py", line 246, in peer_stores
self._peer_stores[url] = store_from_url(url)(url, self)
File "/Users/theoturner/Downloads/python-pythorrent-master/pythorrent/peer_stores.py", line 47, in store_from_url
parsed_url = urlparse(url)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/parse.py", line 367, in urlparse
url, scheme, _coerce_result = _coerce_args(url, scheme)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/parse.py", line 123, in _coerce_args
return _decode_args(args) + (_encode_result,)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/parse.py", line 107, in _decode_args
return tuple(x.decode(encoding, errors) if x else '' for x in args)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/parse.py", line 107, in
return tuple(x.decode(encoding, errors) if x else '' for x in args)
AttributeError: 'list' object has no attribute 'decode'
I have no idea where this is caused, if it's in the module itself or in your script. Thanks again.

[yamatt]

the error is being thrown a long way in to the standard library, which isn't helpful in debugging except that it most likely means the last bit of the code in this project line 47 is the culprit.

I think there are a couple of things here to help resolve it.

first my hunch is that url is a byte array when it should be a string and can be fixed by running .encode on it before being passed through.

second, it would be helpful to improve the logging to help find that out.

also I assume this isn't fixed by #6?

[tejt99]

Hi I tried adding url.encode() above parsed_url = urlparse(url) but that failed, giving the original error and

url.encode()
AttributeError: 'list' object has no attribute 'encode'.

And no it wasn't changed by #6

[yamatt]

right, done a bit of digging. What has happened is, in torrent.py line 154 I had intended to un-wrap the list in a list (maybe it was a python2 thing). By removing the brackets it made it work, but it is still returning a list rather than a string.

I think the simplest fix is to update the line to:

lambda url: url[0], metainfo['announce-list']

After fixing this issue it looks like there is still a lot of additional python2/3 stuff that's going to cause problems.

To be honest if I had the time I would like to re-architect this whole project, adding some linting and tests, making it python3 only, and moving away from maps and lambdas and using list comprehension in line with Guido's wishes.
I've mostly moved my work to my personal gitlab these days. Though GitHub actions look cool for CI/CD so I might have a play this afternoon.

I like the idea of this project being a beacon of good Pythonic and development practices, but you know, time pressures 😉

Good luck! I look forward to the next issue.

[tejt99]

Yes, I now get:

Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/Users/theoturner/Downloads/python-pythorrent-master/pythorrent/main.py", line 48, in
client.run()
File "/Users/theoturner/Downloads/python-pythorrent-master/pythorrent/torrent.py", line 336, in run
for peer in list(self.peers.values())[:self.MAX_PEERS]:
File "/Users/theoturner/Downloads/python-pythorrent-master/pythorrent/torrent.py", line 286, in peers
for peer_store in list(self.peer_stores.values()):
File "/Users/theoturner/Downloads/python-pythorrent-master/pythorrent/torrent.py", line 244, in peer_stores
self._peer_stores[url] = store_from_url(url)(url, self)
File "/Users/theoturner/Downloads/python-pythorrent-master/pythorrent/peer_stores.py", line 215, in init
self.tracker_url_parts = urlparse(self.tracker_url)
AttributeError: 'UDPTracker' object has no attribute 'tracker_url'

I assume like you said these are py2 errors caused by running in py3. I am so sorry you've done all the work but I am a bit basic at the moment with python.

[yamatt]

we've all got to learn sometime 😀

that looks like an inheritance problem and is missing a super at the top of the init.

[tejt99]

In torrent.py I changed:

class TorrentClient(object):
    def __init__(self, save_path, torrents=[]):
        self.save_path = save_path
        self.torrents = []

to:

class TorrentClient(object):
    def __init__(self, save_path, torrents=[]):
        super()
        self.save_path = save_path
        self.torrents = []

I am not sure if this is what you meant but it doesn't generate any errors on it's own, however, I get all the same errors as before. Thanks so much for all your help

[yamatt]

Hi, yes, unfortunately that's not quite how super() works.

The key thing is that super() is related to object inheritance in Object Oriented programming and is used to gain access to the "parent" class of the running method. The super() therefore needs more work than what is provided here, namely the method and arguments to call from the parent class, which is the same method it is in. this will then run the method from the parent class, meaning you can retain the parent methods actions and augmenting them with your own.

[tejt99]

Hi, sorry for the slow reply but super(self, save_path, torrents=[]) doesn't work either.

[yamatt]

it should be super().__init__(save_path, torrents=[])

[tejt99]

Hi thanks. I now get these errors:

Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/Users/USER/Downloads/python-pythorrent-master/pythorrent/main.py", line 48, in
client.run()
File "/Users/USER/Downloads/python-pythorrent-master/pythorrent/torrent.py", line 336, in run
for peer in list(self.peers.values())[:self.MAX_PEERS]:
File "/Users/USER/Downloads/python-pythorrent-master/pythorrent/torrent.py", line 286, in peers
for peer_store in list(self.peer_stores.values()):
File "/Users/USER/Downloads/python-pythorrent-master/pythorrent/torrent.py", line 244, in peer_stores
self._peer_stores[url] = store_from_url(url)(url, self)
File "/Users/USER/Downloads/python-pythorrent-master/pythorrent/peer_stores.py", line 215, in init
self.tracker_url_parts = urlparse(self.tracker_url)
AttributeError: 'UDPTracker' object has no attribute 'tracker_url'

Sorry I won't be able to respond for a couple of weeks.

[yamatt]

Thanks for the update. I think I would have to see the code to understand why that is happening. If you want to update your PR, or paste it here that would be useful.

[tejt99]

#    This file is part of PYThorrent.
#
#    PYThorrent is free software: you can redistribute it and/or modify
#    it under the terms of the GNU General Public License as published by
#    the Free Software Foundation, either version 3 of the License, or
#    (at your option) any later version.
#
#    PYThorrent is distributed in the hope that it will be useful,
#    but WITHOUT ANY WARRANTY; without even the implied warranty of
#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#    GNU General Public License for more details.
#
#    You should have received a copy of the GNU General Public License
#    along with PYThorrent.  If not, see <http://www.gnu.org/licenses/>.

from hashlib import sha1
from random import choice, randint
import string
import socket
from struct import pack, unpack
from collections import OrderedDict
import os
from random import choice

from . import splice
from .peer_stores import store_from_url
from .peer import Peer, BitTorrentPeerException
from .pieces import PieceLocal
from . import BitTorrentException

import requests
from torrentool.bencode import Bencode
from bitstring import BitArray

import logging
            
class Torrent(object):
    PIECE = PieceLocal
    CLIENT_NAME = "pythorrent"
    CLIENT_ID = "PY"
    CLIENT_VERSION = "0001"
    CLIENT_ID_LENGTH = 20
    CHAR_LIST = string.ascii_letters + string.digits
    PIECE_DIR = "_pieces"
    PROTOCOL_ID = "BitTorrent protocol"
    RESERVED_AREA = "\x00"*8
    DEFAULT_PORT = 6881
    MAX_PEERS = 20
    TORRENT_CACHE_URL="https://torcache.net/torrent/" \
        "{info_hash_hex}.torrent"
        
    @classmethod
    def from_info_hash(cls, info_hash, save_path):
        """
        Takes an info hash of a torrent as a binary string and builds a
        torrent from it.
        :param info_hash: A hashlib hash object.
        :param save_path: A string path to where the torrent should be
            saved.
        """
        return cls.from_info_hash_hex(
            info_hash.encode("hex")
        )
        
    @classmethod
    def from_info_hash_sha(cls, info_hash_sha, save_path):
        """
        Takes an info hash of a torrent as a hashlib sha object and
        builds a torrent from it.
        :param info_hash_sha:A hashlib hashed object.
        :param save_path: A string path to where the torrent should be
            saved.
        """
        return cls.from_info_hash_hex(
            info_hash_sha.hexdigest()
        )
        
    @classmethod
    def from_info_hash_hex(cls, info_hash_hex, save_path):
        """
        Takes an info hash of a torrent as a string and builds a URL for
        it so it can be looked up.
        :param info_hash_hex: A hex encoded string of the torrent's info
            hash.
        :param save_path: A string path to where the torrent should be
            saved.
        """
        url = self.TORRENT_CACHE_URL.format(
            info_hash_sha.hexdigest()
        )
        return cls.from_url(url)
    
    @classmethod
    def from_url(cls, url, save_path):
        """
        Takes a torrent from a URL, downloads it and loads it.
        :param url: The URL to the torrent file
        :param save_path: A string path to where the torrent should be
            saved.
        """
        response = requests.get(url)
        if response.status_code < 200 or response.status_code >= 300:
            raise BitTorrentException("Torrent file not found at:" \
                "{url}".format(url=url)
            )
        return cls.from_string(response.content, save_path)
        
    @classmethod
    def from_path(cls, path, save_path):
        """
        Takes a torrent from a path and loads it.
        :param path: A path as string to the path torrent file.
        :param save_path: A string path to where the torrent should be
            saved.
        """
        return cls.from_torrent_dict(
            Bencode.read_file(path),
            save_path
        )
        
    @classmethod
    def from_string(cls, s, save_path):
        """
        Takes a torrent file as a string and loads it.
        :param s:
        :param save_path: A string path to where the torrent should be
            saved.
        """
        return cls.from_torrent_dict(
            Bencode.read_string(s),
            save_path
        )
        
    @classmethod
    def from_torrent_dict(cls, metainfo, save_path):
        """
        Takes a torrent metainfo dictionary object and processes it for
        use in this object.
        :param metainfo:
        :param save_path: A string path to where the torrent should be
            saved.
        """
        info = metainfo['info']
        files = OrderedDict()
        if 'files' in info:
            for f in info['files']:
                files[os.path.join(*f['path'])] = f['length']
        else:
            files[info['name']] = info['length']
            
        return cls(
            name=info['name'],
            announce_urls=[url[0] for url in metainfo['announce-list']],
            # Note that info_hash is generated here because torrentool
            # returns the info_hash as hex encoded, which is really not
            # useful in most situations
            info_hash=sha1(Bencode.encode(info)).digest(), 
            piece_length=info['piece length'],
            files=files,
            piece_hashes=splice(info['pieces'], 20),
            save_path=save_path
        )
        
    def __init__(
        self, name, announce_urls, info_hash, piece_length, files, \
        piece_hashes, save_path
    ):
        """
        Represent a Torrent file and handle downloading and saving.
        :param name: Name of the torrent
        :param announce_urls: a list of URLs to find DHTs or trackers.
            The scheme is used to identify what kind of announce it is.
            http/https for normal HTTP trackers
        :param info_hash:The binary encoded info hash.
        :param piece_length: the default length of all (except the last)
            piece
        :param files: A dictionary of all the files where the key is the
            name of the file and the value is the size.
        :param piece_hashes: A list of all piece hashes as strings
        :param save_path: A string path to where the torrent should be
            saved.
        """
        self.name = name
        self.announce_urls = announce_urls
        self.info_hash = info_hash
        self.piece_length = piece_length
        self.files = files
        self.piece_hashes = piece_hashes
        self.save_path = save_path
        
        self._peer_stores = None
        self._pieces = None
        self._peer_id = None
        
        self._peers = {}
        
    @property
    def handshake_message(self):
        """
        Generate the string used to declare the protocol when passing
        messages during handshake
        """
        return "".join([
            chr(len(self.PROTOCOL_ID)),
            self.PROTOCOL_ID,
            self.RESERVED_AREA,
            self.info_hash,
            self.peer_id
        ])
        
    @property
    def peer_id(self):
        """
        Generate the peer id so that it is possible to identify other
        clients, and identify if you've connected to your own client
        """
        if self._peer_id is None:
            known_id = "-{id_}{version}-".format(
                id_=self.CLIENT_ID,
                version=self.CLIENT_VERSION,
            )
            remaining_length = self.CLIENT_ID_LENGTH - len(known_id)
            gubbins = "".join(
                [ choice(self.CHAR_LIST) for _ in range(remaining_length) ]
            )
            self._peer_id = known_id + gubbins
        return self._peer_id
        
    @property
    def port(self):
        """
        External port being used for incoming connections
        """
        return self.DEFAULT_PORT
        
    @property
    def peer_stores(self):
        """
        Returns a list of all the peer stores set up
        """
        if self._peer_stores is None:
            self._peer_stores = {}
            for url in self.announce_urls:
                self._peer_stores[url] = store_from_url(url)(url, self)
        return self._peer_stores
    
    @property
    def total_size(self):
        """
        The size of all the files
        """
        return sum(self.files.values())
        
    @property
    def downloaded(self):
        """
        Calculate the size of all the pieces that have been downloaded
        so far
        """
        return sum([self.piece_length if piece.valid else 0 for piece in list(self.pieces.values())])
        
    @property
    def uploaded(self):
        # TODO
        return 0
    
    @property
    def remaining(self):
        """
        Calculates how much is left
        """
        return self.total_size - self.downloaded
        
    @property
    def complete(self):
        """
        Checks if all pieces have downloaded
        """
        return not any([not piece.valid for piece in list(self.pieces.values())])
        
    @property
    def peers(self):
        """
        A list of peers that gets updated when necessary
        """
        for peer_store in list(self.peer_stores.values()):
            self._peers.update(peer_store.peers)
        return self._peers
            
    @property
    def pieces(self):
        """
        Map the pieces from the torrent file to memory.
        """
        if self._pieces is None:
            self._pieces = OrderedDict()
            for hash_index in range(len(self.piece_hashes)):
                piece_hash = self.piece_hashes[hash_index]
                piece = self.PIECE(piece_hash, hash_index)
                piece_path = piece.piece_path(self.piece_directory)
                if os.path.isfile(piece_path):
                    logging.info("Piece found on disk: {0}".format(
                        piece_path
                    ))
                    piece.load(piece_path)
                    if not piece.valid:
                        raise Exception("Not valid")
                        logging.warning("Piece on disk not valid. " \
                            "Clearing."
                        )
                        piece.clear()
                self._pieces[piece_hash] = piece
        return self._pieces
    
    @property
    def save_directory(self):
        """
        Return path for where the files should go.
        """
        return os.path.join(self.save_path, self.name)
    
    @property
    def piece_directory(self):
        """
        Return path for where pieces should go.
        """
        return os.path.join(self.save_directory, self.PIECE_DIR)
        
    def run(self):
        """
        Just download the torrent. No messing.
        """
        self.create_directory()
        while True:
            try:
                for peer in list(self.peers.values())[:self.MAX_PEERS]:
                    if peer.status == peer.ESTATUS.NOT_STARTED or \
                        peer.status == peer.ESTATUS.CLOSED:
                        peer.run()
                        logging.info("Handshake OK. Client ID: " \
                            "{client}".format(
                                client = peer.client
                            )
                        )
                        peer.handle_message() # should be bitfield
                
                # not a smart way to select pieces
                piece = choice([piece for piece in list(self.pieces.values()) if not piece.valid])
                logging.info("Piece {sha} selected".format(
                    sha=piece.hex
                ))
                peer = choice([peer for peer in list(self.peers.values()) if peer.pieces[piece.sha].have and not \
                        peer.status == peer.ESTATUS.CHOKE])
                logging.info("Peer {host} selected".format(
                    host=peer.hostport[0]
                ))
                piece.complete(peer.acquire(piece))
                piece_path = piece.piece_path(self.piece_directory)
                logging.info("Piece complete. Saving to: {0}".format(
                    piece_path
                ))
                with open(piece_path, "wb") as f:
                    piece.save(f)
                    

                if self.complete:
                    self.split_out()
                    
                self.advertise_piece(piece)
                
            except BitTorrentPeerException as e:
                logging.warning("Peer {host} disconnected".format(
                    host=peer.hostport[0]
                ))
                logging.debug("Disconnected because: {e}".format(e=e))
            self.clean_peers()
                
    def advertise_piece(self, piece):
        """
        For all peers let them know I now have this piece.
        :param piece: `class Piece` object
        """
        index = list(self.pieces.values()).index(piece)
        for peer in list(self.peers.values()):
            if peer.status == peer.ESTATUS.OK:
                peer.send_have(index)
                
    def split_out(self):
        """
        For all the files in the torrent, get the pieces and create the
        files.
        """
        extra_data = ""
        for file_name, size in list(self.files.items()):
            size_count = 0
            file_path = os.path.join(
                self.save_directory,
                file_name
            )
            self.make_file_path(file_path)
            with open(file_path, "wb") as f:
                f.write(extra_data) # if any
                for piece in self.pieces:
                    size_count += len(piece.data)
                    if size_count > size:
                        overrun = size_count - size
                        data = piece.data[:overrun]
                        extra_data = piece.data[overrun:]
                    else:
                        data = piece.data
                    
    def make_file_path(self, file_path):
        """
        Create any necessary sub directories for torrent files
        :path file_path: directory to make
        """
        file_dir = os.path.dirname(file_path)
        if file_dir is not "" and not os.path.exists(file_dir):
            os.makedirs(file_dir)
        
    def create_directory(self):
        """
        Create directory for files to be saved in
        """
        if not self.save_directory.startswith(self.save_path):
            raise RuntimeError(
                "Torrent name innappropriately naviages directories. " \
                "Resulting path: {0}".format(path)
            )
        if not os.path.isdir(self.save_directory):
            try:
                os.mkdir(self.save_directory)
            except OSError as e:
                raise BitTorrentException("Cannot create path '{0}'. " \
                    "Does base path exist?".format(e))
        
        if not os.path.isdir(self.piece_directory):
            os.mkdir(self.piece_directory)
            
    def clean_peers(self):
        """
        Look for any peers in `self.peers` that have had their
        connections closed and remove them from the peer list.
        """
        for k, peer in list(self.peers.items()):
            if peer.status == peer.ESTATUS.CLOSED:
                del self._peers[k]
            
            
class TorrentClient(object):
    def __init__(self, save_path, torrents=[]):
        super().__init__(save_path, torrents=[])
        self.save_path = save_path
        self.torrents = []
        
    def add_torrent(self, torrent_path):
        self.torrents.append(Torrent.from_path(
            torrent_path,
            self.save_path
        ))

torrent.py

///

#    This file is part of PYThorrent.
#
#    PYThorrent is free software: you can redistribute it and/or modify
#    it under the terms of the GNU General Public License as published by
#    the Free Software Foundation, either version 3 of the License, or
#    (at your option) any later version.
#
#    PYThorrent is distributed in the hope that it will be useful,
#    but WITHOUT ANY WARRANTY; without even the implied warranty of
#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#    GNU General Public License for more details.
#
#    You should have received a copy of the GNU General Public License
#    along with PYThorrent.  If not, see <http://www.gnu.org/licenses/>.

from datetime import timedelta, datetime
from struct import unpack, pack
import socket
from collections import OrderedDict

from bitstring import BitArray

from .pieces import PieceRemote
from . import BitTorrentException

import logging

class BitTorrentPeerException(BitTorrentException):
    pass

class Peer(object):
    PIECE = PieceRemote
    CONNECTION_TIMEOUT = 10
    BLOCK_SIZE = pow(2,14)
    class ESTATUS:
        """
        Connection status.
        """
        BAD = -3
        NOT_STARTED = -2
        CLOSED = -1
        CHOKE = 0
        OK = 1
    
    def __init__(self, hostport, torrent):
        """
        Represents the peer you're connected to.
        :param hostport: tuple containing IP and port of remote client
        :param torrent: Torrent object representing what is being
            downloaded.
        """
        self.hostport = hostport
        self.torrent = torrent
        self.status = self.ESTATUS.NOT_STARTED
        self.reserved = None
        self.info_hash = None
        self.peer_id = None
        self._pieces = None
        self.uploaded = 0
        self.buff = ""
        
    @property
    def client(self):
        """
        Quick and dirty check to find the name of the peer's client
        """
        return self.peer_id[1:7]
        
    @property
    def pieces(self):
        """
        Return all the peices associated with this torrent but from the
        peer's perspective.
        Returns list of RemotePiece
        """
        if self._pieces is None:
            self._pieces = OrderedDict()
            for torrent_piece in list(self.torrent.pieces.values()):
                peer_piece = self.PIECE(
                    torrent_piece.sha,
                    self
                )
                self._pieces[peer_piece.sha] = peer_piece
        return self._pieces
        
    def run(self):
        """
        Get everything in place to talk to peer
        """
        self.setup()
        self.handshake()
        self.recv_handshake()
        
    def setup(self):
        """
        Open socket with peer
        """
        self.conn = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        self.conn.settimeout(self.CONNECTION_TIMEOUT)
        try:
            self.conn.connect(self.hostport)
        except socket.error:
            self.close()
            raise BitTorrentPeerException(
                "Could not connect to peer: {0}".format(self.hostport)
            )
        self.status = self.ESTATUS.OK
            
    def close(self):
        """
        Central closing function
        """
        self.status = self.ESTATUS.CLOSED
        self.conn.close()
            
    def bad(self):
        """
        Like the close function but marks this peer as a bad peer so
        that the connection does not get reopened.
        """
        self.conn.close()
        self.status = self.ESTATUS.BAD
        
    def send(self, message):
        """
        Send a byte string to the peer.
        :param message: Binary string of bytes to send
        """
        try:
            self.conn.sendall(message)
        except socket.error as e:
            self.close()
            raise BitTorrentPeerException(
                "Connection closed by peer when sending"
            )
        
    def recv(self, length):
        """
        Grab custom lengths of data rather than binary divisible, or
        find that we didn't get all of it.
        Possibly not best practice but wanted a central place to do it.
        
        :param length: Integer for much to receive and recieve from the
                       socket.
        """
        buff = ""
        remaining = length
            
        while len(buff) < length:
            try:
                if remaining > 4096:
                    recvd = self.conn.recv(4096)
                else:
                    recvd = self.conn.recv(remaining)
            except socket.error as e:
                self.close()
                raise BitTorrentPeerException(
                    "Connection closed by peer when receiving"
                )
            if not recvd:
                raise BitTorrentPeerException("Received {0} {1}".format(
                    len(buff), remaining
                ))
            buff += recvd
            remaining -= len(recvd)
        return buff
        
    # HANDSHAKE
        
    def handshake(self):
        """
        Send handshake message
        """
        self.send(self.torrent.handshake_message)
        
    def recv_handshake(self):
        """
        Handle receiving handshake from peer
        """
        protocol_id_length = ord(self.recv(1))
        protocol_id = self.recv(protocol_id_length)
        if protocol_id != self.torrent.PROTOCOL_ID:
            self.bad()
            raise BitTorrentPeerException("Connection is not serving " \
                "the BitTorrent protocol. Closed connection.")
        self.reserved = self.recv(8)
        self.info_hash = self.recv(20)
        self.peer_id = self.recv(20)
        
    # HANDLE MESSAGES
    
    def handle_message(self):
        """
        Takes the message from the socket and processes it.
        """
        message_type, payload_length = self.handle_message_type()
        message_type(payload_length)
                    
    def handle_message_type(self):
        """
        Takes the message header from the socket and processes it.
        Returns the message_type as a function and the payload length as
        a tuple.
        """
        message_type, payload_length = self.handle_message_header()
        logging.debug("Message type: {0}".format(message_type))
        logging.debug("Message payload length: {0}".format(
            payload_length
        ))
        return self.type_convert(message_type), payload_length
            
    def handle_message_header(self):
        """
        Accepts the first 5 bytes from the socket that make up the
        message header and returns the message type number and the
        length of the payload as a tuple.
        """
        payload_length = unpack(">I", self.recv(4))[0]
        # protection from a keep-alive
        if payload_length > 0:
            message_type = ord(self.recv(1))
        else:
            message_type = None
        return message_type, payload_length-1
        
    def type_convert(self, message_type):
        """
        Take an integer and translate it in to the function that can
        handle that message type.
        :param message_type: integer
        Returns function.
        """
        if 0 == message_type: return self.recv_choke
        if 1 == message_type: return self.recv_unchoke
        if 2 == message_type: return self.recv_interested
        if 3 == message_type: return self.recv_uninterested
        if 4 == message_type: return self.recv_have
        if 5 == message_type: return self.recv_bitfield
        if 6 == message_type: return self.recv_request
        if 7 == message_type: return self.recv_piece
        if 8 == message_type: return self.recv_cancel
        if 9 == message_type: return self.recv_port
        return self.recv_keep_alive
            
    def acquire(self, torrent_piece):
        """
        Takes a torrent piece and makes sures to get all the blocks to
        build that piece.
        :param torrent_piece: Piece object.
        Returns Piece if correctly downloaded or None if not downloaded
        """
        index = list(self.torrent.pieces.values()).index(torrent_piece)
        peer_piece = self.pieces[torrent_piece.sha]
        
        logging.info("Sending requests for piece: {0}".format(
            torrent_piece.hex
        ))
        self.send_interested()
        next_block = peer_piece.size
        while next_block < self.torrent.piece_length:
            self.send_request(index, next_block)
            next_block += self.BLOCK_SIZE
        
        received = 0
        while not peer_piece.valid or received < self.BLOCK_SIZE:
            message_type, payload_len = self.handle_message_type()
            if message_type == self.recv_piece:
                received += len(message_type(payload_len))
            else:
                message_type(payload_len)
        logging.info("Download complete for piece: {0}".format(
            torrent_piece.hex
        ))
        if peer_piece.valid:
            return peer_piece
            
        self.bad()
        raise BitTorrentPeerException("Downloaded piece not valid. " \
            "Cut off peer."
        )
        
    # RECEIVE FUNCTIONS
        
    def recv_keep_alive(self, length=None):
        """
        These messages are sent to check the connection to the peer is
        still there.
        :param length: Does nothing here.
        """
        pass
        
    def recv_choke(self, length=None):
        """
        Recieved when the remote peer is over-loaded and won't handle
        any more messages send to it.
        :param length: Does nothing here.
        """
        self.status = self.ESTATUS.CHOKE
        payload = self.recv(length)
        
    def recv_unchoke(self, length=None):
        """
        Recieved when the remote peer has stopped being over-loaded.
        :param length: Does nothing here.
        """
        self.status = self.ESTATUS.OK
        payload = self.recv(length)
        
    def recv_interested(self, length=None):
        """
        Received when remote peer likes the look of one of your sexy
        pieces.
        :param length: Does nothing here.
        """
        payload = self.recv(length)
        
    def recv_uninterested(self, length=None):
        """
        Received when remote peer decides it can get the piece it wants
        from someone else :,(
        :param length: Does nothing here.
        """
        payload = self.recv(length)
       
    def recv_have(self, length):
        """
        Received when a peer was nice enough to tell you it has a piece
        you might be interested in.
        :param length: The size of the payload, normally a 4 byte
            integer
        """
        payload = self.recv(length)
        index = unpack(">I", payload)[0]
        list(self.pieces.values())[index].have = True
    
    def recv_bitfield(self, length):
        """
        Received only at the start of a connection when a peer wants to
        tell you all the pieces it has in a very compact form.
        :param length: The size of the payload, a number of bits
            representing the number of pieces
        """
        payload = self.recv(length)
        bits = BitArray(bytes=payload)
        for i in range(len(self.torrent.pieces)):
            sha, piece = list(self.torrent.pieces.items())[i]
            piece = self.PIECE(sha, self, have=bits[i])
            self.pieces[sha] = piece
            
    def recv_request(self, length):
        """
        Received when a peer wants a bit of one of your pieces.
        :param length: The size of the payload, a bunch of 4 byte
            integers representing the data the peer wants.
        """
        payload = self.recv(length)
        index, begin, size = unpack(">III", payload)
        if size > self.BLOCK_SIZE:
            self.bad()
            raise BitTorrentPeerException("Peer requested too much " \
                "data for a normal block: {size}".format(
                size=size
            ))
        data = list(self.pieces.values())[index].data[begin:size]
        self.send_piece(index, begin, data)
            
    def recv_piece(self, length):
        """
        The good stuff. Receives a block of data, not a whole piece, but
        makes up a part of a piece.
        :param length: The size of the payload, two 4 byte integers
            representing the position of the block, and then the block
            itself.
        """
        payload = self.recv(length)
        index, begin = unpack(">II", payload[:8])
        block = payload[8:]
        list(self.pieces.values())[index].insert_block(begin, block)
        return block
            
    def recv_cancel(self, length):
        """
        Received when a peer has made a request of a piece (or block),
        and you haven't yet fulfilled it, but the peer doesn't want it
        any more anyway.
        :param length: Not used here.
        """
        pass
            
    def recv_port(self, length):
        """
        Received when we're talking DHT, but for now, not used.
        :param length: Not used here.
        """
        payload = self.recv(length)
        
    # SEND
        
    def send_payload(self, message_type, payload=""):
        """
        Handy shortcut for sending messages.
        :param message_type: integer representing the message type
        :param payload: string of bytes of what to send
        """
        encoded_message_type = pack(">B", message_type)
        message_length = pack(
            ">I",
            len(payload) + len(encoded_message_type)
        )
        self.send(message_length + encoded_message_type + payload)
        
    def send_keep_alive(self):
        """
        Used to make sure the connection remains open.
        Doesn't use `send_payload` as it doesn't have a message type.
        """
        self.send("\x00"*4)
            
    def send_choke(self):
        """
        Tell the peer that this client can't handle any more messages
        and that all messages received will be ignored.
        """
        self.send_payload(0)
            
    def send_unchoke(self):
        """
        Tell a peer that it won't ignore it any more.
        """
        self.send_payload(1)
            
    def send_interested(self):
        """
        Tell the peer that you're interested in one of it's delicious
        pieces.
        """
        self.send_payload(2)
            
    def send_uninterested(self):
        """
        Tell the peer you've found someone better and there's plenty more
        fish in the sea.
        """
        self.send_payload(3)
        
    def send_have(self, index):
        """
        Tell the peer that this client now has this piece.
        :param index: Index of the piece you want to tell the peer is
            here.
        """
        payload = pack(">I", index)
        self.send_payload(4, payload)
        
    def send_bitfield(self):
        """
        Tell the peer all the pieces you have in a really compact form.
        """
        # the pieces are compacted in sequence in to bits
        field = BitArray(
            list(map(
                lambda sha, piece : sha == piece.digest,
                list(self.torrent.pieces.items())
            ))
        )
        self.send_payload(5, field.tobytes())
            
    def send_request(self, index, begin, length=None):
        """
        Request a block (rather than piece) from the peer, using the
        location and start point of the piece. The BitTorrent protocol
        uses a request/send method of downloading, so this asks a peer
        to send me a piece. It may work, it may not.
        :param index: Integer index of the piece being requested.
        :param begin: Integer of the byte that represents the block to
            download.
        :param length: Optional integer for how much to send, can also
            work it out itself.
        """
        if length is None:
            length = self.BLOCK_SIZE
        payload = pack(">III", index, begin, length)
        self.send_payload(6, payload)
            
    def send_piece(self, index, begin, data=None):
        """
        Send a requested piece to the peer, using the location and start
        point of the piece. Can work out what to send itself or can send
        whatever you like.
        :param index: Integer index of the piece was requested.
        :param begin: Integer of the byte that represents the block to
            upload.
        """
        if data is None:
            data = list(self.torrent.pieces.values())[index] \
                [begin:self.BLOCK_SIZE]
        header = pack(">II", index, begin)
        self.send_payload(7, header + data)
        self.uploaded += len(data)
            
    def send_cancel(self, index, begin, length=None):
        """
        Tells the peer that any pieces that have been requested can now
        be ignored and not sent to this client. The index, begin, and
        possibly length are used as identifiers to find that request and
        remove it from the queue of requests.
        :param index: Integer index of the piece was requested.
        :param begin: Integer of the byte that represents the block to
            upload.
        :param length: Integer representing the size of the block that
            was requested. If not known, will guess.
        """
        if length is None:
            length = self.BLOCK_SIZE
        payload = pack(">III", index, begin, length)
        self.send_payload(8, header + payload)
            
    def send_port(self):
        """
        Send when we're talking DHT, but for now, not used.
        :param length: Not used here.
        """
        pass

peer.py
///

#    This file is part of PYTHorrent.
#
#    PYTHorrent is free software: you can redistribute it and/or modify
#    it under the terms of the GNU General Public License as published by
#    the Free Software Foundation, either version 3 of the License, or
#    (at your option) any later version.
#
#    PYTHorrent is distributed in the hope that it will be useful,
#    but WITHOUT ANY WARRANTY; without even the implied warranty of
#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#    GNU General Public License for more details.
#
#    You should have received a copy of the GNU General Public License
#    along with PYTHorrent.  If not, see <http://www.gnu.org/licenses/>.

import socket
import random
from urllib.parse import urlparse
#from urlparse import urlparse
from struct import unpack
from datetime import datetime, timedelta

from . import splice, BitTorrentException
from .peer import Peer

import requests
from torrentool.bencode import Bencode

import logging

def decode_binary_peers(peers):
    """ Return a list of IPs and ports, given a binary list of peers,
    from a tracker response. """

    peers = splice(peers, 6)	# Cut the response at the end of every peer
    return [(socket.inet_ntoa(p[:4]), decode_port(p[4:])) for p in peers]
    
def decode_port(port):
    """ Given a big-endian encoded port, returns the numerical port. """
    return unpack(">H", port)[0]
    
def store_from_url(url):
    """
    Get a store object based upon the URL used to access that peer
    store.
    :param url: URL used to access the peer store as string.
    """
    url.encode()
    parsed_url = urlparse(url)
    if parsed_url.scheme.startswith("http"):
        return HTTPTracker
    if parsed_url.scheme.startswith("udp"):
        return UDPTracker
    # DHT would go here
    
class TrackerException(Exception):
    pass
    
class Tracker(object):
    PEER = Peer
    TRACKER_INTERVAL = 1800 # seconds
    
    def __init__(self, tracker_url, torrent):
        """
        Represents a tracker to get a list of peers from.
        :param tracker_url: URL to access the tracker at.
        :param torrent: Torrent object to hook back in to details needed
            to access the tracker.
        """
        self.tracker_url = tracker_url
        self.torrent = torrent
        self._peers = {}
        self.tracker_interval = self.TRACKER_INTERVAL
        self.last_run = None
    
    @property
    def now(self):
        """
        Use a consistent datetime now value.
        Returns DateTime object with value of now.
        """
        return datetime.utcnow()
        
    @property
    def delta(self):
        """
        Shortcut to get TimeDelta for delay between tracker requests.
        Returns TimeDelta object representing delay between tracker
            requests.
        """
        return timedelta(seconds=self.tracker_interval)
        
    @property
    def next_run(self):
        """
        Shortcut to find out when the tracker request should be next
        run.
        Returns DateTime object representing the next time the tracker
            wants an announce from the client as a minimum.
        """
        return self.last_run + self.delta
    
    @property
    def ok_to_announce(self):
        """
        Returns True when enough time has elapsed since last_run to
        announce again, and False if it has not, or has not been set,
        but also handles if has never been run.
        """
        if self.last_run is None:
            return True
        return self.now > self.next_run
        
    @property
    def peers(self):
        """
        If it is ok to get new peers will update the list of peers, then
        will return the list of peers.
        Returns list of Peer objects.
        """
        if self.ok_to_announce:
            logging.debug("Announcing to {url}".format(
                url=self.tracker_url
            ))
            self._peers.update(self.announce())
        return self._peers
        
    def announce(self):
        raise NotImplemented("Use this method to announce to tracker")
        
class HTTPTracker(Tracker):
    @property
    def announce_payload(self):
        """
        Returns the query params used to announce client to tracker.
        Returns dictionary of query params.
        """
        return {
            "info_hash" : self.torrent.info_hash,
            "peer_id" : self.torrent.peer_id,
            "port" : self.torrent.port,
            "uploaded" : self.torrent.uploaded,
            "downloaded" : self.torrent.downloaded,
            "left" : self.torrent.remaining,
            "compact" : 1
        }
    
    def announce(self):
        """
        Announces client to tracker and handles response.
        Returns dictionary of peers.
        """
        
        # Send the request
        try:
            response = requests.get(
                self.tracker_url,
                params=self.announce_payload,
                allow_redirects=False
            )
            logging.debug("Tracker URL: {0}".format(response.url))
        except requests.ConnectionError as e:
            logging.warn(
                "Tracker not found: {0}".format(
                    self.tracker_url
                )
            )
            return {}
            
        if response.status_code < 200 or response.status_code >= 300:
            raise BitTorrentException(
                "Tracker response error '{0}' for URL: {1}".format(
                    response.content,
                    response.url
                )
            )
            
        self.last_run = self.now
        
        decoded_response = Bencode.decode(response.content)
        
        self.tracker_interval = decoded_response.get(
            'interval',
            self.TRACKER_INTERVAL
        )
        logging.debug("Tracking interval set to: {interval}".format(
            interval=self.tracker_interval
        ))
        
        if "failure reason" in decoded_response:
            raise BitTorrentException(decoded_response["failure reason"])
    
        if "peers" in decoded_response: # ignoring `peer6` (ipv6) for now
            peers = decode_binary_peers(decoded_response['peers'])
        else:
            peers = []
        
        return dict([(
                    hostport, self.PEER(hostport, self.torrent)
                ) for hostport in peers])

class UDPTracker(Tracker):
    DEFAULT_CONNECTION_ID = 0x41727101980
    
    class EACTION:
        NEW_CONNECTION = 0x0
        SCRAPE = 0x2
        ERROR = 0x3
    
    @staticmethod
    def generate_transaction_id():
        return int(random.randrange(0, 255))
    
    def __init__(self, *args, **kwargs):
        self.tracker_url_parts = urlparse(self.tracker_url)
        self._socket = None
        self._connection = None
        
    @property
    def socket(self):
        if self._socket is None:
            self._socket = socket.socket(
                socket.AF_INET,
                socket.SOCK_DGRAM
            )
        return self._socket
        
    @property
    def connection(self):
        if self._connection is None:
            ip = socket.gethostbyname(self.tracker_url_parts.hostname)
            self._connection = (ip, self.tracker_url_parts.port)
        return self._connection
    
    def announce(self):
        connection_id = self.connection_request()
        peers, leechers, complete = self.scrape_request(connection_id)
        
    def generate_message(self,
        connection_id,
        action,
        transaction_id,
        payload=""
    ):
        buf = struct.pack("!q", connection_id)
        buf += struct.pack("!i", action)
        buf += struct.pack("!i", transaction_id)
        buf += payload
        return buf
        
    def send(self, msg):
        self.socket.sendto(msg, self.connection);
        
    def connection_request(self):
        transaction_id = self.generate_transaction_id()
        connection_request = self.generate_message(
            self.DEFAULT_CONNECTION_ID,
            self.EACTION.NEW_CONNECTION,
            transaction_id
        )
        
        self.send(connection_request)
        
        return parse_connection_response(transaction_id)

        
    def parse_connection_response(self, transaction_id):
        buf = self.socket.recvfrom(2048)[0]
        
        if len(buf) < 16:
            raise TrackerException(
                "Wrong response length getting connection id: " \
                "{0}".format(len(buf))
            )
        action = struct.unpack_from("!i", buf)[0]

        res_transaction_id = struct.unpack_from("!i", buf, 4)[0]
        if res_transaction_id != transaction_id:
            raise TrackerException("Transaction ID doesnt match in " \
                "connection response during connection request. " \
                "Expected {local}, got {response}".format(
                    local = transaction_id,
                    response = res_transaction_id
                )
            )

        if action == self.EACTION.NEW_CONNECTION:
            connection_id = struct.unpack_from("!q", buf, 8)[0] 
            return connection_id
        elif action == self.EACTION.ERROR:		
            error = struct.unpack_from("!s", buf, 8)
            raise TrackerException("Error while trying to get a " \
                "connection response: {0}".format(error))

    def scrape_request(self, connection_id):
        transaction_id = self.generate_transaction_id()
        request = self.generate_message(
            connection_id,
            self.EACTION.SCRAPE,
            transaction_id,
            struct.pack("!20s", self.torrent.info_hash)
        )
        self.send(request)
        
        return parse_scrape_response(transaction_id, info_hash)
    
    def parse_scrape_response(self, sent_transaction_id, info_hash):
        buf = self.socket.recvfrom(2048)[0]
        if len(buf) < 16:
            raise TrackerException(
                "Wrong response length while scraping: {0}".format(
                    len(buf)
                )
            )
                
        action = struct.unpack_from("!i", buf)[0]
        res_transaction_id = struct.unpack_from("!i", buf, 4)[0]	
        if res_transaction_id != sent_transaction_id:
            raise TrackerException("Transaction ID doesnt match in " \
                "connection response during scrape. Expected " \
                "{local}, got {response}".format(
                    local = transaction_id,
                    response = res_transaction_id
                )
            )
                
        if action == self.EACTION.SCRAPE:
            offset = 8; 
            seeds = struct.unpack_from("!i", buf, offset)[0]
            offset += 4
            complete = struct.unpack_from("!i", buf, offset)[0]
            offset += 4
            leeches = struct.unpack_from("!i", buf, offset)[0]
            offset += 4	
            return seeds, leeches, complete
            
        elif action == self.EACTION.ERROR:
            #an error occured, try and extract the error string
            error = struct.unpack_from("!s", buf, 8)
            raise TrackerException("Error while scraping: %s" % error)
    

///
peer_stores.py