Better Handling for Directories/Leaves

[PsypherPunk]

Likely issue here for sites where a 'directory' contains content. i.e. If http://www.bl.uk/subdirectory/page.html has been created and then we come across http://www.bl.uk/subdirectory/ the latter will be skipped as the node already exists as a directory.

Similarly, is_leaf() is used to determine whether something is a file. This means for the above a 'dummy' leaf needs to be created to hold the content.