Initial commit

Author: eliemichel

.gitignore

@@ -0,0 +1,3 @@
+__pycache__/
+*.pyc
+/Builds/

README.md

@@ -0,0 +1,98 @@
+![Lily Surface Scrapper](doc/imported.png)
+
+Lily Surface Scrapper
+=====================
+
+There are many sources for getting PBR textures on the Internet, but it is always a repetitive task to setup the shader once in Blender. Some sources provide an add-on to automatically handle this, but it remains painful to install a new add-on for each of them, and learn over how they slightly differ from the other one.
+
+LilySurfaceScrapper suggest a very intuitive and unified workflow: browse your favorite library in your browser. Once you have made your choice, just copy the URL of the page and paste it in Blender. The script will prompt for potential variants if some are detected, then download the maps and setup the material.
+
+This add-on has been designed to make it very easy to add new sources. You can just add a script in the `Scrappers/` directory, it will be automatically detected. See bellow for more information.
+
+## Installation
+
+Download the [last release](https://github.com/eliemichel/LilySurfaceScrapper/releases), then in Blender, go to `Edit > Preferences`, `Add-on`, `Install`, browse to the zip file.
+
+![Add-on loaded in the User Preferences](doc/addon.png)
+
+## Usage
+
+ 1. Browse the website of the source of your choice, until you find a texture, say https://cc0textures.com/view.php?tex=Metal01
+ 2. Copy this URL
+
+ 3. If you just opened Blender and did not save yet, the add-on is not available:
+
+![Add-on loaded in the User Preferences](doc/not-saved.png)
+
+ 4. Save your file, this is needed because the textures will be loaded in a directory called `LilySurface` next to it.
+
+![Add-on loaded in the User Preferences](doc/saved.png)
+
+ 5. Paste the URL of the texture:
+
+![Add-on loaded in the User Preferences](doc/paste-url.png)
+
+ 6. Select the variant, if there is more than one available on the page:
+
+![Add-on loaded in the User Preferences](doc/select-variant.png)
+
+You can then browse the downloaded files next to your blend. Note that they are not downloaded twice if you use the same URL and variant again.
+
+![Add-on loaded in the User Preferences](doc/files.png)
+
+
+## Supported sources
+
+There are currently two supported sources:
+
+ - cgbookcase: https://www.cgbookcase.com/
+ - CC0Textures: https://cc0textures.com/
+
+## Adding new sources
+
+I tried to make it as easy as possible to add new sources of data. The only thing to do is to add a python file in `Scrappers/` and define in it a class deriving from `AbstractScrapper`.
+
+You can start from a copy of [`Cc0texturesScrapper.py`](Scrappers/Cc0texturesScrapper.py) or [`CgbookcaseScrapper.py`](Scrappers/CgbookcaseScrapper.py). The former loads a zip and extracts maps while the second looks for a different URL for each map (base color, normal, etc.).
+
+The following three methods are required:
+
+### canHandleUrl(cls, url)
+
+A static method (just add `@staticmethod` before its definition) that returns `True` only if the scrapper recognizes the URL `url`.
+
+### fetchVariantList(self, url)
+
+Return a list of variant names (a list of strings), to prompt the user. This is useful when a single page provides several versions of the material, like different resolutions (2K, 4K, etc.) or front/back textures.
+
+This method may save info like the html page in `self`, to reuse it in `fetchVariant`.
+
+### fetchVariant(self, variant_index, material_data)
+
+Scrap the information of the variant numbered `variant_index`, and write it to `material_data`. The following fields of `material_data` can be filled:
+
+ - `material_data.name`: The name of the texture, typically prefixed by the source, followed by the texture name, then the variant name.
+ - `material_data.maps['baseColor']`: The path to the base color map, or None
+ - `material_data.maps['normal']`: The path to the normal map, or None
+ - `material_data.maps['opacity']`: The path to the opacity map, or None
+ - `material_data.maps['roughness']`: The path to the roughness map, or None
+ - `material_data.maps['metallic']`: The path to the metallic map, or None
+
+To implement these methods, you can rely on the following utils:
+
+### self.fetchHtml(url)
+
+Get the url as a [lxml.etree](https://lxml.de/tutorial.html) object. You can then call the `xpath()` method to explore the page using the very convenient [xpath synthax](https://en.wikipedia.org/wiki/XPath#Examples).
+
+### fetchImage(self, url, material_name, map_name)
+
+Get an image from the URL `url`, place it in a directory whose name is generated from the `material_name`, and call the map `map_name` + extension (if an extension is explicit in the URL). The function returns the path to the downloaded texture, and you can directly provide it to `material_data.maps[...]`.
+
+### fetchZip(self, url, material_name, zip_name)
+
+Get a zip file from the URL `url`. This works like `fetchImage()`, returning the path to the zip file. You can then use the [zipfile](https://docs.python.org/3/library/zipfile.html) module, like [`Cc0texturesScrapper.py`](Scrappers/Cc0texturesScrapper.py) does.
+
+## TODO
+
+ - Handle bump map
+ - Handle AO map
+ - Include `lxml` more properly. I don't know the idiomatic way to package such a dependency in a Blender addon.

blender/LilySurfaceScrapper/CyclesMaterialData.py

@@ -0,0 +1,102 @@
+# Copyright (c) 2019 Elie Michel
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the “Software”), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# The Software is provided “as is”, without warranty of any kind, express or
+# implied, including but not limited to the warranties of merchantability,
+# fitness for a particular purpose and noninfringement. In no event shall
+# the authors or copyright holders be liable for any claim, damages or other
+# liability, whether in an action of contract, tort or otherwise, arising from,
+# out of or in connection with the software or the use or other dealings in the
+# Software.
+#
+# This file is part of LilySurfaceScrapper, a Blender add-on to import materials
+# from a single URL
+
+import os
+import bpy
+from mathutils import Vector
+from .MaterialData import MaterialData
+
+class CyclesMaterialData(MaterialData):
+    def _getCyclesImage(self, imgpath):
+        """Avoid reloading an image that has already been loaded"""
+        for img in bpy.data.images:
+            if os.path.abspath(img.filepath) == os.path.abspath(imgpath):
+                return img
+        return bpy.data.images.load(imgpath)
+    
+    def _autoAlignNodes(self, root):
+        def makeTree(node):
+            descendentCount = 0
+            children = []
+            for i in node.inputs:
+                for l in i.links:
+                    subtree = makeTree(l.from_node)
+                    children.append(subtree)
+                    descendentCount += subtree[2] + 1
+            return node, children, descendentCount
+
+        tree = makeTree(root)
+
+        def placeNodes(tree, rootLocation, xstep = 400, ystep = 250):
+            root, children, count = tree
+            root.location = rootLocation
+            childLoc = rootLocation + Vector((-xstep, ystep * count / 2.))
+            acc = 0.25
+            for child in children:
+                print(child[0].name, acc)
+                acc += (child[2]+1)/2.
+                placeNodes(child, childLoc + Vector((0, -ystep * acc)))
+                acc += (child[2]+1)/2.
+
+        placeNodes(tree, Vector((0,0)))
+
+
+    def createMaterial(self):
+        mat = bpy.data.materials.new(name=self.name)
+        mat.use_nodes = True
+        nodes = mat.node_tree.nodes
+        links = mat.node_tree.links
+        principled = nodes["Principled BSDF"]
+        mat_output = nodes["Material Output"]
+        principled.inputs["Roughness"].default_value = 1.0
+        # Translate our internal map names into cycles principled inputs
+        input_tr = {
+            'baseColor': 'Base Color',
+            'normal': 'Normal',
+            'roughness': 'Roughness',
+            'metallic': 'Metallic',
+            'opacity': '<custom>',
+        }
+        for input, img in self.maps.items():
+            if img is None or input not in input_tr:
+                continue
+            texture_node = nodes.new(type="ShaderNodeTexImage")
+            texture_node.image = self._getCyclesImage(img)
+            texture_node.color_space = 'COLOR' if input == 'baseColor' else 'NONE'
+            if input == 'opacity':
+                transparence_node = nodes.new(type="ShaderNodeBsdfTransparent")
+                mix_node = nodes.new(type="ShaderNodeMixShader")
+                links.new(texture_node.outputs[0], mix_node.inputs[0])
+                links.new(transparence_node.outputs[0], mix_node.inputs[1])
+                links.new(principled.outputs[0], mix_node.inputs[2])
+                links.new(mix_node.outputs[0], mat_output.inputs[0])
+            elif input == "normal":
+                normal_node = nodes.new(type="ShaderNodeNormalMap")
+                links.new(texture_node.outputs["Color"], normal_node.inputs["Color"])
+                links.new(normal_node.outputs["Normal"], principled.inputs["Normal"])
+            else:
+                links.new(texture_node.outputs[0], principled.inputs[input_tr[input]])
+
+        self._autoAlignNodes(mat.node_tree.nodes['Material Output'])
+
+        return mat

blender/LilySurfaceScrapper/MaterialData.py

@@ -0,0 +1,108 @@
+# Copyright (c) 2019 Elie Michel
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the “Software”), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# The Software is provided “as is”, without warranty of any kind, express or
+# implied, including but not limited to the warranties of merchantability,
+# fitness for a particular purpose and noninfringement. In no event shall
+# the authors or copyright holders be liable for any claim, damages or other
+# liability, whether in an action of contract, tort or otherwise, arising from,
+# out of or in connection with the software or the use or other dealings in the
+# Software.
+#
+# This file is part of LilySurfaceScrapper, a Blender add-on to import materials
+# from a single URL
+
+import os
+
+from .Scrappers.AbstractScrapper import AbstractScrapper
+from .settings import TEXTURE_DIR, UNSUPPORTED_PROVIDER_ERR
+
+# dirty but useful, for one to painlessly write scrapping class
+# and just drop them in the scrappers dir
+def makeScrappersList():
+    import importlib
+    scrappers_names = []
+    scrappers_dir = os.path.join(os.path.dirname(os.path.realpath(__file__)), "Scrappers")
+    for f in os.listdir(scrappers_dir):
+        if f.endswith(".py") and os.path.isfile(os.path.join(scrappers_dir, f)):
+            scrappers_names.append(f[:-3])
+    scrappers = []
+    for s in scrappers_names:
+        module = importlib.import_module('.Scrappers.' + s, package='LilySurfaceScrapper')
+        for x in dir(module):
+            if x == 'AbstractScrapper':
+                continue
+            m = getattr(module, x)
+            if isinstance(m, type) and issubclass(m, AbstractScrapper):
+                scrappers.append(m)
+    return scrappers
+all_scrappers = makeScrappersList()
+
+class MaterialData():
+    """Internal representation of materials, responsible on one side for
+    scrapping texture providers and on the other side to build blender materials.
+    This class must not use the Blender API. Put Blender related stuff in subclasses
+    like CyclesMaterialData."""
+    
+    @classmethod
+    def makeScrapper(cls, url):
+        global all_scrappers
+        for S in all_scrappers:
+            if S.canHandleUrl(url):
+                return S()
+        return None
+    
+    def __init__(self, url, texture_root=""):
+        """url: Base url to scrap
+        texture_root: root directory where to store downloaded textures
+        """
+        self.url = url
+        self.name = "Name"
+        self.error = None
+        self.texture_root = texture_root
+        self.maps = {
+            'baseColor': None,
+            'normal': None,
+            'opacity': None,
+            'roughness': None,
+            'metallic': None,
+        }
+        self._variants = None
+        self._scrapper = MaterialData.makeScrapper(url)
+        if self._scrapper is None:
+            self.error = UNSUPPORTED_PROVIDER_ERR
+        else:
+            self._scrapper.texture_root = texture_root
+        
+    def getVariantList(self):
+        if self.error is not None:
+            return None
+        if self._variants is not None:
+            return self._variants
+        self._variants = self._scrapper.fetchVariantList(self.url)
+        if self._variants is None:
+            self.error = self.scrapper.error
+        return self._variants
+
+    def selectVariant(self, variant_index):
+        if self.error is not None:
+            return False
+        if self._variants is None:
+            self.getVariantList()
+        if not self._scrapper.fetchVariant(variant_index, self):
+            return False
+        return True
+    
+    def createMaterial(self):
+        """Implement this in derived classes"""
+        raise NotImplementedError
+