notebooks/020_Python_files.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img alt='UCL' src=\"images/ucl_logo.png\" align='center'>\n",
    "\n",
    "\n",
    "[<img src=\"images/noun_post_2109127.svg\" width=\"50\" align='right'>](021_URLs.ipynb)\n",
    "[<img src=\"images/noun_pre_2109128.svg\" width=\"50\" align='right'>](018_Running_Python.ipynb)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 020 Files, Streams and related issues"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Introduction\n",
    "\n",
    "\n",
    "\n",
    "### Purpose\n",
    "\n",
    "In this session, we will learn about files and streams. We will introduce the standard Python library [`pathlib`](https://docs.python.org/3/library/pathlib.html) which is how we deal with file paths. We will also cover opening and closing files, and some simple read- and write-operations.\n",
    "\n",
    "### Prerequisites\n",
    "\n",
    "You will need some understanding of the following:\n",
    "\n",
    "\n",
    "* [001 Using Notebooks](001_Notebook_use.ipynb)\n",
    "* [002 Unix](002_Unix.ipynb) with a good familiarity with the UNIX commands we have been through.\n",
    "* [003 Getting help](003_Help.ipynb)\n",
    "* [004_Accounts](004_Accounts.ipynb)\n",
    "* [005_Packages](005_Packages.ipynb)\n",
    "* [010 Variables, comments and print()](010_Python_Introduction.ipynb)\n",
    "* [011 Data types](011_Python_data_types.ipynb) \n",
    "* [012 String formatting](012_Python_strings.ipynb)\n",
    "* [013_Python_string_methods](013_Python_string_methods.ipynb)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data resources\n",
    "\n",
    "### Resource location\n",
    "\n",
    "We store information on a computer or 'on the web' in files, or file-like resources. We will use the term 'file' below to mean either of these concepts, other than specific issues relating to particular types of file/resource.\n",
    "\n",
    "To get information from files, we need to be able to specify some **address** for the file/resource location, along with some way of interacting with the file. These concepts are captured in the idea of a [URI](https://en.wikipedia.org/wiki/Uniform_Resource_Identifier) (Uniform Resource Indicator). You will most likely have come across the related idea of a [Uniform Resource Locator (URL)](https://en.wikipedia.org/wiki/URL), which is a URL such as [https://www.geog.ucl.ac.uk/people/academic-staff/philip-lewis](https://www.geog.ucl.ac.uk/people/academic-staff/philip-lewis)\n",
    "that gives:\n",
    "\n",
    "* the location of the resource: `people/academic-staff/philip-lewis`\n",
    "* the access and interpretation protocol: [`https`](https://en.wikipedia.org/wiki/HTTPS) (secure [`http`](https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol))\n",
    "* the network domain name: [`www.geog.ucl.ac.uk`](https://www.geog.ucl.ac.uk)\n",
    "\n",
    "When we visit this URL using an appropriate tool such as a browser, the tool can access and interpret the information in the resource: in this case, interpret the [html code](https://www.w3schools.com/html) in the file pointed to by the URL.\n",
    "\n",
    "Similarly, we will be used to the idea of accessing `files` on the computer. These may be in the local file system, or on some network or cloud storage that might be accessible from the local file system. An example of such a file would be some Python code file such as \n",
    "[`geog0111/helloWorld.py`](http://localhost:8888/edit/notebooks/geog0111/helloWorld.py).\n",
    "\n",
    "Whilst there are low-level tools we can use to access these various types of information, it is better from a programmer's point of view to be able to do this in a consistent and object-oriented manner: as a programmer, shouldn't really need distinguish greatly between a file accessed by a URL and one on the local file system. And if you do, you should be able to use much the same sort of methods to do much the same sort of tasks.\n",
    "\n",
    "It doesn't quite work like that yet in Python, but the [`pathlib`](https://docs.python.org/3/library/pathlib.html) package goes a long way towards that for local files. This doesn't yet exist for URLs in standard Python, but is implemented to a large extent by the package [`urlpath`](https://github.com/chrono-meter/urlpath). These are the tools we will be learning to use to access and manipulate files and similar objects.\n",
    "\n",
    "We will first introduce [`pathlib`](https://docs.python.org/3/library/pathlib.html) to learn how to deal with local files."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### binary and text data\n",
    "\n",
    "It is useful at this point to distinguish text (ASCII) and binary resources. Text resources are in a human-readable format, so you can just directly 'look at' the file contents to see what is there. Examples of this are [csv-format](https://en.wikipedia.org/wiki/Comma-separated_values) files, [html pages](https://en.wikipedia.org/wiki/HTML), [yaml](https://en.wikipedia.org/wiki/YAML) or [json](https://en.wikipedia.org/wiki/JSON) configuration files, and even these [Jupyter notebooks](https://jupyter.org/). \n",
    "\n",
    "Information in text files is, as noted, easily readable: you can open the file in any text editor to see the contents. However, for large datasets, and datasets with particular structures (e.g. on grids), it is very inefficient.\n",
    "\n",
    "Typical **binary** files include (most) image data and compressed data (e.g. [zip](https://en.wikipedia.org/wiki/Zip_(file_format)) files), where size is more important, as well as encrypted data, where human-readability might not be desirable.\n",
    "\n",
    "Text files designed for different purposes will mostly have some format definition or conventions. This means that although we can easily scan e.g. a [json file](https://json.org/example.html`), there is a set way of laying such files out, so we can interpret them by eye or in software.\n",
    "\n",
    "Binary files will also have some defined format, but in this case, the user is generally forced to use some software reader to interpret the data correctly.\n",
    "\n",
    "When you wish to access some file or dataset, you will generally know which format the file will be in, so can target an appropriate reader. Quite often though, we have to we might need to examine a dataset before deciding on some parameters for an interpreter, particularly with text datasets. \n",
    "\n",
    "In these notes, we will learn how to access and use binary and text datasets in Python, from the local file system and online from URLs over the next few sessions.  In a URI sense, there is much in common between online and local files\n",
    "\n",
    "\n",
    "We will be also using `yaml`, `json` and [`pandas`](https://pandas.pydata.org/) packages for reading particular text file types, and introducing [`gdal`](https://gdal.org/python/) for binary image files."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `Path`, `import`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have come across the idea of packages in [005 Packages](005_Packages.ipynb). `Path` is part of the `pathlib` package, so in any Python codes, we first must import this into our workspace:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This imports the module `Path` from the library `pathlib`.\n",
    "\n",
    "We may repeat this `import` in the codes below, for illustration purposes. You need only import it once in any file or session though. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `posix`\n",
    "\n",
    "The main class we will use from `pathlib` is `Path`. This provides an object-oriented way of dealing with files and paths, that is portable to any operating system. Recall from [002 Unix](002_Unix.ipynb) that unix-like operating systems use `posix` filenames, separated by forward slash `/` (or just *slash*). Windows uses a backslash `\\`. But *we* want to write Python codes that are generic and shouldn't need to greatly worry about such issues. Using `Path` greatly helps in this regard, though there will always the occasional time we do need to distinguish `posix` and non-`posix` systems.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Common `Path` methods"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can start with a table of [commonly-used methods](https://stackabuse.com/introduction-to-the-python-pathlib-module/#:~:text=The%20Pathlib%20module%20can%20deal,(usually%20the%20current%20directory).) using `Path` and give their [Unix equivalents](002_Unix.ipynb), most of which we have already learned.\n",
    "\n",
    "|command| unix-equivalent | purpose|\n",
    "|---|---|---|\n",
    "|`Path.cwd()`| `pwd` |Return path object representing the current working directory|\n",
    "|`Path.home()`| `~`| Return path object representing the home directory|\n",
    "|`Path.stat()`| `ls -l`† | return info about the path. File size is `Path.stat().st_size` |\n",
    "|`Path.chmod()`| `chmod` | change file mode and permissions|\n",
    "|`Path.glob(pattern)`| `ls *` | Glob the pattern given in the directory that is represented by the path, yielding matching files of any kind|\n",
    "|`Path.mkdir()`| `mkdir` | to create a new directory at the given path|\n",
    "|`Path.rename()`| `mv` | Rename a file or directory to the given target|\n",
    "|`Path.rmdir()`| `rmdir` | Remove the empty directory|\n",
    "|`Path.unlink()`| `rm`‡ | Remove the file or symbolic link|\n",
    "|`Path.touch()` | `touch` | create (zero-sized) or update a file |\n",
    "|`Path.parent` | `..` ✟ | The parent object (one level up in directory tree) |\n",
    "\n",
    "\n",
    "Some other common utilities:\n",
    "\n",
    "|command|  purpose|\n",
    "|---|---|\n",
    "|`Path.as_posix()`| Return the object as a posix filename string |\n",
    "|`Path.exists()` | `True` if exists, `False` if not|\n",
    "|`Path.parent` |  the parent object (one level up in directory tree), or the directory a file is in |\n",
    "|`Path.name` | the file name |\n",
    "|`Path.is_file()` | `True` if object is a file |\n",
    "|`Path.is_dir()` | `True` if object is a directory |\n",
    "\n",
    "We will also deal with opening and closing files and moving information around, using these functions:\n",
    "\n",
    "\n",
    "|command|  purpose|\n",
    "|---|---|\n",
    "|`Path.open()`| open a file and return a file descriptor|\n",
    "|`Path.read_text()`|  read text|\n",
    "|`Path.write_text()`| write text|\n",
    "|`Path.read_bytes()`| read byte data|\n",
    "|`Path.write_bytes()`| write byte data|\n",
    "\n",
    "\n",
    "\n",
    "##### Notes to table\n",
    "\n",
    "† Really, `Path.stat()` equates to the `unix` command `stat`, but this contains the information we access using `ls -l`.\n",
    "\n",
    "‡ You can use `shutil.rmtree()` to do recursive delete, if you really need to.\n",
    "\n",
    "✟ `..` isn't exactly the same as `Path.parent`, but expresses a similar idea. So if you had a directory `foo/bar`, then `foo/bar/..` would be the parent i.e. `foo`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we are already familiar with most of these commands in `unix`, we can get straight into using them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I am in directory /Users/philiplewis/Documents/GitHub/geog0111/notebooks\n",
      "My home is /Users/philiplewis\n"
     ]
    }
   ],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "print(f'I am in directory {Path.cwd()}')\n",
    "print(f'My home is {Path.home()}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Really, the object `Path.home()` is a `Path` object. When we direct `print()` to print it, it converts it to a string. If you need to convert a `Path` object to a string in any other context (e.g. passing to some function that needs a filename as a string), then use `Path.as_posix()` to convert:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "My home is /Users/philiplewis\n",
      "<class 'pathlib.PosixPath'>\n",
      "<class 'str'>\n"
     ]
    }
   ],
   "source": [
    "# print posix path as string\n",
    "myhome=Path.home()\n",
    "print(f'My home is {myhome.as_posix()}')\n",
    "\n",
    "# Path object \n",
    "print(type(myhome))\n",
    "# convert to string\n",
    "print(type(myhome.as_posix()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To keep the filenames generic between different operating systems, we can form a filename from a list using `Path()`, so `Path('bin','README')` would refer to the filename `bin/README` on a `posix` system, and `bin\\README` on Windows. However, this is interpreted the same as just using `bin/README` which will often be clearer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "they are the same\n"
     ]
    }
   ],
   "source": [
    "assert Path('bin','README') == Path('bin/README')\n",
    "print('they are the same')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use the function `exists` to check if a file exists:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Does bin/README exist? True\n"
     ]
    }
   ],
   "source": [
    "readme_file = Path('bin/README')\n",
    "print(f'Does {readme_file} exist? {readme_file.exists()}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To add a sub-directory or file to a `Path` object, we can use the list form, or we can just specify the separator '/' between the components, provided the first term is a `Path` object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "starting from the relative directory \"bin\": bin\n",
      "README file is: bin/README\n",
      "README file is: bin/README\n"
     ]
    }
   ],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "# the relative directory bin\n",
    "bindir = Path('bin')\n",
    "print(f'starting from the relative directory \"bin\": {bindir}')\n",
    "\n",
    "# add README to the path\n",
    "readme_file = Path(bindir,'README')\n",
    "print(f'README file is: {readme_file}')\n",
    "\n",
    "# another way to do it using /\n",
    "readme_file = bindir/'README'\n",
    "print(f'README file is: {readme_file}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "solution2": "hidden",
    "solution2_first": true
   },
   "source": [
    "#### Exercise 1\n",
    "\n",
    "There is a file called `environment.yml` in the directory `copy`.\n",
    "\n",
    "* use `Path` to generate the a variable `copy_dir` containing the pathname of the `copy` directory\n",
    "* create a variable `env_file` which adds add the file `environment.yml` to this \n",
    "* check to see if the file exists"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "solution2": "hidden"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "does copy/environment.yml exist? True\n"
     ]
    }
   ],
   "source": [
    "from pathlib import Path\n",
    "# ANSWER\n",
    "\n",
    "# There is a file called environment.yml in the directory copy.\n",
    "# use Path to generate the a variable copy_dir containing the \n",
    "# pathname of the copy directory\n",
    "copy_dir = Path('copy')\n",
    "\n",
    "# create a variable env_file which adds add the file \n",
    "# environment.yml to this\n",
    "env_file = copy_dir / 'environment.yml'\n",
    "# or\n",
    "env_file = Path(copy_dir,'environment.yml')\n",
    "\n",
    "# check to see if the file exists\n",
    "print(f'does {env_file} exist? {env_file.exists()}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating and deleting files and directories\n",
    "\n",
    "#### Directories\n",
    "\n",
    "To create a directory, use:\n",
    "\n",
    "    Path().mkdir()\n",
    "    \n",
    "We can use the options:\n",
    "\n",
    "    Path().mkdir(parents=True,exist_ok=True)\n",
    "    \n",
    "which will make all of the sub-directories needed, and also no fail if the directory already exists. This is often the safest way to make a directory in your code, provided you are sure that your path will be reasonable.\n",
    "\n",
    "To remove an empty directory, use:\n",
    "\n",
    "    Path().rmdir()\n",
    "    \n",
    "If we need to test to see if the object we want to delete really is a directory, we can use `Path.is_dir`. If we want to first check it exists, then use `Path.exists()` to make the code more robust:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# make a temp directory\n",
    "tmp = Path('tmp')\n",
    "tmp.mkdir(parents=True,exist_ok=True)\n",
    "\n",
    "# then delete tmp (if it exists and is a dir)\n",
    "tmp.exists() and tmp.is_dir() and tmp.rmdir()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Files\n",
    "\n",
    "To create a blank (zero-sized) file, we can use `Path.touch()`, and then `Path.unlink()` to remove it. \n",
    "\n",
    "We can use `Path.parent` to refer to the parent object, e.g. the directory a file is in, or for a directory, one level up (`..`). So, if we are creating a file for the first time, we might want to make sure that its parent directory exists before doing so (e.g. using `Path.parent.mkdir(parents=True,exist_ok=True)` as above).\n",
    "\n",
    "If we need to test to see if the object we want to delete really is a file (not a directory), we can use `Path.is_file`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "now you see it: True\n"
     ]
    }
   ],
   "source": [
    "tmp = Path('tmp','myfile')\n",
    "\n",
    "# make sure directory (parent) exists\n",
    "tmp.parent.mkdir(parents=True,exist_ok=True)\n",
    "# create zero size file\n",
    "tmp.touch()\n",
    "\n",
    "# call `Path.exists()`\n",
    "print(\"now you see it:\",tmp.exists())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "now you don't: False\n"
     ]
    }
   ],
   "source": [
    "# delete inside if it exists and is a file\n",
    "tmp.exists() and tmp.is_file() and tmp.unlink()\n",
    "\n",
    "# let's make a unix call here to check the file size etc\n",
    "print(\"now you don't:\",tmp.exists())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####  `rm -r` options\n",
    "\n",
    "Note that there is no recursive delete in `pathlib` like `rm -r` in linux: you have to delete files and directories explicitly in `pathlib`. Really, `rm -r` is a really dangerous command to have, as you might accidently delete all sorts of things you didn't mean to. So, don't even go there unless you need to.\n",
    "\n",
    "If you do, you can look on the web, you can find [various solutions to this to make it easier](https://stackoverflow.com/questions/50186904/pathlib-recursively-remove-directory). Maybe the easiest is to use `rmtree` from the package `shutil`, though this doesn't fit so well with our object-oriented principles.\n",
    "\n",
    "    import shutil\n",
    "    shutil.rmtree( '/tmp/mydir' )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "solution2": "hidden",
    "solution2_first": true
   },
   "source": [
    "#### Exercise 2\n",
    "\n",
    "Create a zero-sized file called `hello.txt` in a directory `mystuff`, using `Path` and show that it exists and is a file. Then delete the file and directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "solution2": "hidden"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== now you see it ===\n",
      "Does mystuff/hello.txt exist? True\n",
      "Is mystuff/hello.txt a file? True\n",
      "=== now you don't ===\n",
      "Does mystuff/hello.txt exist? False\n",
      "Is mystuff/hello.txt a file? False\n",
      "=== now you don't ===\n",
      "Does mystuff exist? False\n",
      "Is mystuff a file? False\n"
     ]
    }
   ],
   "source": [
    "from pathlib import Path\n",
    "# ANSWER\n",
    "# import package\n",
    "\n",
    "# Form the Path object for the file\n",
    "myfile = Path('mystuff','hello.txt')\n",
    "\n",
    "# Make sure the parent directory exists\n",
    "myfile.parent.mkdir(parents=True,exist_ok=True)\n",
    "\n",
    "# Create the zero-sized file\n",
    "myfile.touch()\n",
    "\n",
    "# Check it exists and is a file\n",
    "print('=== now you see it ===')\n",
    "print(f'Does {myfile.as_posix()} exist? {myfile.exists()}')\n",
    "print(f'Is {myfile.as_posix()} a file? {myfile.is_file()}')\n",
    "\n",
    "# delete the file \n",
    "myfile.exists() and myfile.is_file() and myfile.unlink()\n",
    "# Check it exists and is a file\n",
    "print(\"=== now you don't ===\")\n",
    "print(f'Does {myfile.as_posix()} exist? {myfile.exists()}')\n",
    "print(f'Is {myfile.as_posix()} a file? {myfile.is_file()}')\n",
    "\n",
    "# delete the directory -- the parent\n",
    "mydir = myfile.parent\n",
    "mydir.exists() and mydir.is_dir() and mydir.rmdir()\n",
    "# Check it exists and is a dir\n",
    "print(\"=== now you don't ===\")\n",
    "print(f'Does {mydir.as_posix()} exist? {mydir.exists()}')\n",
    "print(f'Is {mydir.as_posix()} a file? {mydir.is_dir()}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### File information\n",
    "\n",
    "The file permissions format we are used to from `ls -l` is accessed through `filename.stat().st_mode` but needs to be converted to octal to match to `ls`\n",
    "\n",
    "    oct(filename.stat().st_mode)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "README file is bin/README\n",
      "       size is 16 bytes\n",
      "       mode is 0o100644\n",
      "      owner is philiplewis\n",
      "-rw-r--r--@ 1 philiplewis  staff    16B 12 Dec  2022 bin/README\r\n"
     ]
    }
   ],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "# similar information to ls -l\n",
    "readme=Path('bin','README')\n",
    "print(f'README file is {readme}')\n",
    "print(f'       size is {readme.stat().st_size} bytes')\n",
    "print(f'       mode is {oct(readme.stat().st_mode)}')\n",
    "print(f'      owner is {readme.owner()}')\n",
    "\n",
    "# confirm with unix\n",
    "!ls -lh bin/README"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "solution2": "hidden",
    "solution2_first": true
   },
   "source": [
    "#### Exercise 3\n",
    "\n",
    "Create a zero-sized file in a new directory, and use `Path.stat()` to show it has size 0 bytes. Then tidy up by deleting the file and directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "solution2": "hidden"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The file size of mystuff/hello.txt is 0 bytes\n"
     ]
    }
   ],
   "source": [
    "from pathlib import Path\n",
    "# ANSWER\n",
    "# import package\n",
    "# very similar to exercise 2 but now we need to check the size\n",
    "\n",
    "# Form the Path object for the file\n",
    "myfile = Path('mystuff','hello.txt')\n",
    "\n",
    "# Make sure the parent directory exists\n",
    "myfile.parent.mkdir(parents=True,exist_ok=True)\n",
    "\n",
    "# Create the zero-sized file\n",
    "myfile.touch()\n",
    "\n",
    "# print the file size\n",
    "print(f'The file size of {myfile.as_posix()} is {myfile.stat().st_size} bytes')\n",
    "\n",
    "# delete the file \n",
    "myfile.exists() and myfile.is_file() and myfile.unlink()\n",
    "# delete the directory -- the parent\n",
    "mydir = myfile.parent\n",
    "mydir.exists() and mydir.is_dir() and mydir.rmdir()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `datetime` \n",
    "\n",
    "You have access to  a wide range of information from [`File.stat()`](https://docs.python.org/3/library/os.html#os.stat_result). Some of this relates to time (e.g. time last viewed, accessed etc). These times are given as Epoch times, in seconds since from 1/1/1970:\n",
    "\n",
    "For example, to get the last access time:\n",
    "\n",
    "    st_mtime : Time of most recent content modification expressed in seconds.\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time of most recent modification for bin/README is 1670840781.0\n"
     ]
    }
   ],
   "source": [
    "readme = Path('bin','README')\n",
    "\n",
    "modified = readme.stat().st_mtime\n",
    "print(f'Time of most recent modification for {readme} is {modified}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That is not always the most convenient form. So we can use the `datetime.datetime.fromtimestamp` from the `datetime` package to convert it into something more human-readable:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time of most recent modification for bin/README is 2022-12-12 10:26:21\n"
     ]
    }
   ],
   "source": [
    "from datetime import datetime\n",
    "\n",
    "modified = readme.stat().st_mtime\n",
    "h_modified = datetime.fromtimestamp(modified)\n",
    "\n",
    "print(f'Time of most recent modification for {readme} is {h_modified}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also use `datetime` to tell us the time now, with `datetime.now()`, for comparison:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time now is 2023-06-28 13:26:46.992071\n"
     ]
    }
   ],
   "source": [
    "print(f'Time now is {datetime.now()}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "solution2": "hidden",
    "solution2_first": true
   },
   "source": [
    "#### Exercise 4\n",
    "\n",
    "Use `Path.touch()` to update the modification time for the file `bin/README` and demonstrate that you have done this and that is the same as the current time (now)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "solution2": "hidden"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before touch: time of most recent modification for bin/README is 2022-12-12 10:26:21\n",
      "After touch: time of most recent modification for bin/README is 2023-06-28 13:26:46.996908\n",
      "Now it is 2023-06-28 13:26:46.997758\n"
     ]
    }
   ],
   "source": [
    "from pathlib import Path\n",
    "from datetime import datetime\n",
    "# ANSWER\n",
    "# import packages\n",
    "\n",
    "readme = Path('bin','README')\n",
    "\n",
    "modified = readme.stat().st_mtime\n",
    "h_modified = datetime.fromtimestamp(modified)\n",
    "\n",
    "print(f'Before touch: time of most recent modification for {readme} is {h_modified}')\n",
    "\n",
    "# touch the file\n",
    "readme.touch()\n",
    "modified = readme.stat().st_mtime\n",
    "h_modified = datetime.fromtimestamp(modified)\n",
    "\n",
    "print(f'After touch: time of most recent modification for {readme} is {h_modified}')\n",
    "print(f'Now it is {datetime.now()}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `glob` generators \n",
    "\n",
    "To use a wildcard (or any pattern) to refer to a list of files, we use `Path.glob()` (or `Path.rglob()` for a recursive list). This function returns a generator, which is a special type of list whereby the items in the list are produced on demand (as we use them)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use glob to get a list of filenames in the directory bin \n",
    "# that end with .sh -> pattern *.sh using a wildcard\n",
    "filenames = Path('bin').glob('*.sh')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The generator does not have all of the attributes of a list, so we cannot, for example, know the length, withjout first converting it tot a list. It is intended that you step through it one item at a time. This is usually done in a `for` loop, or similar construct:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "file 0 is bin/notebook-mkdocs.sh\n",
      "file 1 is bin/setup.sh\n",
      "file 2 is bin/fixA.sh\n",
      "file 3 is bin/notebook-run.sh\n",
      "file 4 is bin/link-set.sh\n",
      "file 5 is bin/clean0111.sh\n",
      "file 6 is bin/tidy.sh\n",
      "file 7 is bin/init.sh\n",
      "file 8 is bin/sort-db.sh\n",
      "file 9 is bin/get-datasets.sh\n",
      "file 10 is bin/init0111.sh\n",
      "file 11 is bin/set-course.sh\n",
      "file 12 is bin/howmany.sh\n",
      "file 13 is bin/database.sh\n",
      "file 14 is bin/git-remove-all.sh\n",
      "file 15 is bin/shellMe.sh\n"
     ]
    }
   ],
   "source": [
    "for i,f in enumerate(filenames):\n",
    "    print(f'file {i} is {f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The advantage of a generator is that it will generally need less memory than fully calculating all items in a list. Once we move on to the next item in the generator, any memory used by current item is freed.\n",
    "\n",
    "But is the size of objects is small, you will not notice much impact if you convert the generator to a list. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[PosixPath('bin/notebook-mkdocs.sh'), PosixPath('bin/setup.sh'), PosixPath('bin/fixA.sh'), PosixPath('bin/notebook-run.sh'), PosixPath('bin/link-set.sh'), PosixPath('bin/clean0111.sh'), PosixPath('bin/tidy.sh'), PosixPath('bin/init.sh'), PosixPath('bin/sort-db.sh'), PosixPath('bin/get-datasets.sh'), PosixPath('bin/init0111.sh'), PosixPath('bin/set-course.sh'), PosixPath('bin/howmany.sh'), PosixPath('bin/database.sh'), PosixPath('bin/git-remove-all.sh'), PosixPath('bin/shellMe.sh')]\n"
     ]
    }
   ],
   "source": [
    "filenames = list(Path('bin').glob('*.sh'))\n",
    "print(filenames)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, you can use the [`*args`](https://www.geeksforgeeks.org/args-kwargs-python/) method to pass the results of the generator through to the print function as a set of arguments:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "bin/notebook-mkdocs.sh bin/setup.sh bin/fixA.sh bin/notebook-run.sh bin/link-set.sh bin/clean0111.sh bin/tidy.sh bin/init.sh bin/sort-db.sh bin/get-datasets.sh bin/init0111.sh bin/set-course.sh bin/howmany.sh bin/database.sh bin/git-remove-all.sh bin/shellMe.sh\n"
     ]
    }
   ],
   "source": [
    "print(*Path('bin').glob('*.sh'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's use `glob` now to get the file permissions of each file `n*` in the directory `bin`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "bin/notebook-mkdocs.sh   \n",
      "bin/notebook-run.sh      \n"
     ]
    }
   ],
   "source": [
    "# use glob to get a list of filenames in the directory bin \n",
    "# that end with .sh -> pattern n* using a wildcard\n",
    "filenames = Path('bin').glob('n*')\n",
    "for f in filenames:\n",
    "    print(f'{str(f):25s}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "solution2": "hidden",
    "solution2_first": true
   },
   "source": [
    "#### Exercise 5\n",
    "\n",
    "* Use `Path` to show the file permissions of all files that end `.md` in the directory `.` (current directory)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "solution2": "hidden"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/Users/philiplewis/Documents/GitHub/geog0111/notebooks/TIMETABLE.md /Users/philiplewis/Documents/GitHub/geog0111/notebooks/Using-the-course-notes.md /Users/philiplewis/Documents/GitHub/geog0111/notebooks/OutsideInstall-Docker-owner.md /Users/philiplewis/Documents/GitHub/geog0111/notebooks/Install.md /Users/philiplewis/Documents/GitHub/geog0111/notebooks/OutsideInstall-Local.md /Users/philiplewis/Documents/GitHub/geog0111/notebooks/OutsideInstall-Docker.md /Users/philiplewis/Documents/GitHub/geog0111/notebooks/README.md /Users/philiplewis/Documents/GitHub/geog0111/notebooks/InstallGDAL.md /Users/philiplewis/Documents/GitHub/geog0111/notebooks/OutsideInstall-Requirements.md\n"
     ]
    }
   ],
   "source": [
    "from pathlib import Path\n",
    "# ANSWER\n",
    "# import packages\n",
    "\n",
    "# Use Path to show the file \n",
    "# permissions of all files that end .md \n",
    "# in the directory . (current directory)\n",
    "\n",
    "# Path().cwd() gives the current directory (.)\n",
    "here = Path().cwd()\n",
    "\n",
    "# clearly, this needs glob the pattern will be *.md\n",
    "# use * to put output as list of arguments\n",
    "print(*here.glob('*.md'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reading and writing information\n",
    "\n",
    "The functions for reading and writing information to and from files are:\n",
    "\n",
    "\n",
    "|command|  purpose|\n",
    "|---|---|\n",
    "|`Path.open()`| open a file and return a file descriptor|\n",
    "|`Path.read_text()`|  read text|\n",
    "|`Path.write_text()`| write text|\n",
    "|`Path.read_bytes()`| read byte data|\n",
    "|`Path.write_bytes()`| write byte data|\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### copy\n",
    "\n",
    "If we are only dealing with reading and writing information in `Path` objects (and those derived from `Path`), then we will normally use `Path.read_text()` and `Path.read_bytes()` for reading text and binary data respectively, and `Path.write_text()` and `Path.write_bytes()` for writing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "type: <class 'str'>\n",
      "content: {\"name\": \"geog0111\", \"channels\": [\"conda-forge\", \"defaults\"], \"dependencies\": [\"git\", \"python>=3.7\", \"nomkl\", \"libgdal\", \"geopandas\", \"rasterio\", \"scipy\", \"gdal\", \"beautifulsoup4\", \"fiona\", \"numpy\", \"statsmodels\", \"cartopy\", \"scikit-image\", \"netcdf4\", \"scikit-learn\", \"matplotlib-base\", \"pandas\", \"shapely\", \"ipyleaflet\", \"pytest\", \"seaborn\", \"hdf5\", \"flake8\", \"ipywidgets\", \"xarray\", \"black\", \"folium\", \"jupyter_console\", \"pandocfilters\", \"nbconvert\", \"jupyterlab\", \"pandoc\", \"pyephem\", \"libnetcdf\", \"ipykernel\", \"ipympl\", \"pip\", \"yapf\", \"autopep8\", \"geemap\", \"jupyter\", \"nbgitpuller\", \"mkdocs\", \"pycodestyle\", \"sphinx\", \"fsspec\", {\"pip\": [\"mkdocs-jupyter\", \"urlpath\", \"jupyter_contrib_nbextensions\", \"mknotebooks\", \"mkdocs-material\", \"mkdocs-exclude\", \"mkdocs-git-revision-date-localized-plugin\"]}]}\n"
     ]
    }
   ],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "# read a text (json) file\n",
    "json_file = Path('bin/copy/environment.json')\n",
    "content = json_file.read_text()\n",
    "\n",
    "print(f'type: {type(content)}')\n",
    "print(f'content: {content}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that the whole file contents are returned as a string. Sometimes this might be what we want. \n",
    "\n",
    "For example, to make a copy of this file in a directory `mydata`, we can simply use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "bin/copy/environment.json size: 801 bytes\n"
     ]
    }
   ],
   "source": [
    "ifile = Path('bin/copy/environment.json')\n",
    "ofile = Path('mydata',ifile.name)\n",
    "\n",
    "# look at ifile size\n",
    "print(f'{ifile} size: {ifile.stat().st_size} bytes')\n",
    "\n",
    "# make sure parent directory exists\n",
    "ofile.parent.mkdir(parents=True,exist_ok=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note the use of `ifile.name` to refer to the file name and make sure the output filename is the same as the input here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly for a binary file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "images/ucl.png size: 1956 bytes\n"
     ]
    }
   ],
   "source": [
    "ifile = Path('images/ucl.png')\n",
    "ofile = Path('mydata',ifile.name)\n",
    "\n",
    "# look at ifile size\n",
    "print(f'{ifile} size: {ifile.stat().st_size} bytes')\n",
    "\n",
    "# make sure parent directory exists\n",
    "ofile.parent.mkdir(parents=True,exist_ok=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1956 bytes written for mydata/ucl.png\n"
     ]
    }
   ],
   "source": [
    "# copy using read_bytes and write_bytes\n",
    "# copy data: write_bytes returns number of bytes written\n",
    "nbytes = ofile.write_bytes(ifile.read_bytes()) #for binary files\n",
    "print(f'{nbytes} bytes written for {ofile}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "# tidy up and remove the file\n",
    "ofile.unlink()\n",
    "ofile.parent.rmdir()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "solution2": "hidden",
    "solution2_first": true
   },
   "source": [
    "#### Exercise 6\n",
    "\n",
    "Copy the file [`geog0111/cylog.py`](geog0111/cylog.py) to a new directory `myfile` and confirm the size of the file copied. Tidy up by deleting the copied file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "solution2": "hidden"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "9249 bytes written for myfile/cylog.py\n"
     ]
    }
   ],
   "source": [
    "from pathlib import Path\n",
    "# ANSWER\n",
    "# import packages\n",
    "\n",
    "# Copy the file geog0111/cylog.py to a new directory \n",
    "# myfile and confirm the size of the file copied. \n",
    "# Tidy up by deleting the copied file.\n",
    "\n",
    "# setup Path object for ifile \n",
    "ifile = Path('geog0111','cylog.py')\n",
    "\n",
    "# setup Path object for ofile \n",
    "ofile = Path('myfile',ifile.name)\n",
    "# create directory \n",
    "ofile.parent.mkdir(parents=True,exist_ok=True)\n",
    "\n",
    "# read ifile, write to ofile (text)\n",
    "nbytes = ofile.write_text(ifile.read_text()) #for binary files\n",
    "print(f'{nbytes} bytes written for {ofile}')\n",
    "\n",
    "# tidy up and remove the file\n",
    "ofile.unlink()\n",
    "ofile.parent.rmdir()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  `with ... as ...`, open, yaml, json\n",
    "\n",
    "Sometimes we want to open a stream to a file and then to pass that open stream on to some other package. A [stream](https://en.wikipedia.org/wiki/Standard_streams) is a channel through which we may send and/or receive information. This is different to a file, which is where information may reside, but we may for instance open a stream to write to a file. In Python, we call the object that we get when opening a stream a *file object*.\n",
    "\n",
    "For instance, in the json example above, we read the text in as a string, but we didn't do anything to parse (i.e. interpret) the information in the file. \n",
    "\n",
    "Quite often then, when we need to interface to some interpreter of information in a file, we will use `Path.open()` to provide a file descriptor to pass through. There are various other reasons we might use `Path.open()`, including when we need some more subtle control on the operations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `pathlib` function for opening a stream is `Path.open`.\n",
    "\n",
    "The usual way of opening a file to get the file object is:\n",
    "\n",
    "    with Path(filename).open('r') as f:\n",
    "       # do some reading with f\n",
    "       pass\n",
    "       \n",
    "We use the form `with ... as ...` here, so that the file object `f` only exists within this construct and the file is automatically closed when we finish. Codes are spaced in inside the construct, as we have seen in `if ...` or `for ... in ...` constructs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To use `Path.open()`, we have to distinguish between [modes of opening a file](https://pathlib.readthedocs.io/en/0.5/#pathlib.Path.open). Mostly we will use `r` (read) and `w` (write) for opening a file for reading and writing respectively. Sometimes, we might also use `a` (write append).\n",
    "\n",
    "Here, we use a file descriptor method `write` to write data to a file, so we use the mode `w`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hello/hello_world.txt content\n",
      "----------\n",
      "hello world\n"
     ]
    }
   ],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "ofile = Path('hello/hello_world.txt')\n",
    "ofile.parent.mkdir(parents=True,exist_ok=True)\n",
    "\n",
    "mytext = \"hello world\"\n",
    "\n",
    "with ofile.open('w') as f:\n",
    "    f.write(mytext)\n",
    "\n",
    "# print it out\n",
    "print(f'{ofile} content\\n{\"-\"*10}\\n{ofile.read_text()}')\n",
    "\n",
    "# tidy\n",
    "# tidy up and remove the file\n",
    "ofile.unlink()\n",
    "ofile.parent.rmdir()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Two common text formats for certain types of data representation are [json](https://docs.python.org/3/library/json.html) and [`yaml`](http://zetcode.com/python/yaml/). The Python library functions for input and output of both of these use streams: `yaml.safe_load()`, `yaml.safe_dump()`, `json.load()` and `json.dump()` respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on function safe_load in module yaml:\n",
      "\n",
      "safe_load(stream)\n",
      "    Parse the first YAML document in a stream\n",
      "    and produce the corresponding Python object.\n",
      "    \n",
      "    Resolve only basic YAML tags. This is known\n",
      "    to be safe for untrusted input.\n",
      "\n",
      "Help on function safe_dump in module yaml:\n",
      "\n",
      "safe_dump(data, stream=None, **kwds)\n",
      "    Serialize a Python object into a YAML stream.\n",
      "    Produce only basic YAML tags.\n",
      "    If stream is None, return the produced string instead.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import yaml\n",
    "\n",
    "help(yaml.safe_load)\n",
    "help(yaml.safe_dump)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similar functions (`json.load()` and `json.dump()`) exist for json format."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we use the `yaml` package to interpret data in the file `bin/copy/environment.yml` in the variable `env`, which will be a dictionary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env is type <class 'dict'>\n",
      "env keys: dict_keys(['name', 'channels', 'dependencies'])\n"
     ]
    }
   ],
   "source": [
    "from pathlib import Path\n",
    "import yaml\n",
    "\n",
    "# form the file name\n",
    "yaml_file = Path('bin/copy/environment.yml')\n",
    "\n",
    "# open stream object 'read'\n",
    "with yaml_file.open('r') as f:\n",
    "    env = yaml.safe_load(f)\n",
    "\n",
    "print(f'env is type {type(env)}')\n",
    "print(f'env keys: {env.keys()}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we dump that information to a json file, using `Path.open()` with `w` (write) mode:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "import json\n",
    "\n",
    "# form the file name\n",
    "json_file = Path('bin/copy/environment.json')\n",
    "\n",
    "with json_file.open('w') as f:\n",
    "    json.dump(env, f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the examples above, we have done more than just copy: we have interpreted the information from one file format (yaml), and translated it into a file for another format (json). The intermediate data in the Python script was help as a form of dictionary, rather than just the ASCII text string we had read earlier."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "solution2": "hidden",
    "solution2_first": true
   },
   "source": [
    "#### Exercise 8\n",
    "\n",
    "* write code to read from the json-format file `bin/copy/environment.json` into a dictionary called `json_data`.\n",
    "* print out the dictionary keys.\n",
    "* print the file size of the json-format file in KB to two decimal places."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "solution2": "hidden"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['name', 'channels', 'dependencies'])\n",
      "file bin/copy/environment.json size  0.78 KB\n"
     ]
    }
   ],
   "source": [
    "# ANSWER\n",
    "# write code to read from the json-format file \n",
    "# bin/copy/environment.json \n",
    "# into a dictionary called json_data.\n",
    "json_file = Path('bin/copy/environment.json')\n",
    "\n",
    "# use with ... as ... as we have been shown\n",
    "with json_file.open('r') as f:\n",
    "    json_data = json.load(f)\n",
    "    \n",
    "# print out the dictionary keys.\n",
    "print(json_data.keys())\n",
    "\n",
    "# print the file size of the \n",
    "# json-format file in KB to two decimal places.\n",
    "print(f'file {json_file} size {json_file.stat().st_size / 1024 : .2f} KB')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary test\n",
    "\n",
    "Let's try to put several of these things together:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "solution2": "hidden",
    "solution2_first": true
   },
   "source": [
    "#### Exercise 9\n",
    "\n",
    "* check that the file `images/ucl.png` exists and print modification time and the file size in KB to two decimal places\n",
    "* make a directory `myfiles` and copy the file `images/ucl.png` to this directory\n",
    "* show the file size of `myfiles/ucl.png`, the modification time, and the time now\n",
    "* after that, tidy up by deleting the file `myfiles/ucl.png` and the directory `myfiles`. Confirm that you have done this.\n",
    "\n",
    "You will need to know how many Bytes in a Kilobyte, and how to [format a string to two decimal places](012_Python_strings.ipynb#String-formating). You will also need to remember how to use [`if` statements](015_Python_control.ipynb#Comparison-Operators-and-if)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "solution2": "hidden"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The file images/ucl.png exists?: True\n",
      "The file images/ucl.png size: 1956 B\n",
      "The file images/ucl.png size:  1.91 KB\n",
      "The file images/ucl.png modification time: 2022-12-12 10:26:22\n",
      "\n",
      "confirm with ls -lh\n",
      "-rw-r--r--@ 1 philiplewis  staff   1.9K 12 Dec  2022 images/ucl.png\n",
      "\n",
      "==== After copying\n",
      "The file myfiles/ucl.png size:  1.91 KB\n",
      "The file myfiles/ucl.png modification time: 2023-06-28 13:26:47.299232\n",
      "Time now is 2023-06-28 13:26:47.300000\n"
     ]
    }
   ],
   "source": [
    "from pathlib import Path\n",
    "from datetime import datetime\n",
    "# ANSWER\n",
    "# import packages\n",
    "\n",
    "# check that the file images/ucl.png exists and \n",
    "# print modification time and the file size in KB to two decimal places\n",
    "\n",
    "# make a directory myfiles and copy the file images/ucl.png \n",
    "# to this directory\n",
    "# show the file size of myfiles/ucl.png, the \n",
    "# modification time, and the time now\n",
    "# after that, tidy up by deleting the file \n",
    "# myfiles/ucl.png and the directory myfiles. \n",
    "# Confirm that you have done this.\n",
    "\n",
    "# check that the file images/ucl.png exists and \n",
    "# print modification time and the file size in KB to two decimal places\n",
    "\n",
    "ifile = Path('images','ucl.png')\n",
    "# check that the file images/ucl.png exists and \n",
    "print(f'The file {ifile} exists?: {ifile.exists()}')\n",
    "# print the file in KB to two decimal places\n",
    "# 1 KB = 1024 bytes\n",
    "ifile_bytes = ifile.stat().st_size\n",
    "print(f'The file {ifile} size: {ifile_bytes} B')\n",
    "# in KB using .2f format for 2 dp\n",
    "ifile_size_kb = ifile.stat().st_size / 1024\n",
    "print(f'The file {ifile} size: {ifile_size_kb : .2f} KB')\n",
    "\n",
    "# show the file size of myfiles/ucl.png, the \n",
    "# modification time, and the time now\n",
    "\n",
    "modified = ifile.stat().st_mtime\n",
    "h_modified = datetime.fromtimestamp(modified)\n",
    "print(f'The file {ifile} modification time: {h_modified}')\n",
    "\n",
    "print(\"\\nconfirm with ls -lh\")\n",
    "!ls -lh {ifile}\n",
    "\n",
    "# make a directory myfiles and copy the file images/ucl.png to this directory\n",
    "ofile = Path('myfiles',ifile.name)\n",
    "# mkdir the parent\n",
    "ofile.parent.mkdir(parents=True,exist_ok=True)\n",
    "# copy text file with read_text() and write_text()\n",
    "ofile.write_bytes(ifile.read_bytes()) #for binary files\n",
    "\n",
    "print('\\n==== After copying')\n",
    "# confirm size\n",
    "print(f'The file {ofile} size: {ofile.stat().st_size / 1024 : .2f} KB')\n",
    "\n",
    "# mod time\n",
    "modified = ofile.stat().st_mtime\n",
    "h_modified = datetime.fromtimestamp(modified)\n",
    "print(f'The file {ofile} modification time: {h_modified}')\n",
    "# now time\n",
    "print(f'Time now is {datetime.now()}')\n",
    "\n",
    "# tidy up\n",
    "ofile.unlink()\n",
    "ofile.parent.rmdir()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "\n",
    "In this section, we have considered URLs and filenames in some detail, and made use of functions from [`pathlib`](https://docs.python.org/3/library/pathlib.html) to access them. \n",
    "\n",
    "The first batch of `Path` commands we saw had much commonality with the core `unix` commands we had previously come across for moving around the file system and finding file information:\n",
    "\n",
    "\n",
    "|command| unix-equivalent | purpose|\n",
    "|---|---|---|\n",
    "|`Path.cwd()`| `pwd` |Return path object representing the current working directory|\n",
    "|`Path.home()`| `~`| Return path object representing the home directory|\n",
    "|`Path.stat()`| `ls -l` | return info about the path. File size is `Path.stat().st_size` |\n",
    "|`Path.chmod()`| `chmod` | change file mode and permissions|\n",
    "|`Path.glob(pattern)`| `ls *` | Glob the pattern given in the directory that is represented by the path, yielding matching files of any kind|\n",
    "|`Path.mkdir()`| `mkdir` | to create a new directory at the given path. We have repeatedly used `ofile.parent.mkdir(parents=True,exist_ok=True)` to make sure the directory for an output file exists|\n",
    "|`Path.rename()`| `mv` | Rename a file or directory to the given target|\n",
    "|`Path.rmdir()`| `rmdir` | Remove the empty directory|\n",
    "|`Path.unlink()`| `rm` | Remove the file or symbolic link|\n",
    "|`Path.touch()` | `touch` | create (zero-sized) or update a file |\n",
    "|`Path.parent` | `..` | The parent object (one level up in directory tree) |\n",
    "\n",
    "Some other common utilities we came across were:\n",
    "\n",
    "|command|  purpose|\n",
    "|---|---|\n",
    "|`Path.as_posix()`| Return the object as a posix filename string |\n",
    "|`Path.exists()` | `True` if exists, `False` if not|\n",
    "|`Path.parent` |  the parent object (one level up in directory tree), or the directory a file is in |\n",
    "|`Path.name` | the file name |\n",
    "|`Path.is_file()` | `True` if object is a file |\n",
    "|`Path.is_dir()` | `True` if object is a directory |\n",
    "\n",
    "\n",
    "We have also seen how to open files, and read and write information from files.\n",
    "\n",
    "|command|  purpose|\n",
    "|---|---|\n",
    "|`Path.open()`| open a file and return a file descriptor|\n",
    "|`Path.read_text()`|  read text|\n",
    "|`Path.write_text()`| write text|\n",
    "|`Path.read_bytes()`| read byte data|\n",
    "|`Path.write_bytes()`| write byte data|\n",
    "\n",
    "\n",
    "\n",
    "We have learned how to use these with the object-oriented `Path` object to do some simple things like creating and deleting files and directories, getting listings of files and so on. These are such common tasks in coding that you need to spend some time making sure you know at least the basics here. A number of exercises are given to allow you to practice and test yourself, but you should also try to make up your own examples.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "[<img src=\"images/noun_post_2109127.svg\" width=\"50\" align='right'>](021_URLs.ipynb)[<img src=\"images/noun_pre_2109128.svg\" width=\"50\" align='right'>](018_Running_Python.ipynb)\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda env:geog0111-geog0111",
   "language": "python",
   "name": "conda-env-geog0111-geog0111-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.13"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": false,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "207px"
   },
   "toc_section_display": false,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}