README update for motifs, can't get them to work in local... because I was editing the heracles file?

Author: rjurney

README.md

@@ -74,10 +74,43 @@ Python 3.10.11 ('graphml') /opt/anaconda3/envs/graphml/bin/python
 
 Note: the Python version is set to `3.10.11` because Jupyter Stacks have not been updated more recently.
 
-## Network Motifs
+## Knowledge Graph Construction in PySpark
+
+We build a knowledge graph from the [Stack Exchange Archive](https://archive.org/details/stackexchange) for the network motif section of the course.
+
+### Docker Exec Commands
+
+To run a bash shell in the Jupyter container, type:
+
+```bash
+docker exec -it jupyter bash
+```
+
+Once you're there, you can run the following commands to download and prepare the data for the course.
+
+First, download the data:
+
+```bash
+graphml_class/stats/download.py stats.meta
+```
+
+Then you will need to convert the data from XML to Parquet:
+
+```bash
+spark-submit --packages "com.databricks:spark-xml_2.12:0.18.0" graphml_class/stats/xml_to_parquet.py
+```
+
+The course covers knowledge graph construction in PySpark in [graphml_class.stats.graph.py](graphml_class/stats/graph.py).
+
+```bash
+spark-submit graphml_class/stats/graph.py
+```
+
+## Network Motifs with GraphFrames
 
 This course now covers [network motifs](https://www.sciencedirect.com/topics/biochemistry-genetics-and-molecular-biology/network-motif#:~:text=A%20network%20motif%20is%20a,multiple%20times%20within%20a%20network.) in property graphs (frequent patterns of structure) using pyspark / [GraphFrames](https://graphframes.github.io/graphframes/docs/_site/index.html) (see [motif.py](https://github.com/Graphlet-AI/graphml-class/blob/main/graphml_class/stats/motif.py), no notebook yet).
 It supports directed motifs, not undirected. All the 4-node motifs are outlined below. Note that GraphFrames can also filter the
 paths returned by its `f.find()` method using any Spark `DataFrame` filter - enabling temporal and complex property graph motifs.
 
 <center><img src="images/illustration-of-directed-graphlets-a-The-40-two-to-four-node-directed-graphlets-G0.png" alt="All 4-node directed network motifs"></center>
+

graphml_class/stats/graph.py

@@ -12,11 +12,7 @@
 os.environ["JAVA_HOME"] = "/usr/lib/jvm/java-17-openjdk-amd64"
 
 # Setup PySpark to use the GraphFrames jar package from maven central
-os.environ["PYSPARK_SUBMIT_ARGS"] = (
-    "--driver-memory 4g pyspark-shell "
-    "--executor-memory 4g pyspark-shell "
-    "--driver-java-options='-Xmx4g -Xms4g' "
-)
+os.environ["PYSPARK_SUBMIT_ARGS"] = "--driver-memory 4g --executor-memory 4g"
 
 # Show all the rows in a pd.DataFrame
 pd.set_option("display.max_columns", None)
@@ -43,8 +39,8 @@
     # Lets the Id:(Stack Overflow int) and id:(GraphFrames ULID) coexist
     .config("spark.sql.caseSensitive", True)
     # Single node mode - 128GB machine
-    .config("spark.driver.memory", "16g")
-    .config("spark.executor.memory", "8g")
+    .config("spark.driver.memory", "4g")
+    .config("spark.executor.memory", "4g")
     .getOrCreate()
 )
 sc: SparkContext = spark.sparkContext
@@ -388,3 +384,6 @@ def add_missing_columns(df, all_cols):
 
 # Write to disk and back again
 relationships_df.write.mode("overwrite").parquet(EDGES_PATH)
+
+spark.stop()
+print("Spark stopped.")