Spark 2 Workbook Answers Guide

print(f"Unique words: unique_word_count")

---

val spark = SparkSession.builder() .appName("DeptSalary") .getOrCreate()

val df = spark.read .option("header","true") .option("inferSchema","true") .csv("hdfs:///data/employees.csv") spark 2 workbook answers

1. **Ingestion** – `spark.read.json` or `textFile`. 2. **Parsing** – `withColumn` + `from_unixtime`, `regexp_extract`. 3. **Cleaning** – filter out malformed rows, `na.drop`. 4. **Enrichment** – join with a static lookup table (broadcast). 5. **Aggregation** – `groupBy(date, status).agg(count("*").as("cnt"))`. 6. **Output** – write to Parquet partitioned by `date` **or** stream to console for debugging.

## 7. Putting It All Together – A Mini‑Project Blueprint

# 3️⃣ Keep only unique words distinct_words = words.distinct() **Aggregation** – `groupBy(date

| Tip | How to Apply | |-----|--------------| | **Show Spark’s lazy evaluation** | Mention that transformations build a DAG, actions trigger execution. | | **Explain the physical plan** | Use `df.explain()` in a note to demonstrate understanding of shuffle, broadcast, etc. | | **State assumptions** | “Assume the input file fits in HDFS and each line is a UTF‑8 string.” | | **Edge‑case handling** | Talk about empty files, null values, or malformed CSV rows. | | **Performance hints** | Suggest `repartition` before a heavy shuffle or using `broadcast` for small lookup tables. | | **Testing** | Show a tiny local test (e.g., `sc.parallelize(["a b","b c"]).flatMap(...).collect()`). | | **Clean code** | Use meaningful variable names, consistent indentation, and short comments. |

Add a short paragraph for each stage, explaining why you chose that API.

If the workbook includes a **mini‑project** (e.g., “process a log dataset and produce a daily report”), you can outline the full pipeline: **Parsing** – `withColumn` + `from_unixtime`

– bulk HTTP calls:

# 2️⃣ Split lines into words and clean them words = lines.flatMap(lambda line: line.split()) \ .map(lambda w: w.lower().strip('.,!?"\''))