Scaling OCSF Validation in Databricks: From 4GB Disk Spills to Zero
TL;DR: During an OCSF 1.8.0 rollout, a validation stage that should have completed in seconds was taking more than 45 minutes and spilling nearly 4GB to local disk. By redesigning our Pandas UDF execution model and replacing interpreted validation with compiled validators, we reduced spills to zero and improved throughput by up to 20x.
The Problem That Started the Investigation
While upgrading our cybersecurity data platform to support OCSF 1.8.0, we noticed validation jobs taking far longer than expected. The workload itself wasn't particularly large, which made the Spark UI metrics confusing. Executors were spilling gigabytes of data to disk, task durations were increasing, and Photon utilization remained surprisingly low.
At first glance, the symptoms looked like a cluster sizing issue. However, increasing resources produced only marginal improvements. The real bottleneck was hidden inside the Python execution layer.
Why OCSF Validation Is Different
Most Spark pipelines operate on a fixed schema. OCSF validation introduces a different challenge. Every incoming event contains a class_uid that determines which schema should be used for validation.
- Authentication events use one schema.
- Network activity events use another.
- Application activity events use yet another.
This means validation cannot rely on a single Spark StructType. The validation framework must dynamically resolve the correct schema at runtime and validate each record accordingly.
Original Architecture
The original implementation was straightforward. Spark records were passed into a Pandas UDF, converted into JSON, and validated using Draft7Validator from the jsonschema library.
What We Found
1. The JSON Expansion Tax
Spark stores data efficiently in compressed columnar formats such as Delta Lake and Parquet. Once those records cross the JVM-Python boundary and become JSON strings, memory consumption increases dramatically.
What looked small inside Spark became significantly larger inside Python workers.
2. The Batch Size Trap
Arrow batches containing thousands of records were being transferred into Python workers simultaneously. Each worker had to deserialize, expand, and validate large numbers of JSON documents before returning results.
As memory pressure increased, Spark began spilling data to local storage.
3. Interpreter Overhead
The jsonschema library is flexible and standards-compliant, but it evaluates schema rules repeatedly. Even though the schema definition remains unchanged, the validation engine spends CPU cycles interpreting the same rules again and again.
Fix #1: Iterator-Based Processing
Insert Image: the-solution.png
The first objective was eliminating memory pressure. Instead of processing an entire Arrow batch as a single unit, we switched to an Iterator Pandas UDF and introduced internal micro-batching.
for batch in iterator:
for chunk in split(batch, 1000):
yield validate(chunk)This change did not alter Spark partition sizes. Instead, it changed how memory was consumed inside Python workers.
By processing smaller chunks and immediately yielding results downstream, the garbage collector could reclaim memory before the next chunk arrived.
Impact
- Disk spill reduced from ~4GB to 0 bytes.
- Memory consumption became predictable.
- Executor stability improved significantly.
Fix #2: Compiled Validation
Once memory pressure disappeared, CPU utilization became the next bottleneck.
Insert Image: compiled-bytecode-engine.png
We replaced jsonschema with fastjsonschema and introduced executor-level caching.
@lru_cache(maxsize=None)
def get_validator(schema_path):
return fastjsonschema.compile(load_schema(schema_path))Instead of interpreting rules repeatedly, validators are compiled once and reused for subsequent records.
The validation path becomes a direct function invocation rather than a schema interpretation cycle.
Production Results
| Metric | Before | After |
|---|---|---|
| Task Duration | 1.42 Minutes | 30.49 Seconds |
| Disk Spill | ~4GB | 0 Bytes |
| Photon Utilization | 3% | 16% |
| Throughput | Baseline | Up to 20x Faster |
Lessons Learned
The biggest lesson was that Spark itself was not the bottleneck. The bottlenecks were hidden inside Python worker execution.
- Serialization costs matter.
- Iterator UDFs are powerful for memory-sensitive workloads.
- Compiled validators can dramatically reduce CPU overhead.
- Disk spill metrics often reveal issues before CPU metrics do.
Final Thoughts
By combining iterator-based processing with compiled schema validation, we transformed a spill-heavy validation pipeline into a stable and scalable platform component. The solution reduced infrastructure waste, improved throughput, and provided a foundation for future OCSF adoption at scale.