APACHE SPARK · INDEPENDENT ENGINEERING
Design, tune, upgrade and operate self-managed Apache Spark for batch, SQL and streaming workloads, with Spark 4 migrations tested against your own jobs and data.
Independent Apache Spark engineering.
These are independent Bassirah engineering and support offerings. They do not imply endorsement by The Apache Software Foundation or the Apache Spark project, official vendor support rights, enterprise feature entitlement or maintenance guarantees.
Apache Spark, Spark and the Spark logo are trademarks of The Apache Software Foundation. Bassirah is not a partner of Databricks, or of Amazon Web Services, Google Cloud or Microsoft for their Spark services.
Supported releases.
Apache Spark is a project of The Apache Software Foundation, released under the Apache License 2.0. Bassirah provides independent engineering for the 4.2, 4.1 and 4.0 release lines, and for the 3.5 line during its extended long-term support period, with migration planning from 3.5 to Spark 4. The exact versions in scope are named in each statement of work. Last reviewed: 26 October 2026.
| Release line | Status (accessed 26 October 2026) | How Bassirah treats it |
|---|---|---|
| 4.2 | First released 14 July 2026; latest release 4.2.1, 27 October 2026. The versioning policy states an 18-month maintenance window for 4.2.x | Default target for new deployments and Spark 4 migrations |
| 4.1 | First released 16 December 2025; latest release 4.1.4, 27 October 2026 | Supported; upgrade planning to 4.2 |
| 4.0 | First released 23 May 2025; latest release 4.0.5, 27 October 2026 | Supported; upgrade planning to 4.2 |
| 3.5 | Latest release 3.5.9, 16 July 2026. Extended LTS with security fixes only, ending November 2027 | Supported for existing workloads; migration to Spark 4 planned before the extended period ends |
Starting with 4.3.0, the project plans quarterly feature releases and an annual major release. The policy names 4.5.0 as the planned 4.x long-term support release.
WHY IT MATTERS
Apache Spark runs data processing in parallel across a cluster, with APIs in SQL, Python, Scala, Java and R. Teams use it for large batch transformations, lakehouse pipelines, Structured Streaming and machine learning data preparation.
Each job is split into stages and tasks that run on executors, with data exchanged between stages in shuffles. Partitioning, memory settings, shuffle volume and data layout decide whether a job finishes reliably or fails late in the run. Security also needs deliberate work: the project’s documentation states that authentication is not enabled by default and that no deployment mode is secure by default. Spark 4 changed defaults too, including ANSI SQL mode. Bassirah engineers the platform, tunes the jobs that matter and tests every upgrade against your own workloads.
WHAT BASSIRAH DELIVERS
FIT FIRST
A Spark platform has many moving parts. We recommend it when the workload needs distributed processing, and we say so when another engine or a single machine is enough.
| Requirement | Apache Spark is worth evaluating when… | Another option is often enough when… |
|---|---|---|
| Data volume | Transformations must process more data than one machine handles within your processing window | The data fits comfortably on one machine, or in a single database |
| Workload type | Pipelines combine SQL with Python, Scala or Java code, user-defined functions or machine learning data preparation | Users mainly run interactive SQL over existing tables: a query engine such as Trino or ClickHouse may fit better |
| Streaming | Streams are processed with the same code and tables as batch, using Structured Streaming | Each event needs stateful, event-at-a-time processing: Apache Flink may fit better |
| Scheduling | You pair Spark with an orchestrator for dependencies and retries | You expect Spark itself to schedule workflows: it does not; see Apache Airflow |
| Operations | Your team, or a contracted provider, will operate a cluster, or you choose a managed Spark service | You want the smallest possible operational footprint |
Already decided on a managed Spark service? We can still review job design, data layout and migration plans. See the support routes below.
ARCHITECTURE AND SECURITY
Spark can run on Kubernetes, Hadoop YARN or its own Standalone cluster manager. Clients can submit applications directly or connect through Spark Connect, a client-server architecture introduced in Spark 3.4. Each choice changes how teams are isolated, how resources scale and how failures are contained. We choose them per workload, secure them explicitly and prove recovery for streaming jobs in tested recovery exercises against objectives you approve.
| Decision | Typical approach | Trade-off we make explicit |
|---|---|---|
| Cluster manager | Kubernetes for container-based platforms, YARN where a Hadoop estate already exists | Fit with existing operations vs. the effort of running a second scheduler |
| Client access | Spark Connect for remote clients and notebooks, direct submission for scheduled jobs | Decoupled clients and upgrades vs. API coverage to test per workload |
| Resource scaling | Dynamic allocation of executors with shuffle tracking, sized per workload | Shared capacity and cost vs. predictable start times for critical jobs |
| Storage and tables | Object storage or HDFS with an open table format, partitioned for the main queries | Write cost and file maintenance vs. read efficiency |
| Security | Authentication, encryption and restricted access to the web UI and submission endpoints, on trusted networks only | Setup and key management effort vs. the exposure of an unsecured cluster |
| Streaming recovery | Structured Streaming queries with checkpoint locations on durable storage | Checkpoint storage and upgrade constraints vs. restart without data loss or duplicates where the sink supports it |
OPERATIONS
Read stage and task metrics, find skew and spills, and tune partitioning, joins and Adaptive Query Execution settings for the jobs that cost the most.
Size files and partitions for the queries that matter, and keep table maintenance on a schedule so reads stay efficient.
Maintenance and feature-release upgrades with dependency checks for Java, Scala, Python and connectors, rehearsed in staging with a rollback decision point.
Executor health, failed tasks, shuffle and memory pressure, and streaming progress, with History Server retention and actionable alerts.
MIGRATION
Delivered through our migration and modernization practice. Spark 4 requires Java 17 or later and Scala 2.13, removes Mesos support, raises minimum PySpark dependency versions and turns on ANSI SQL mode by default.
Start with a compatibility review of your own jobs and dependencies.
USE CASES
Large transformations that build curated tables in object storage on open table formats.
Structured Streaming jobs that read events, for example from Apache Kafka, and write to the same tables as batch jobs.
Feature and training data preparation in Python or SQL at a scale one machine cannot handle.
SUPPORT ROUTES
Apache Spark is open source. Several companies also offer commercial platforms and managed services that run Spark, including Databricks, Amazon EMR, Google Cloud Managed Service for Apache Spark (formerly Dataproc) and, on Microsoft Azure, Azure HDInsight, Spark pools in Azure Synapse Analytics and Microsoft Fabric. Bassirah supports self-managed Apache Spark independently, is not a partner of these vendors and is clear about where its commitments end.
| Decision factor | Community only | Bassirah independent support | Commercial platform or managed service |
|---|---|---|---|
| Who operates Spark | Your team | Your team with Bassirah engineering, or Bassirah under a managed scope | The vendor operates the service, under its own terms |
| Help with incidents | Public community channels, no response commitment | Response targets per your agreement | Vendor support under the vendor's own terms |
| Code defects | Reported by you to the project | Reproduced, worked around and reported upstream by Bassirah | Handled by the vendor under its terms |
| Upstream fixes and releases | On the project's timeline | On the project's timeline; we track them, test them and apply them through change control | Vendor runtime versions and timelines, under the vendor's terms |
| Commercial products | Not included | Not included: Bassirah does not resell or license commercial Spark platforms or managed Spark services | Contracted directly with the vendor |
If you need a commercial platform or a managed Spark service, that is contracted with the vendor. How cases, severities and response targets work for our independent support is set out in our open-source support model. For fully contracted operations, see Managed Services.
WORKS WITH
Spark is one engine in our data processing practice. We place it alongside orchestration, streaming, query engines and object storage where each fits, and test every integration first.
Orchestrates Spark jobs with dependencies, retries and schedules.
Event streams that Structured Streaming jobs can read from and write to.
S3-compatible object storage for lakehouse tables, where the deployment calls for it.
Yes. Apache Spark is a project of The Apache Software Foundation, released under the Apache License 2.0. Commercial platforms and managed services built on Spark are separate products with their own terms. Bassirah supports self-managed Apache Spark and is not endorsed by the ASF or the Spark project.
We work on the 4.2, 4.1 and 4.0 release lines, and on 3.5 during its extended long-term support period, which the project says ends in November 2027 with security fixes only. The versions in scope are confirmed in each statement of work.
Spark 4 requires Java 17 or later and Scala 2.13, turns on ANSI SQL mode by default, removes Mesos support and raises minimum Python dependency versions. Some defaults and data source mappings change too. We test your jobs against the project’s migration guides before cutover.
Spark suits large transformations that mix SQL with code. For interactive SQL over existing tables, a query engine such as Trino is often simpler. Many platforms use both, and we help you decide per workload.
No. Bassirah is not a partner of Databricks or of the cloud providers’ managed Spark services, and does not provide vendor support, vendor security fixes or vendor escalation. Our services are independent engineering and support, with scope and response targets defined in your agreement. Code fixes and releases follow the project’s timeline. See our support model.
No. The project’s documentation states that security features such as authentication are not enabled by default, and it advises against exposing clusters to untrusted networks. We configure authentication, encryption and access to the web UI and submission endpoints as part of every deployment.
Tell us about your clusters, jobs and upgrade plans. We will scope an assessment and a practical next step.
Operations & Reliability
Spark 3.5 now gets security fixes only, until November 2027. What Spark 4 changes for existing jobs, and how to plan a migration you can…
Apache Spark
Read article : Apache Spark 4 Migration: Planning the Move Off Spark 3.5→
