Apache Spark 4 Migration: Planning the Move Off Spark 3.5

Apache Spark 3.5 is now in an extended long-term support period that the project says will end in November 2027. During that period, the 3.5 line receives security fixes only. Teams still running Spark 3.5 have a defined window to move to Spark 4, and the move is not a routine upgrade: Spark 4 changes the Java and Scala baseline, SQL semantics, several defaults and the minimum Python dependencies.

This article explains where each Spark release line stands on 26 October 2026, what the project’s versioning policy now says about maintenance, which Spark 4 changes affect existing jobs most, and how to plan a migration that you can test and reverse.

Where the Spark release lines stand

The dates below come from the Apache Spark news archive and the versioning policy, accessed 26 October 2026. A re-check on 27 October 2026 found new maintenance releases for three lines, shown in the table.

Line	First release	Latest release	What the policy says
4.2	4.2.0, 14 July 2026	4.2.1, 27 October 2026	An 18-month maintenance window for 4.2.x
4.1	4.1.0, 16 December 2025	4.1.4, 27 October 2026	No end date published
4.0	4.0.0, 23 May 2025	4.0.5, 27 October 2026	No end date published
3.5	3.5.0, 13 September 2023	3.5.9, 16 July 2026	Extended LTS, security fixes only, ending November 2027

The 4.2.1 release notes describe it as a maintenance release containing security and correctness fixes. The policy also states that critical security patches are backported to all actively maintained branches. It does not publish calendar end dates for 4.0, 4.1 or 4.2, so this article does not estimate them.

What changed in the versioning policy

The versioning policy now describes a faster cadence. Starting with 4.3.0, the project plans feature releases every three months and a major release every twelve months. Feature releases must not upgrade third-party dependencies by default, must not change SQL semantics or configuration defaults, and may add public APIs but not change or remove them. Breaking changes are reserved for major releases.

Under the new cadence, the last feature release of each major line is its long-term support release, with an 18-month maintenance window. Other lines are maintained for six months. As a transition exception, the policy names 4.5.0, not 4.3.0, as the 4.x long-term support release. It also states that 4.2.x keeps its existing 18-month window.

For planning, two consequences matter. First, 4.2 is a sensible landing point today, because it is the newest line and has a stated maintenance window. Second, the next major line will bring breaking changes again, so the work you do now to test behavior and remove legacy settings will pay off at the next major upgrade too.

The extended period for 3.5 has limits. The policy says it applies only to the main Spark repository, not to sub-projects released separately, such as the Spark Connect clients for Swift, Rust and Go or the Spark Kubernetes operator. It also warns that a dependency security fix that is not available for Java 8 may be marked as won’t-fix or shipped without Java 8 support.

Spark 4 changes that affect existing jobs

The Spark 4.0.0 release notes and the migration guides list the changes. These are the ones we see most often in planning work.

Platform baseline

• Java: Spark 4.0 dropped JDK 8 and 11 and made JDK 17 the default. The current Spark 4.2 documentation lists Java 17, 21 and 25.
• Scala: Scala 2.12 support was dropped; Spark 4 is built with Scala 2.13. Applications must use the Scala version Spark was compiled for, so every Scala library and connector needs a 2.13 build.
• Python: Python 3.8 was dropped in 4.0, and the 4.2 documentation lists Python 3.10 and later. The PySpark migration guide raises the minimum versions to pandas 2.0.0, NumPy 1.21 and PyArrow 11.0.0, and removes several pandas API on Spark methods.
• Cluster managers: support for Apache Mesos was removed. Kubernetes, YARN and Standalone remain.

SQL behavior

The SQL migration guide starts with the largest change: spark.sql.ansi.enabled is on by default. Under ANSI mode, operations that previously returned null or a wrapped value, such as some invalid casts or overflows, raise errors instead. Jobs that ran cleanly on 3.5 can fail on Spark 4, often deep into a long run.

Other SQL changes in the 3.5 to 4.0 section include:

• CREATE TABLE without USING or STORED AS now uses spark.sql.sources.default instead of Hive.
• The default ORC compression codec changed from snappy to zstd.
• Hive metastore versions before 2.0.0 are no longer supported.
• Legacy datetime rebasing settings with the spark.sql.legacy prefix were removed and replaced with new names.
• Type mappings changed for the PostgreSQL and MySQL JDBC data sources, including how timestamps with time zones are read and written.

Most of these changes have a legacy setting that restores the previous behavior. Use those settings to unblock a migration, but record each one and plan its removal: legacy settings hide behavior differences rather than resolve them.

Core and operations

The Core migration guide lists operational changes. Event logs are now rolled and compressed by default. The servlet API moved from javax to jakarta, which matters for custom UI filters. The shuffle service database backend defaults to RocksDB. On Kubernetes, executor pod allocation and persistent volume claim access modes changed. Speculative execution is less aggressive by default, and a log context key was renamed, which can break log parsing.

Choosing a target line

Target	When it fits	What to watch
4.2	New migrations from 3.5, and 4.0 or 4.1 estates ready to move	The newest line: confirm connector and library support for it first
4.1	Estates already on 4.1 that need time before the next step	No published end date; track the news archive and plan the move to 4.2
4.0	Estates already on 4.0	The oldest 4.x line; plan the move to 4.2
Stay on 3.5 for now	Jobs blocked by a dependency without Spark 4 support	Security fixes only, ending November 2027; Java 8 limits on dependency fixes

Whatever the target, check the libraries you depend on first. Table format libraries, connectors and in-house Scala code often decide the schedule more than Spark itself.

A migration plan you can test and reverse

1. Inventory. List every job, notebook and library, with the Java, Scala and Python versions, connectors, Spark configuration and schedule for each.
2. Check dependencies. Confirm a Spark 4 and Scala 2.13 build exists for every library and connector. Replace or rebuild what is missing.
3. Read the guides against your estate. Go through the SQL, Core and PySpark migration guides and mark each change as not relevant, needs a code change, or needs a legacy setting for now.
4. Test under ANSI mode. Run representative jobs on Spark 4 with production-like data. Capture errors, and fix data quality issues or casts at the source rather than turning ANSI mode off everywhere.
5. Run in parallel and reconcile. Run the 3.5 and 4.x versions side by side for an agreed period, and compare row counts, checksums and key aggregates with agreed control checks.
6. Cut over in waves. Move jobs in groups, starting with lower-risk pipelines, with an explicit rollback decision point for each wave.
7. Retire legacy settings. After cutover, remove legacy settings one at a time, with a test run for each.

Streaming jobs need extra care. Structured Streaming queries keep progress in a checkpoint location, so test whether each query can restart from its existing checkpoint on Spark 4, and plan a controlled restart for any query that cannot.

Security while you migrate

Do not treat the extended period for 3.5 as a reason to defer security work. The project’s security page states that Spark’s security features are opt-in and that no deployment mode is secure by default. It strongly discourages exposing clusters, UIs and submission endpoints to the public internet or untrusted networks. Review authentication, encryption and network access on both the old and the new clusters, and check the known issues listed on that page against the versions you run.

How Bassirah can help

Bassirah plans and runs Spark 3.5 to Spark 4 migrations for self-managed clusters as independent engineering, from dependency checks to parallel runs and cutover. See our Apache Spark practice and our migration and modernization work. Bassirah is not a partner of Databricks or the cloud providers’ managed Spark services, and upstream fixes follow the project’s timeline, as our open-source support model explains.