## apache/spark — v4.2.0…v4.2.1-rc1

_134 commits._

### Features
- **[SPARK-59798][4.2][K8S] Support `spark.executor.extraLibraryPath` for K8s executors** (7a6fdcd)
- **Revert "[SPARK-59798][K8S] Support `spark.executor.extraLibraryPath` for K8s executors"** (b9a3fe9)
- **[SPARK-59798][K8S] Support `spark.executor.extraLibraryPath` for K8s executors** (e5dd1ca)
- **[SPARK-59087][4.2][CONNECT][FOLLOWUP] Add extraServerConfs hook to SparkConnectServerTest** (41092d0)
- **[CORE][MINOR] Add RDD name escaping to avoid breaking DAG visualization** (f5a1e0f)
- **[SPARK-58871][INFRA] Add timeout to disk cleanup steps** (db2eea4)
- **[SPARK-58678][CORE][TEST] Add `OpenHashMapBenchmark`** (f56544a)
- **[SPARK-58334][K8S][DOC] Introduce Apache Spark K8s Operator in `running-on-kubernetes.md`** (c37f938)
- **[SPARK-58335][K8S][TEST] Add `SparkKubernetesClientFactorySuite`** (8b029ae)
- **[SPARK-57234][4.2][SS][DOC] Add Real-time Mode documentation page to the Structured Streaming guide** (5471de0)

### Fixes
- **[SPARK-59406][UI] Fix UI display on weird custom log names** (e938550)
- **[SPARK-59688][4.2][SQL] Fix wrong results when a storage-partitioned join skips the reduce on coinciding partition keys** (a341144)
- **[SPARK-59562][ML] Fix stale values in reused multinomial logistic aggregation buffer** (6641056)
- **[SPARK-58713][SQL] Fix integer overflow in overlay() position arithmetic** (d470d7a)
- **[SPARK-59108][4.2][SQL] Fix Avro positional matching under column pruning** (84c89bf)
- **[SPARK-58708][SQL] Fix out-of-range length handling in substring/lpad/rpad** (441313d)
- **[SPARK-58996][4.2][SQL] Fix SPJ partially clustered data correctness when EnsureRequirements re-runs** (bc8e972)
- **[SPARK-58985][CORE] Fix HistoryServerDiskManager double-counting store size on concurrent release and makeRoom** (5f451e1)
- **[SPARK-59252][SQL] Fix an SPJ scan reporting a different partitioning at execution than at planning** (2ffb252)
- **[SPARK-59011][PS] Fix descending rank first tie ordering** (76ba3bb)
- **[SPARK-59176][4.2][SQL] Fix a storage-partitioned join that fails when one side reduced onto no key** (9dff80a)
- **[SPARK-59121][4.2][SQL] Fix wrong results when a storage-partitioned join reduces the partition keys of both sides** (06f41f0)
- **[SPARK-58968][4.2][SQL] Fix SPJ allowKeysSubsetOfPartitionKeys correctness for non-join operators** (5512ab9)
- **[SPARK-59045][4.2][SQL] Fix SPJ ClassCastException when reducer changes partition key data type** (773f49a)
- **[SPARK-58775][UI] Fix SQL metrics filter and min,med,max table interaction** (5f21938)
- **[SPARK-59120][4.2][SQL] Fix ClassCastException in a storage-partitioned join whose partition keys were reduced** (483b021)
- **[SPARK-59054][CORE][SQL] Fix `KeyGroupedPartitioner` to compare partition keys by value in storage-partitioned join shuffles** (6686830)
- **[SPARK-59084][CORE] Fix anchors styled as buttons to keep the button hover style** (cc520d4)
- **[SPARK-59066][CORE] Fix History Server Event Log `Download` button to use a Bootstrap 5 class** (08805ea)
- **[SPARK-59067][CORE] Fix `Download` to be a button aligned with the other buttons** (38ad1b3)
- **[SPARK-55271][SS] Fix NullPointerException in Kafka micro-batch streaming metrics reporting** (b7150f0)
- **[SPARK-59008][K8S] Fix NPE in ExecutorPodsLifecycleManager when pod is deleted between get() calls** (650761b)
- **[SPARK-59025][SQL] Fix subset join key grouping when the keyed side reports a PartitioningCollection** (91cd9a3)
- **[SPARK-58994][ML] Fix prediction column metadata in GBTRegressionModel** (36f7c71)
- **[SPARK-58627][SQL] Mark raise error Throwable & fix sequence throwable** (fac1785)
- **[SPARK-58886][CORE] Fix Int overflow in `CoarseGrainedSchedulerBackend.requestExecutors`** (36116e4)
- **[SPARK-58714][SS] Fix AdminClient leak in KafkaTokenUtil.obtainToken** (97a020f)
- **[SPARK-58296][SQL] Fix to_time returning STRING type when the format is a foldable NULL** (9b28825)
- **[SPARK-52246][SQL][TEST] Add bucket transform regression test for one-side shuffle with join key tail of partition keys** (05e5a26)
- **[SPARK-58341][SQL] Fix wrong results with 5+ positional parameters in `sql`** (fc6979c)
- **[SPARK-58165][SQL][CONNECT] Fix SubqueryExpression dropping nested subquery plans in IN values** (c19664f)
- **[SPARK-58241][4.2][PYTHON] Fix the columnar Arrow Python UDF runner writing the input schema in the legacy protocol position** (44432cb)
- **[SPARK-57521][ML][CONNECT] Use parentless copy in Model.estimatedSize to fix ML cache overcounting** (5de73b2)
- **[SPARK-58223][SQL] Fix SparkSession.Builder.classic() building a Connect session** (6a705bd)
- **[SPARK-58102][SQL] Fix SLAM scope extraction bailing out on ReusedSubqueryExec** (17c9fde)
- **[SPARK-58100][SQL] Use IndexedSeq to fix quadratic complexity in partition resolution** (38c4753)
- **[SPARK-57932][4.2][SQL] Fix regexp_instr and regexp_replace returning wrong results for supplementary characters** (666afb8)

### Backend
- **Preparing Spark release v4.2.1-rc1** (72c1876)
- **[SPARK-59353][4.2][UI] Avoid pre-fetch and crawlers triggering actions in UI- #59029** (69bef14)
- **[SPARK-59407][4.2][UI] Improve compressed line handling in Spark History server** (5e2e9f6)
- **[SPARK-59087][CONNECT] Authenticate before the other gRPC interceptors run** (209ab15)
- **[SPARK-59673][SQL] Floor the NAAJ broadcast threshold** (3c04f6d)
- **[SPARK-59334][CORE] Handle invalid application/worker links** (206c987)
- **[SPARK-59352][SQL] Validate prefix lengths in the vectorized DELTA_BYTE_ARRAY reader** (d7c7f5f)
- **[SPARK-59405][SQL] XSD Validation improvements** (a2414d9)
- **[SPARK-59508][PYTHON] Harden client safety with restricted unpickle serializer** (31e8ae9)
- **[SPARK-59602][SQL] Handle floating-point normalization in array set operations** (de0f99c)
- **[SPARK-59656][K8S] Skip `spark.exit-exception` annotation in the submission client of cluster mode** (71dc4c4)
- **[SPARK-59641][K8S] Redact `spark.exit-exception` annotation with `spark.redaction.string.regex`** (e848706)
- **[SPARK-59572][4.2][SQL][FOLLOWUP] Clarify PartitionPredicate migration guide entry** (cead927)
- **[SPARK-59572][SQL] PartitionPredicate evaluation failure must not be reported as a partition match** (5ef9607)
- **[SPARK-36082][SQL][FOLLOWUP] Restore NAAJ broadcast hash join by default** (bdeb68d)
- **[SPARK-59439][CORE] Release the HistoryServerDiskManager lease reservation exactly once on a failed commit** (ec7f0ff)
- **[SPARK-59081][SQL] HiveSessionStateBuilder does not apply injectHintResolutionRule** (aa31ad9)
- **[SPARK-59468][K8S][TEST] Replace dead minio/minio image with alpine/minio** (b480f41)
- **[SPARK-59434][SQL] Non-deterministic predicates should not be pushed down to CTE definitions** (ac091c6)
- **[SQL][MINOR] Bound nesting depth when converting protobuf Any to JSON** (3a363c2)
- **[SPARK-58897][4.2][SQL] Propagate NaN through the infinity norm in vector_norm** (6c3ac3a)
- **[SPARK-59279][4.2][SQL] Don't re-read the sorted-merge config after GroupPartitionsExec is planned** (080af70)
- **[SPARK-59212][4.3][K8S][INFRA] Pin the Minikube container runtime to docker in K8s CI** (2bc782e)
- **[SPARK-58821][SQL] Return the recomputed length instead of an internal error in Sequence** (3544f46)
- **[SPARK-59187][4.2][SQL] Compare partition key rows at types with the naming erased** (edd513b)
- **[SPARK-59316][K8S][DOC] Remove the obsolete `-Pvolcano` build instruction from K8s docs** (61b4489)
- **[SPARK-59050][4.2][SQL] SPJ one-side shuffle with out-of-set keys produces wrong results in multi-joins** (fc65d76)
- **[SPARK-59080][SQL][4.2] Pick one ShuffleSpecCollection member for the SPJ pushdown and the re-shuffle** (c4b1024)
- **[SPARK-59141][4.2][SQL] Interleave partitions in columnar UnionExec** (3c705cb)
- **[CORE][MINOR] Validate array counts against received bytes in shuffle fetch decoders** (fcd6894)
- **[PYTHON][MINOR] Use a unique per-run rendezvous id for distributed torch training** (f87f67d)
- **[MINOR][SQL] Validate DELTA_BINARY_PACKED page header in the vectorized reader** (0da2811)
- **[MINOR] Improve resilence on raw socket stream and steer people towards higher level streaming APIs** (6d2e644)
- **Match the pattern of receive in AbstractAuthRpcHandler** (2ba15b4)
- **Refinements to stream id bookkeeping in OneForOneStreamManager** (6c4182f)
- **[CORE][MINOR] Expand default redaction regex** (8c031c8)
- **[SPARK-58000][CONNECT][SS][TEST] Mitigate flakiness between `streaming query listener` and `listener event` in `ClientStreamingQuerySuite`** (2c5a047)
- **[SPARK-58836][UI] Respect UI root when loading thread dump flame graph resources** (bb00dd2)
- **[MINOR][CORE] Avoid accidental collisions in SHS using LevelDB** (70ad8b4)
- **[SPARK-57242][4.2][CORE] Avoid unbounded page allocation retries after allocator OOM** (c4d9818)
- **[SPARK-59019][4.2][SQL] Make RewriteWithExpression non-excludable** (0e37b68)
- **[CORE][MINOR] Minor tidying in the history page listing** (0cf8163)
- **[MINOR][SQL] Assorted tidying in the JDBC dialects** (ab15fb2)
- **[MINOR] Small adjustments in JsonProtocol event decoding** (71e09fd)
- **[CORE][MINOR] Improve ESS file consistency** (d6d2ad6)
- **[CORE][UI] Escape executor id, host and status in the stage task timeline** (1d9949d)
- **Validate app store path in history server disk management** (82c6c41)
- **RemoteBlockPushResolver validate paths** (0f3ae6b)
- **[MINOR] Improve Transport Context Stream Manager Delegation** (c4ebc4d)
- **[MINOR] Handle client identity after the handshake is complete** (167c2a8)
- **[SPARK-58750][4.2][CORE] Tolerate FileAlreadyExistsException from checkpoint part file rename** (149b29a)
- **[SPARK-59009][SQL] Re-map outputOrdering in InMemoryRelation.newInstance()** (77a245a)
- **[SPARK-58933][4.2][SQL] Resolve expressions in INSERT target IDENTIFIER clauses** (4b8c2c6)
- **[SPARK-58932][4.2][SS] Close the accepted socket in TransformWithStateInPySparkStateServer** (6cb5216)
- **[SPARK-58601][4.2][PYTHON][TEST] Guard mapInPandas/mapInArrow list-return tests with the legacy acceptAnyIterable flag** (4835344)
- **[SPARK-58857][K8S] Bind the result of Utils.randomize in LocalDirsFeatureStep** (49b4fb4)
- **[SPARK-58658][CONNECT] Do not return redacted configurations from the Config RPC** (61fbccb)
- **[SPARK-58834][4.2][INFRA][FOLLOWUP] Bump remaining docker build-push-action pins to allowlisted version** (d9e8f78)
- **[SPARK-58725][K8S] Resolve the pod before deciding whether --kill found it** (e9d3de1)
- **[SPARK-58841][K8S][DOC] Update `YuniKorn` docs with `1.9.0`** (e0ba789)
- **[SPARK-58834][INFRA] Bump docker CI actions to latest ASF-allowlisted versions** (4c3e1a2)
- **[SPARK-58684][PYTHON] Make PySpark error sub-conditions inherit the parent condition error state** (2bd3a47)
- **[SPARK-58751][SS][PYTHON] Stop leaking Python workers on TransformWithState init failure** (bb827ba)
- **[SPARK-57523][SQL] Declarative Pipelines should not retry flows that fail due to streaming source changes** (a05bea6)
- **[SPARK-58682][DOC] Update outdated `OpenHashMap`/`OpenHashSet` performance claims** (d69dd04)
- **[SPARK-58386][SS] Restore backward compatibility of SupportsRealTimeRead.nextWithTimeout** (c0f2762)
- **[SPARK-58529][4.2][PYTHON] Relax RESULT_ROWS_MISMATCH assertion for cross-version old-client compatibility** (ae7559a)
- **[SPARK-58449][SQL] Only push a Join through a Union when the right side is the broadcast side** (f612225)
- **[SPARK-58521][UI] Forward `X-Forwarded-Context` header and skip Master CSP on reverse-proxied UI requests** (1e4d40c)
- **[SPARK-58440][4.2][SQL] Convert Sequence codegen error parameters to String** (faf2474)
- **[SPARK-55199][K8S][DOC] Improve K8s integration tests `README`** (dbe7969)
- **[SPARK-58370][SQL] Check table write privileges when the write target is also read in the same statement** (a39b308)
- **[SPARK-57638][SQL] Avoid busy-waiting in Declarative Pipelines flow resolution** (b7c9225)
- **[SPARK-57635][SQL] Make Declarative Pipelines run-termination reason deterministic** (bc1d958)
- **[SPARK-58324][SQL] Drop unused sameOrderExpressions from GroupPartitionsExec k-way merge ordering** (fb34cbe)
- **[SPARK-58323][SQL] Materialize AliasAware output ordering and partitioning to avoid StackOverflowError** (5915191)
- **[SPARK-58326][K8S][TEST] Pin MinIO image version in DepsTestsSuite** (6d94b7e)
- **[SPARK-58328][K8S] Remove stale TODO comments from `DriverKubernetesCredentialsFeatureStep`** (d8de563)
- **[SPARK-58180][4.2][INFRA] Prepare branch-4.2 build workflows for decoupling** (0e25e40)
- **[SPARK-58120][SQL] doCanonicalize preserves keyGroupedPartitioning expression order** (7de8de3)
- **[SPARK-58019][PYTHON][FOLLOWUP] Convert Arrow list columns to Python rows in bulk at foreachPartition and data source write** (57c17e6)
- **[SPARK-58158][CORE][UI] Escape the executor removal reason in `executorspage.js`** (67248ee)
- **[SPARK-58156][CORE][UI] Escape the executor removal reason in the timeline tooltip** (29caa14)
- **[SPARK-58153][CORE][R] Reject JAR entries escaping the extraction directory in `RPackageUtils`** (a1aec4b)
- **[SPARK-58152][K8S] Avoid logging credentials in Kubernetes client config debug log** (235cf79)
- **[SPARK-58136][INFRA] Automate downloads.md and versions.json updates in the release script** (f87f207)
- **Preparing development version 4.2.1-SNAPSHOT** (2bc48f3)

_Recap by [Repo Wrapped](https://repowrapped.com/gh/apache/spark?utm_source=github-action)._