## apache/spark — v4.2.0-rc5…v4.2.0-rc6

_45 commits._

### Features
- **[SPARK-57974][PYTHON][DOC] Add pandas upper bound `<3.0.0` in `dev/requirements.txt` and `install.rst`** (904bfea)
- **[SPARK-57966][YARN][DOC] Update Java 17 support description in `running-on-yarn.md` for Hadoop 3.5.0** (8f78c87)
- **[SPARK-57957][ML][TEST] Deflake GaussianMixtureSuite 'GMM support instance weighting' on macOS using well-posed data** (6a83f31)

### Fixes
- **[SPARK-56227][CORE] Fix GcmTransportCipher to correctly handle multiple messages per channel** (c188cb7)
- **[SPARK-58047][4.2][DOC] Fix the required NumPy version to 1.21 in MLlib guide** (2a4349f)
- **[SPARK-58038][DOC] Fix to use Hadoop 3.5 in PySpark installation guide and Spark Connect overview** (bf5ccb4)
- **[SPARK-58034][DOC] Fix `Connect Overview` doc to use `site.SPARK_VERSION`** (657687c)
- **[SPARK-58033][PYTHON][DOC] Fix `py4j` version requirement in PySpark installation guide** (a66e3fe)
- **[SPARK-57958][SQL][FOLLOWUP] Close ORC schema-inference Reader uninterruptibly to fix leaked file streams** (21df7e2)
- **[SPARK-57961][SQL][TEST][FOLLOWUP] Fix scalastyle line length violations in MetricsFailureInjectionSuite** (a45dd16)
- **[SPARK-57958][SQL][TEST] Close the ORC Reader in OrcSourceSuite to fix leaked file streams** (0a4fafe)
- **[SPARK-57993][SS][TEST] Fix flaky RocksDBStateStoreIntegrationSuite bounded memory usage test by isolating global RocksDBMemoryManager** (0ae1b97)

### Backend
- **Preparing Spark release v4.2.0-rc6** (32f7299)
- **[SPARK-56972][SS][FOLLOWUP] Reject sink evolution combined with async progress tracking** (7a25c75)
- **[SPARK-58024][PYTHON] Convert Arrow struct and map columns to Python rows in bulk** (44daf01)
- **[SPARK-58019][PYTHON] Convert Arrow list columns to Python rows in bulk** (d0856c7)
- **[MINOR][UI][4.2] Encode query parameters when generating log page scripts** (e79fd9b)
- **[SPARK-57962][PYTHON] Guard against path traversal in install_spark tar extraction** (fc46f2a)
- **[MINOR][UI] Escape HTML when rendering error messages in the Web UI tables** (380776d)
- **[SQL] Allow more control over UDT creation during input loading** (c40f044)
- **[SPARK-58042][CONNECT] Validate the UDT jvm_class is a UserDefinedType before instantiating it** (5a347f9)
- **[SPARK][SQL] LDAP Thriftserver improvement** (e3c6298)
- **[SPARK-57775][4.2][SQL] Return one row for GROUP BY GROUPING SETS (()) over empty input** (4f2d78a)
- **[SPARK-58057][K8S][TEST] Make VolumeSuite OnDemand PVC test robust to slow container startup** (3689140)
- **[SPARK-58030][SQL][TEST] Make FileDataSourceV2FallBackSuite robust to extra listener events** (396178d)
- **[SPARK-58028][INFRA][TEST] Retry Docker image pull in DockerJDBCIntegrationSuite to avoid transient 502 aborts** (4a0eda5)
- **[SPARK-58026][INFRA] Bound SparkR Windows test step to avoid silent 2h job timeout** (1b08b5d)
- **[SPARK-58025][INFRA] Redirect build/mvn fallback message to stderr to avoid corrupting version captures** (9b0423b)
- **[SPARK-58018][SQL][TEST] Deflake AQE 'should avoid to submit shuffle job on cancellation' test** (f29f5cf)
- **[SPARK-58017][SS][TEST] Deflake KafkaSourceStressForDontFailOnDataLossSuite** (35277aa)
- **[SPARK-57963][SS] Restore interrupt status unexpectedly cleared so that `StreamExecution.stop` can stop micro-batch streaming query** (185d5df)
- **[SPARK-57960][SQL] Apply escapeSql to table/schema identifiers in JDBC index metadata queries** (8f82d56)
- **[SPARK-57991][CORE][TEST] Deflake BarrierTaskContextSuite wall-clock skew assertions on macOS-26** (5444e84)
- **[SPARK-57959][SQL][TEST] Deflake MetricsFailureInjectionSuite non-deterministic injection test** (eb88a2b)
- **[SPARK-57951][ML][TEST] Tolerate last-ULP FP differences in MLTest single-prediction checks for macOS** (2f2901d)
- **[SPARK-57952][CORE][TEST] Retry retryOnOOM a few times to reduce SorterSuite flaky OOM** (7322c77)
- **[SPARK-57947][SS][KAFKA][TEST] Deflake KafkaSourceStressForDontFailOnDataLossSuite** (fdc585d)
- **[SPARK-57710][YARN][TEST][FOLLOWUP] Size YarnClusterSuite mini NodeManager via the yarn.minicluster.* key** (d286518)
- **[SPARK-57710][YARN][TEST] Reduce YarnClusterSuite flakiness from CI runner contention** (fec32fc)
- **[SPARK-57945][K8S] Avoid persisting Java options to disk in `entrypoint.sh`** (24cfbbc)
- **[SPARK-57882][SQL] Enforce AuthV2 metadata authorization on GetPrimaryKeys and GetCrossReference operations** (8e47b13)
- **[SPARK-57920][CORE][SQL][YARN] Use `Files.createTempFile` to create temporary files with owner-only permissions** (141c975)
- **[SPARK-57889][CORE] Authorize StreamRequest consistently with ChunkFetchRequest in the transport layer** (e16ae3a)
- **[SPARK-57803][CORE] Delete the temp file when closeAndRead fails on the fetch-to-disk path** (90c920b)
- **Preparing development version 4.2.1-SNAPSHOT** (210733c)

_Recap by [Repo Wrapped](https://repowrapped.com/gh/apache/spark?utm_source=github-action)._