## apache/spark — v4.2.0-rc3…v4.2.0-rc4

_67 commits._

### Features
- **[SPARK-57682][BUILD] Add `javax.servlet-api` to `LICENSE-binary`** (27ab790)
- **[SPARK-57687][BUILD] Add `arrow-compression` to `LICENSE-binary`** (26bddfc)
- **[SPARK-57589][CORE] Support `spark.ui.contentSecurityPolicy.enabled`** (484cd71)
- **Revert "[SPARK-56555][K8S] Add ubuntu mirror repositories to K8s Dockerfile for redundancy"** (726adfb)
- **[SPARK-57452][DOC] Add missing 4.2 behavior changes to migration guides** (eee39a2)

### Fixes
- **[SPARK-57721][SQL] Fix `LowLatencyMemoryStream` package to match its directory** (4da95e7)
- **[SPARK-57720][SQL] Fix `RealTimeModeAllowlist` package to match its directory** (2b014d6)
- **[SPARK-57686][BUILD] Fix `log4j-slf4j2-impl` artifact name in `LICENSE-binary`** (5fff7a7)
- **[SPARK-57650][YARN][TEST] Allow AMs to use the whole queue in BaseYarnClusterSuite to fix ACCEPTED-state timeouts** (10427c3)
- **[SPARK-57438][SS] Fix NullPointerException in Kafka source metrics when latest partition offsets are unavailable** (3fa6f9f)
- **[SPARK-57547][SQL] Fix incorrect InMemoryRelation materialization under conrrent queries** (37fae57)
- **[SPARK-57520][SQL] Fix UTF8String.codePointFrom and copyUTF8String reading past the end of a truncated trailing UTF-8 sequence** (e97565b)
- **[SPARK-57506][SQL] Fix RTRIM-collation trimRight dropping trailing spaces for strings with supplementary characters** (f1f369d)
- **[SPARK-57507][SQL] Fix `UTF8String.reverse` reading past the end of a truncated trailing UTF-8 sequence** (8ed470b)
- **[SPARK-57371][CONNECT] Fix data race in ArrowDeserializers** (9eb44a4)
- **[SPARK-57529][SQL] Fix possible ORC reader leak in OrcPartitionReaderFactory** (478c317)
- **[SPARK-57364][SQL] Fix Oracle TRUNC pushdown to map Spark format strings to Oracle format** (cea76e4)
- **Revert "[SPARK-57445][SQL] Fix PushVariantIntoScan exception semantics."** (16086bc)
- **[SPARK-57445][SQL] Fix PushVariantIntoScan exception semantics.** (5e9c4b8)
- **[SPARK-57482][CORE][TEST] Fix flaky SparkLauncherSuite.testInProcessLauncher under CI load** (79db6cd)
- **[SPARK-56395][SQL][FOLLOWUP] Fix RewriteNearestByJoin nullability for LEFT OUTER** (87f9c71)

### Backend
- **Preparing Spark release v4.2.0-rc4** (f92a807)
- **[SPARK-57719][SQL][TEST] Move `IdentifierClauseParserSuite` to the catalyst parser package** (38feeea)
- **[SPARK-57652][4.2][SS][TEST] Deflake snapshotStartBatchId-with-transformWithState StateDataSource test** (87ac6bd)
- **[SPARK-57544][SQL] Rework column ID validation for nested fields in DSv2** (eadcd9a)
- **[SPARK-57697][SQL][TEST] Mark `HiveClientSuite` as `SlowHiveTest` like `HiveClientSuites`** (29d0bc6)
- **[SPARK-57599][SQL] Decode Variant strings and keys as UTF-8** (f46a990)
- **[SPARK-57689][BUILD] Replace `kubernetes-httpclient-okhttp` with `kubernetes-httpclient-vertx` in `LICENSE-binary`** (1f4bb05)
- **[SPARK-57662][CORE] Use unsigned byte comparison for KVStore iterator keys** (80c1964)
- **[SPARK-57683][SQL][TEST] Move `StateDataSourceTransformWithStateSuiteWithRowChecksum` to `ExtendedSQLTest`** (bd68c58)
- **[SPARK-57675][SQL][TEST] Mark `StateDataSourceTransformWithStateSuiteCheckpointV2` as `ExtendedSQLTest`** (02df81d)
- **[SPARK-57667][SQL][TEST] Make SQL tests be independent from the default value of `maxShuffledHashJoinLocalMapThreshold`** (a9b8163)
- **[SPARK-57642][SQL] Require predicateSql to be present for the DSv2 CHECK constraint** (96ff1c2)
- **[SPARK-57647][SQL][TEST] Make `FileSourceStrategySuite` be independent from the default value of `maxPartitionBytes`** (c3d73f8)
- **[SPARK-57595][PYTHON] Declarative Pipelines analysis context should not mask the original error when registration fails** (fd33cc3)
- **[SPARK-57637][BUILD][TEST] Upgrade `oracle-free` docker image to `23.26.2-slim`** (54a22fb)
- **[SPARK-57601][CONNECT][UI] Make the Spark Connect tab available in the History Server** (9791e6e)
- **[SPARK-52729][SQL][FOLLOWUP] Model table and view as a common Relation** (bf78164)
- **[SPARK-57610][4.2][SQL][TEST] Tolerate null ArithmeticException message in TruncTimestamp overflow test on JDK 25** (3c5f606)
- **[SPARK-57581][4.2][SQL] Encode the TIME data type in Avro with a unit-correct logical type** (f223a16)
- **[SPARK-56919][SQL] Move setupJob before materializeAdaptiveSparkPlan to prevent table path loss** (f49cb12)
- **[SPARK-57329][SQL] mode() returns incorrect result when input contains both -0.0 and 0.0** (45e4208)
- **[SPARK-54918][SQL] Clean up text of `NormalizeFloatingNumbers` docstring** (387dc07)
- **[SPARK-57580][SQL] Redact credentials embedded in JDBC URLs** (245179e)
- **[SPARK-56032][SQL][FOLLOWUP] Skip FilterExec CSE codegen when every common subexpression is cheap** (360bab8)
- **[SPARK-57538][YARN] Revise a warning message in `YarnShuffleService`** (b8fab80)
- **[SPARK-57336][CONNECT] Send the bearer token in the standard Authorization header** (2cfb0ab)
- **[SPARK-57505][SQL] Do not wrap window functions in an `AggregateExpression` when converting an expression-backed `Column`** (c383a94)
- **[SPARK-57477][PYTHON][TEST] Remove `labels` test for boxplot** (445e35f)
- **[SPARK-57510][CORE] Enforce the modify ACL on the `/workers/kill` endpoint in MasterWebUI** (e12240f)
- **[SPARK-57496][4.2][SQL][BUILD] Keep the Types Framework ops and UDF worker packages out of the published API** (a0e459c)
- **[SPARK-57509][CORE] Return `FORBIDDEN` instead of `METHOD_NOT_ALLOWED` for disallowed `/workers/kill` requests** (64684f8)
- **[SPARK-57485][BUILD] Exclude package-private Scala types from the generated Javadoc** (534b59d)
- **[SPARK-53209][YARN][FOLLOWUP][4.2] Move `spark.yarn.am.limitActiveProcessorCount.enabled` to yarn module** (979d2dc)
- **[SPARK-56724][INFRA][4.2] Make docker/* GitHub Actions up-to-date** (785ff22)
- **[SPARK-57475][CORE] Validate length before allocation in `Encoders.BitmapArrays.decode`** (7cca942)
- **[SPARK-57474][CORE] Validate length before allocation in `Encoders.StringArrays.decode`** (3960c2a)
- **[SPARK-57498][SQL] Strengthen cookie-signing secret in ThriftHttpServlet** (def9100)
- **[SPARK-57451][CORE] Remove the check rejecting `spark.authenticate.secret` with Master REST server** (a6480c1)
- **[SPARK-57447][SQL] Escape the index name in `(H2|MySQL|Postgres)Dialect.indexExists`** (85a0972)
- **[SPARK-57446][SQL] Apply `escapeSql` to `comment` in JDBC `table/schema` comment queries** (bf50ad9)
- **[SPARK-57439][CORE] Validate length before allocation in `Encoders.(Int|Long)Arrays` decode** (77b3f64)
- **[SPARK-57426][CORE] Validate length before allocation in `Encoders.Strings.decode`** (b7ba53c)
- **[SPARK-57430][CORE] Validate length before allocation in `Encoders.ByteArrays.decode`** (04ca5e6)
- **[SPARK-57429][SQL] Revise debug message in `ThriftHttpServlet`** (68ae6b1)
- **[SPARK-57434][SQL][TEST] Move JavaGeographyTypeSuite and JavaGeometryTypeSuite to the Java test source root** (8f9da70)
- **Preparing development version 4.2.1-SNAPSHOT** (b4b48e3)

_Recap by [Repo Wrapped](https://repowrapped.com/gh/apache/spark?utm_source=github-action)._