## NVIDIA-NeMo/Curator — v1.1.0…v1.2.0rc0

_153+ commits._

### Features
- **Add GPU marker to test_per_stage_runtime_env.py (#1770)** (e93305e)
- **Add Interleaved filters (#1583)** (a12858b)
- **Add Nemotron-Parse PDF processing pipeline (#1693)** (d4d2fd6)
- **[benchmarking] updates SlackSink and process logging to support parallel benchmark runs (#1678)** (94904c7)
- **Add `transformers_init_kwargs` to Transformer's `from_pretrained` function (#1357)** (c03a1f5)
- **Add DNS Challenge Read Speech tutorial and dataset stage (#1664)** (b8ed627)
- **Add AudioDataFilterStage composite pipeline for end-to-end audio curation (#1640)** (34a76ef)
- **Add SIGMOSFilterStage for audio quality assessment (#1577)** (7799d58)
- **Add VADSegmentationStage for voice activity detection (#1578)** (166fb5c)
- **feat(interleaved): add InterleavedParquetReader and InterleavedWebdatasetWriter (#1657)** (5fbad20)
- **Add UTMOSFilterStage for audio quality filtering (#1639)** (5f8f7d3)
- **Add SpeakerSeparationStage for speaker diarization (#1579)** (0f78876)
- **Add ADV audio processing dependencies (#1574)** (8c2dab8)
- **add audio stage-wise profiling and benchmark enhancements (#1676)** (04ff088)
- **Add BandFilterStage for audio bandwidth classification (#1576)** (a767ba2)
- **Add audio preprocessing stages (MonoConversion, SegmentConcatenation, TimestampMapper) (#1575)** (6237215)
- **Add math modality benchmarking support (#1604)** (23db077)
- **feat(interleaved): Foundation IO improvements — schema utilities, base-class fixes, and bug fixes (#1652)** (0297167)
- **Add Nemotron VLM support in video captioning (#1160)** (53508d6)
- **Add more args to fuzzy workflow (#1603)** (78b0672)
- **Add Fern docs preview workflow (#1660)** (1f0462d)
- **Add some basic workflows for the GitHub project board (#1648)** (424de69)
- **Add ray data for image (#1610)** (7303d91)
- **Add ray data for audio modality (#1592)** (e55623a)
- **Support Ray Serve through InferenceModelConfig and InferenceServer (#1541)** (0cdf763)
- **Add ray to each benchmarking entry and make sure no benchmarking/script has RayClient (#1593)** (0ba6869)
- **Add ALM Data Pipeline tutorial and stages (#1419)** (7c850c4)
- **Support inserting based on batchsize into shuffler (#1369)** (63c5b32)
- **[REVIEW] Add multimodal reader/writer pipeline and scoped benchmarks (#1517)** (3e2145d)
- **Add Math pipeline (#1058)** (29d1e56)
- **Add supporting max_calls to the ray data executor (#1534)** (7278160)
- **Add retry on 503 error (#1496)** (8c17d17)
- **Introduce ray status verification to get client (#1181)** (6de7ccc)
- **[REVIEW] Add model model_weights_path (#1503)** (eaae339)
- **[benchmark] Add FastText filter benchmarking script (#1411) (#1452)** (235fa2e)
- **Support Nemo Data Designer for Curator (#1449)** (b83c9ff)
- **Add feedback to tutorials (#1476)** (a886901)
- **Add check to make sure all reserved keywords are attributes and not properties (#1490)** (2762f6e)
- **Add progress bars for actor pool executor (#1457)** (568f86c)
- **[benchmark] Add Video Benchmarks (#1430)** (c022a7e)
- **[benchmarking] Add Semantic Deduplication Identification (#1410)** (983f11e)
- **Add benchmarking for modifiers (#1407)** (d177aaa)
- **Add metrics for `ScoreFilter` benchmarks (#1385)** (2ac59c8)
- **[benchmarking] Adds support for WorkflowRunResult, refactors benchmark scripts (#1377)** (2f8f3f8)
- **Add warning for small n_clusters in SemanticDeduplicationWorkflow (#1376)** (1ba3e20)
- **Add benchmarking for `ScoreFilter` (#1373)** (e4fe2a7)
- **Add vLLM and Sentence Transformers support for embedding generation (#1346)** (0982d8a)
- **Add workflow results (#1275)** (a8e0040)

### Fixes
- **fix CVEs: nemo-toolkit >=2.7.2, xgrammar >=0.1.32, delete ray_dist.jar (#1612)** (6a5ccf5)
- **Make vLLM as default for text embedding generation in Semantic Dedup Workflow + vLLM <> Ray Data bug fix (#1606)** (a88826b)
- **Fix arxiv benchmark doc count metrics: use add_id stage for input docs (#1605)** (a4f8db5)
- **Fix ruff formatting for interleaved module introduced in #1517 (#1572)** (d378727)
- **Fix benchmarking failures caused by Ray cluster cleanup and responsiveness checks (#1556)** (9e893bf)
- **Fix bugs in prometheus / grafana setup and support clusters with multiple ray users (#1523)** (0f4ae7d)
- **(fix): refactor audio stage names to be shown after running benchmark (#1470)** (53d943a)
- **Fix: fasttext predict call for numpy>2 (#1482)** (53a320e)
- **Fix broken NeMo link (#1463)** (2e6ae1a)
- **Fix vllm API compatibility with Video Pipeline + Upgrade vLLM to 0.14 (#1421)** (605321b)
- **[benchmarking] Bug fixes and UX improvements (#1409)** (9735c69)
- **Fix bug in Gliner tutorial (#1372)** (84c7758)
- **Fix bug in SDG example (#1370)** (a086dd4)

### Backend
- **[Curator video caption pipeline] integration tests(vLLM + HF transformers) for Qwen  (#1614)** (535e677)
- **[benchmarking] uses `gpustat` utility before/after each entry, report memory useage to console and Slack, capture all entry logs to `stdouterr.log` (#1751)** (9e75b5d)
- **Move ray actor pool out of experimental (#1754)** (7fc9de4)
- **[Feat] Per Stage Runtime Env  (#1623)** (da88629)
- **Improve error messages on large file reads (#1630)** (e89b14e)
- **ipynb Notebook for NDD (#1598)** (931420f)
- **Bump few package versions with CVEs (#1736)** (eb56765)
- **Adding Streaming Sortformer Stage (#1573)** (cccfefc)
- **Update README with Nemo-Data Designer-based SDG (#1670)** (1763b53)
- **Bump onnx from 1.20.0 to 1.21.0 (#1713)** (6ce5826)
- **Bump pyasn1 from 0.6.2 to 0.6.3 (#1616)** (4a40bdd)
- **Bump pyopenssl from 22.0.0 to 26.0.0 (#1669)** (f13eefe)
- **Improve `split_large_files` script (#1658)** (550ca17)
- **Llane/fern docs off main (#1647)** (54bd313)
- **Bump pygments from 2.19.2 to 2.20.0 (#1681)** (5359217)
- **Bump cryptogrpahy and update lockfile, also added min version for uv to prevent lockfile rev from changing (#1682)** (b651462)
- **Update token handling for classifiers (#1528)** (ecb77af)
- **more project board gha updates (#1663)** (a4d5e0e)
- **Nemotron-CC SDG recipe using NDD (#1588)** (c6b5172)
- **Inital codeowners setup (#1645)** (33078e3)
- **Move ray data out of experimental folder (#1619)** (f3d3b84)
- **[benchmarking] Update ndd_ray_serve timeout (#1622)** (863a1a0)
- **Update exact value for num_clips_generated in nightly benchmark configuration from 300 to 377 (#1599)** (6789731)
- **Remove nvenc/decs plus update Xenna to 0.2.0 (#1571)** (0bc7337)
- **fixed (#1590) vLLM setup race condition** (e1441bb)
- **Adds ability to ping specific users on Slack when corresponding benchmarks fail (#1589)** (9b820b0)
- **Update ray to 2.54 (#1557)** (b795fe2)
- **Expose a few more args to the exact dedup workflow (#1561)** (f4aee3d)
- **Fixes #1232 reorganize document filter document modifier dir (#1472)** (43d838e)
- **Update test input args to account for inserted magicmock objects (#1535)** (333506c)
- **Restore CHANGELOG.md (#1531)** (4d88dc0)
- **Update ruff.yml (ci/cd) to 0.14.10 to be consistent with pre-commit / pyproject.toml (#1525)** (71de538)
- **Warn and resolve conflict when both blocksize and files_per_partition are specified (#1508)** (c626c1d)
- **Refactor video frame extraction to improve PyNvCodec availability check (#1511)** (d3bd608)
- **Remove MegatronTokenizerWriter from writer/__init__.py (#1418) (#1507)** (4bd1d85)
- **[benchmarking] Updates Slack sink to address table size limitation (#1480)** (eeaa0a5)
- **Update tutorial to be more explicit about num_gpus (#1492)** (731a24f)
- **Update transformers dependency to exact version 4.55.2 in pyproject.t… (#1471)** (f1eccef)
- **Llane/sdg ray docs (#1347)** (43fdcc9)
- **Fuse document iterate and extract stages (#1458)** (b7febb3)
- **Video tutorial improve (#1367)** (b0f8ef2)
- **Purge InternVideo2 (#1451)** (3dd213b)
- **[REVIEW] Benchmarking Script for E2E Pipeline (#1389)** (f85c3c5)
- **Benchmarking script for image pipeline (#1441)** (f2263a8)
- **heuristic page updates (#1422)** (b524c15)
- **standardize ID field names across deduplication workflows (#1390)** (f5846be)
- **Exact dedup identification benchmark (#1400)** (4cdf6c7)
- **Ray Pool Executor (#1415)** (3974061)
- **Clarify instructions for downloading the Llama Nemotron Post-Training Dataset (#1416)** (4d86ee1)
- **Bump pyasn1 from 0.6.1 to 0.6.2 (#1396)** (86c9976)
- **[tests] Improve speed for SemDedup unit tests #1412** (6348b44)
- **Increases dedup_removal timeout for raydata to 1100s (#1406)** (1679597)
- **Fixed `MegatronTokenizerWriter` to download just the tokenizer files (#1392)** (723fbd3)
- **Update instructions for AWS credentials in ArXiv download and extract tutorial (#1380)** (058fa93)
- **Revert "Remove nvenc/dec for xenna 0.1.6 (#1202)" (#1374)** (ac77f88)
- **[benchmarking] Update metrics to track for text (#1386)** (3fbb0b1)
- **Clean up benchmarking scripts (#1382)** (37b1b63)
- **Address aiohttp and urllib3 cve (#1379)** (f8e6f79)
- **[benchmarking] Adds audio curation benchmark to nightly (#1360)** (fb88643)
- **[benchmarking] Adds image curation benchmark to nightly (#1341)** (d9ade75)

### Tests
- **test: Update ci benchmark (#1756)** (dcdebbf)

### Docs
- **docs: vdr feedback (#1477)** (242c9aa)

### Chore
- **ci: Update codecov yml to correct curator path (#1782)** (d6eccb9)
- **ci: Remote docker cache + gpu test parallelization (#1779)** (4890e77)
- **ci: Update onnuxruntime to x86 only (#1744)** (c77cbac)
- **ci: Resolve wandb CVE (#1739)** (afee1d1)
- **chore: bump FW-CI-templates workflow pins to v0.88.0 (#1735)** (f86296a)
- **ci: Update coverage download path (#1707)** (705e780)
- **chore: bump _code_freeze workflow to v0.86.0 (#1692)** (e05b442)
- **ci: upgrade GitHub Actions for Node.js 24 compatibility (#1638)** (e40f781)
- **ci: Move benchmarking testing dep into pyproject (#1666)** (febc404)
- **refactor(audio): AudioBatch -> AudioTask single-dict redesign (#1608)** (1b6023b)
- **ci: Adding automation config for benchmark test (#1597)** (01fc6fc)
- **chore: test FW-CI-templates ko3n1g/fix/linkcheck-retry-backoff (#1631)** (066f37d)
- **ci: Override nixl cu12 from being installed on arm (#1628)** (ec27874)
- **ci: Update timeout to 30 minutes for gpu tests (#1629)** (064acf2)
- **ci: Fix sso user check (#1627)** (5fd58bc)
- **ci: Enable claude review (#1615)** (fd6bab5)
- **ci: Set hf token to environment (#1558)** (b96a0d9)
- **ci: Clean up copybot config (#1543)** (e42ceb7)
- **ci: Bump Curator release workflow to include generated changelog (#1522)** (0351df2)
- **ci: Address new CVEs from rc4 (#1497)** (582ed0d)
- **ci: Remove thirdparty aiohttp file from ray (#1469)** (90ce791)
- **ci: Update vllm to 0.14.1 and override conflict (#1467)** (c2c626c)
- **ci: Add secrets detector (#1447)** (e008525)
- **ci: Update cve for python-multipart (#1450)** (68590a4)
- **ci: Optimize docker layer and uv with no cache (#1444)** (9f28a59)
- **ci: Address setuptools CVE (#1438)** (d3bb54a)
- **ci: Enable AWS runners (#1388)** (d5ef575)

_Recap by [Repo Wrapped](https://repowrapped.com/gh/NVIDIA-NeMo/Curator?utm_source=github-action)._