Skip to content

fix(ptyfs): release the pts file on unregister - #12029

Open
BernardXiong wants to merge 2 commits into
RT-Thread:masterfrom
BernardXiong:fix/ptyfs-pts-file-leak
Open

BernardXiong wants to merge 2 commits into
RT-Thread:masterfrom
BernardXiong:fix/ptyfs-pts-file-leak

Conversation

@BernardXiong

Copy link
Copy Markdown
Member

Summary

ptyfs_unregister_pts() removes the pts file from the root list and recycles its unit number, but never frees the struct ptyfs_file allocated by ptyfs_register_pts(). Because the unit number is recycled, repeated open()/close() of /dev/pts/ptmx leaks one file per cycle without hitting the live-pair limit.

The failure path also had two problems:

  • pts_unit was zero after rt_calloc(), so a failed registration made the free path unregister unit 0.
  • lwp_tty_delete() unconditionally called rt_device_unregister() on a tty that had never been registered when ptyfs_register_pts() failed, corrupting the object list in the OOM path.
  • On lwp_tty_create() failure, the softc was freed before its condvars were detached.

Fix

  • Free the pts file after unlinking it from the root list.
  • Initialize pts_unit to -1 so the free path only unregisters a real unit.
  • Detach the condvars before freeing the softc.
  • In lwp_tty_delete(), only unregister the device if the object was actually registered as a device.

Verification

Built bsp/qemu-virt64-aarch64 with RT_USING_SMART and RT_USING_DFS_PTYFS (aarch64 musl GCC 12.2.0), booted the ext4 rootfs under QEMU, mounted ptyfs and ran 20,000 : < /dev/pts/ptmx open/close cycles:

  • Before the fix: available 245164 KB -> 237932 KB (-7232 KB, about 369 bytes/cycle).
  • After the fix: available 245156 KB -> 245116 KB (-40 KB).

The test ran from the boot shell (root), not from an unprivileged process, and the OOM crash path itself was not forced.

Fixes #12023.

ptyfs_unregister_pts() removed the pts file from the root list and
recycled its unit number but never freed the allocation. Repeated
open/close of /dev/pts/ptmx therefore leaked one struct ptyfs_file per
cycle and eventually exhausted the kernel heap.

Free the file after unlinking it. Also initialize pts_unit to -1 so a
failed registration does not unregister unit 0, detach the condvars
before freeing the softc on the lwp_tty_create() failure path, and skip
rt_device_unregister() for a tty that was never registered.
@github-actions

Copy link
Copy Markdown

👋 感谢您对 RT-Thread 的贡献!Thank you for your contribution to RT-Thread!

为确保代码符合 RT-Thread 的编码规范,请在你的仓库中执行以下步骤运行代码格式化工作流(如果格式化CI运行失败)。
To ensure your code complies with RT-Thread's coding style, please run the code formatting workflow by following the steps below (If the formatting of CI fails to run).


🛠 操作步骤 | Steps

  1. 前往 Actions 页面 | Go to the Actions page
    点击进入工作流 → | Click to open workflow →

  2. 点击 Run workflow | Click Run workflow

  • Use workflow from 保持默认分支(通常为 master)
    Keep the default branch (usually master) in Use workflow from
  • 在 branch 输入框填写 PR 分支 fix/ptyfs-pts-file-leak
    Enter PR branch fix/ptyfs-pts-file-leak in the branch field
  • 设置需排除的文件/目录(目录请以"/"结尾)
    Set files/directories to exclude (directories should end with "/")
  1. 等待工作流完成 | Wait for the workflow to complete
    格式化后的代码将作为独立提交推送至你的分支。
    The formatting changes will be pushed to your branch as a separate commit.

完成后,提交将自动更新至 fix/ptyfs-pts-file-leak 分支,关联的 Pull Request 也会同步更新。
Once completed, commits will be pushed to the fix/ptyfs-pts-file-leak branch automatically, and the related Pull Request will be updated.

如有问题欢迎联系我们,再次感谢您的贡献!💐
If you have any questions, feel free to reach out. Thanks again for your contribution!

@github-actions github-actions Bot added RT-Smart RT-Thread Smart related PR or issues component: lwp Component labels Nov 10, 2026
@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

📌 Code Review Assignment

🏷️ Tag: components

Reviewers: @Maihuanyi

Changed Files (Click to expand)
  • components/dfs/dfs_v2/filesystems/ptyfs/ptyfs.c
  • components/lwp/terminal/freebsd/tty_pts.c
  • components/lwp/terminal/tty_device.c

🏷️ Tag: components_lwp

Reviewers: @xu18838022837

Changed Files (Click to expand)
  • components/lwp/terminal/freebsd/tty_pts.c
  • components/lwp/terminal/tty_device.c

📊 Current Review Status (Last Updated: 2026-11-14 00:16 CST)


📝 Review Instructions

  1. 维护者可以通过单击此处来刷新审查状态: 🔄 刷新状态
    Maintainers can refresh the review status by clicking here: 🔄 Refresh Status

  2. 确认审核通过后评论 LGTM/lgtm
    Comment LGTM/lgtm after confirming approval

  3. PR合并前需至少一位维护者确认
    PR must be confirmed by at least one maintainer before merging

ℹ️ 刷新CI状态操作需要具备仓库写入权限。
ℹ️ Refresh CI status operation requires repository Write permission.

@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

MemBrowse Memory Report · Explore what changed →

at32f415-start

  • ROM: .text +80 B (+0.1%, 65,524 B / 262,144 B, total: 25% used)

esp32-c3

  • dram0_0_seg: .dram0.data +36 B, .stack +4 B (+0.0%, 216,832 B / 325,120 B, total: 67% used)
  • iram0_2_seg: .flash.text +16 B (+0.0%, 421,880 B / 8,388,576 B, total: 5% used)

gd32105r-start

  • CODE: .text +80 B (+0.1%, 69,436 B / 262,144 B, total: 26% used)

hc32f334

  • FLASH: .text +80 B (+0.1%, 99,692 B / 131,072 B, total: 76% used)

hpmicro-hpm5100evk

  • ILM: .fast +32 B (+0.1%, 42,672 B / 262,144 B, total: 16% used)
  • XPI0: .fast +32 B, .text +24 B (+0.1%, 102,980 B / 1,048,576 B, total: 10% used)

hpmicro-hpm5301evklite

  • XPI0: .text +24 B (+0.0%, 76,788 B / 1,048,576 B, total: 7% used)

infineon-psoc6

  • flash: .text +80 B (+0.1%, 116,292 B / 262,144 B, total: 44% used)

k230

  • SRAM: .data +16 B, .eh_frame +432 B, .sbss -4 B, .stack -16 B, .text -4,016 B (-0.3%, 1,391,680 B / 60,682,240 B, total: 2% used)

loongson-ls1cdev

  • Code: .text +540 B (+0.1%, 363,948 B)

nordic-nrf51822

  • FLASH: .text +76 B (+0.1%, 52,132 B / 262,144 B, total: 20% used)

nuvoton-m487

  • CODE: .text +332 B (+0.1%, 356,912 B / 524,288 B, total: 68% used)

qemu-virt64-aarch64

  • Code: .eh_frame +80 B, .text +1,360 B (+0.1%, 1,011,196 B)
  • Data: .bss +16 B (+0.0%, 1,068,120 B)

raspberry-pico-rp2040

  • FLASH: .rodata +40 B, .text +40 B (+0.1%, 114,272 B / 2,097,152 B, total: 5% used)

renesas-ra2l1

  • FLASH: .text +188 B (+0.2%, 98,492 B / 262,144 B, total: 38% used)

simulator

  • Code: .rela.dyn +1,728 B, .rodata +224 B, .text +1,644 B (+0.1%, 2,676,654 B)
  • Data: .data +1,824 B (+0.1%, 1,610,852 B)

stm32f407-rt-spark

  • CODE: .text +80 B (+0.1%, 86,168 B / 1,048,576 B, total: 8% used)

stm32l475-atk-pandora-llvm

  • ROM: .text +92 B (+0.1%, 82,804 B / 524,288 B, total: 16% used)

wch-ch32v208w-r0

  • FLASH: .text +88 B (+0.1%, 159,976 B / 491,520 B, total: 33% used)

x86

  • Code: .rodata +32 B, .text +304 B (+0.2%, 199,277 B)

xuantie-e901plus

  • ISRAM: .rodata +48 B, .text +16 B (+0.1%, 46,488 B / 131,072 B, total: 35% used)

No memory changes detected for:


Catch memory regressions in CI with MemBrowse. Free for open source.

@Rbb666 Rbb666 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

释放遗漏的 pts 文件和失败路径修正方向正确,但新增释放与现有目录读取/属性操作之间缺少生命周期同步,会引入并发访问已释放对象的问题。请先处理下方行内意见,并补充目录遍历与 PTY 创建/关闭并发的验证。

另一个非阻塞建议:标题及提交标题建议统一为 [dfs][ptyfs] Release the pts file on unregister。

本次验证为 diff 与调用链静态核对,未进行板级或 QEMU 并发复现。

rid_bitmap_put(ptsno_pool, ptsno);
/* the file is no longer reachable from the root list */
rt_free(pts_file);
rc = 0;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] 释放 pts_file 前需要同步目录读者的生命周期

这里新增 rt_free(pts_file) 后,仅在 ptyfile_remove_from_root() 中短暂持有 sb->lock 不足以保证没有读者持有这个对象。devpty_deffops_getdents() 在不持有该锁的情况下遍历 root_file.subdirs 并读取 n_file->type/basename;另一个线程关闭 PTY 后可经 ptsdrv_free() 删除并释放同一个节点,目录遍历随后就会访问已释放内存。ptyfile_lookup() 也在返回裸指针前释放锁,ptyfs_ops_setattr() 等调用方仍会在返回后访问它。DFS 外层的 dfs_file_getdents() 和 dfs_file_setattr() 没有与关闭路径共享的全局锁来消除这个窗口。

请让查找/遍历后的使用与摘链释放由同一把锁覆盖,或为 ptyfs_file 引入明确的引用生命周期;仅把 free 移进 remove 的锁区仍不能保护已经取得指针的读者。建议增加一个线程反复打开/关闭 ptmx、另一个线程遍历 /dev/pts(并执行属性查询/修改)的并发验证。

ptyfile_lookup() returned a raw pointer after releasing sb->lock, while
ptyfs_unregister_pts() removed and freed the same node. A concurrent
getdents(), setattr() or vnode lookup could then read freed memory.

Add a reference count protected by sb->lock: lookup takes a reference and
the last ptyfile_put() frees the node. The root list reference is dropped
when the file is removed, so unregistering a pts no longer frees a file
that a reader still uses. getdents() also holds the lock while walking the
subdir list, keeping the entries alive during the copy.
@BernardXiong

Copy link
Copy Markdown
Member Author

已按 [P1] 意见补上生命周期同步,推送到 3cb78faebd:

  • struct ptyfs_file 增加受 sb->lock 保护的引用计数;ptyfile_lookup() 在持锁找到节点时取引用,新增 ptyfile_put() 在最后一个引用释放时 rt_free()。
  • ptyfs_unregister_pts() 依次摘链、释放 ptsno、ptyfile_put();摘链时同时释放根目录列表持有的引用。读者已经取得的指针不会在关闭 PTY 时被释放。
  • devpty_deffops_getdents() 在遍历 root_file.subdirs 期间持有 sb->lock,避免遍历与摘链释放并发;ptyfs_ops_setattr() 的 mode 更新也在锁内完成。
  • ptyfs_ops_lookup()、ptyfs_ops_create_vnode()、ptyfs_ops_setattr()、getdents() 的所有返回路径都会释放 lookup 引用。

验证:bsp/qemu-virt64-aarch64(RT-SMART,启用 DFS v2/PTYFS)编译通过;QEMU 中挂载 ptyfs 后连续 500 次打开/关闭 /dev/ptmx,无内核断言,目录最终只剩 ptmx(无泄漏)。并发 readdir 压力测试受 busybox ash 后台作业影响,未作为结论;后续可以补一个双线程 utest 做回归。

English: lifecycle synchronization added in 3cb78faebd. ptyfs_file now has an sb->lock-protected reference count taken by ptyfile_lookup() and released by ptyfile_put(); unregister drops both the root-list reference and the lookup reference, and getdents() holds the lock while walking the subdir list. Build passes on the RT-SMART aarch64 BSP, and 500 open/close cycles of /dev/ptmx in QEMU left no leaked nodes.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

component: lwp Component RT-Smart RT-Thread Smart related PR or issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] ptyfs: ptyfs_unregister_pts() never frees the pts file — unprivileged open/close loop exhausts kernel memory and panics the kernel

2 participants