Repository navigation
Query: Select random redirect posts by looking up random post IDs. - #16050
aaronjorbin wants to merge 1 commit into
Conversation
Random content requests, such as `/random/`, currently select the post with `ORDER BY RAND() LIMIT 1`. The database reads and sorts every eligible post for every request, so the cost grows with the size of the site. As the redirect is not cached, extremely large sites can not safely enable the feature. The post is now selected by rejection sampling: * 50 random post IDs between 1 and the highest post ID are looked up by the primary key, limited to the posts eligible for the requested archive, and one of the eligible posts found is selected at random. * If none of the IDs are eligible posts, other random IDs are looked up, up to three times in total. * If no post is found, eligible posts are too sparse among the post IDs, for example a small category on a large site. A pool of up to 1,000 randomly ordered eligible posts is selected instead and stored in the object cache. It is used for 10 minutes. Why this is better than the current approach: * The cost of a request no longer depends on the number of posts. In local testing, a request read 53 rows at both 100,000 and 10,000,000 posts, where `ORDER BY RAND()` read every eligible post. The median time to select a post at 10,000,000 posts fell from around 9 seconds to around 5 milliseconds. * It does not rely on a persistent object cache. Only the fallback for sparse archives is cached. * Every eligible post remains equally likely to be selected, whether or not there are gaps between post IDs. Every ID is equally likely to be looked up, so gaps only reduce how often a lookup finds a post. This was confirmed with 100,000 selections from posts with large gaps between their IDs. * The database is only read from. The object cache is used directly, without transients, so sites using a read-only database user for front end requests are unaffected. The fallback for sparse archives is the only part that still reads every eligible post of an archive, so its cost is limited: * The pool is selected without excluding password protected posts, which are skipped when the post is selected from it. The post password is not indexed, so this allows the pool to be selected from an index rather than reading each post in full. * Only one request selects the pool at a time. Once the pool is 10 minutes old, the first request to need it selects the next pool, and other requests select from the previous pool until it has. If there is no previous pool, concurrent requests each select one. On sites with very few posts, the request makes one more query than before. Trac ticket: https://core.trac.wordpress.org/ticket/66654 AI disclosure: This change, its tests and the benchmarks behind the figures above were produced with AI assistance (Claude Code, Claude Fable 5.2), directed and reviewed by Aaron Jorbin. See #66654. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
|
The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. Core Committers: Use this line as a base for the props when committing in SVN: To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook. Props Bot commandsYou can trigger Props Bot actions by commenting on this PR:
|
Test using WordPress PlaygroundThe changes in this pull request can previewed and tested using a WordPress Playground instance. WordPress Playground is an experimental project that creates a full WordPress instance entirely within the browser. Some things to be aware of
For more details about these limitations and more, check out the Limitations page in the WordPress Playground documentation. |
Change Random content requests from using `ORDER BY RAND() LIMIT 1` to using a sampling with rejection method. The post is now selected by looking up random post IDs by primary key, which keeps the cost of a request independent of the number of posts while leaving every eligible post equally likely to be selected. Archives whose posts are too sparse among the post IDs to be found this way fall back to a pool of randomly ordered posts stored in the object cache. Developed in: #16050 Follow-up to r64402, r65042. Props jorbin, peterwilsoncc, sukhendu2002, rmccue, gddrt, renehermi, djpaul. Fixes #66654. See #64498. git-svn-id: https://develop.svn.wordpress.org/trunk@65542 602fd350-edb4-49c9-b593-d223f7449a82
Change Random content requests from using `ORDER BY RAND() LIMIT 1` to using a sampling with rejection method. The post is now selected by looking up random post IDs by primary key, which keeps the cost of a request independent of the number of posts while leaving every eligible post equally likely to be selected. Archives whose posts are too sparse among the post IDs to be found this way fall back to a pool of randomly ordered posts stored in the object cache. Developed in: WordPress/wordpress-develop#16050 Follow-up to r64402, r65042. Props jorbin, peterwilsoncc, sukhendu2002, rmccue, gddrt, renehermi, djpaul. Fixes #66654. See #64498. Built from https://develop.svn.wordpress.org/trunk@65542 git-svn-id: http://core.svn.wordpress.org/trunk@64615 1a063a9b-81f0-0310-95a4-ce76da25c4cd
Random content requests currently select the post with
ORDER BY RAND() LIMIT 1. The database reads and sorts every eligible post for every request, so the cost grows with the size of the site.The post is now selected by rejection sampling:
Why this is better than the current approach:
ORDER BY RAND()read every eligible post. The median time to select a post at 10,000,000 posts fell from around 9 seconds to around 5 milliseconds.The fallback for sparse archives is the only part that still reads every eligible post of an archive, so its cost is limited:
Trac ticket: https://core.trac.wordpress.org/ticket/66654
AI assistance: Yes
Tool(s): Claude Code
Model(s): Fable 5.2
Used for: This change, its tests and the benchmarks behind the figures above were produced with AI assistance, directed and reviewed by me.
This Pull Request is for code review only. Please keep all other discussion in the Trac ticket. Do not merge this Pull Request. See GitHub Pull Requests for Code Review in the Core Handbook for more details.