Startup resources: 92 to 653, from the full source scrape - #7
Merged
Conversation
The university collection arrived complete rather than in the fragments the Internet Archive had, so the index goes from 92 resources to 653 across nine pages. 561 are new, 37 appear in both source collections, 55 come only from the climate list. Within each page, sub-headings are the topic the resource was filed under in its source, so a 132-entry page stays navigable rather than becoming a wall. Classification is now topic-level rather than per-resource. The source files 41 topics under 7 sections; mapping those 41 topics onto the ten groups here is 41 reviewable decisions instead of 653 individual judgements, and it does not drift between one resource and the next. The mapping is in the generator. Two things caught in the arithmetic rather than by eye: The URL de-duplication keyed on host plus path and dropped the query string, which collapsed all 206 YouTube links onto a single key -- 205 resources would have vanished silently. The key now keeps identity-bearing query parameters, strips only decoration (timestamps, referrers, campaign tags), and canonicalises youtu.be/ID and watch?v=ID onto one key so the same video does not appear twice. Link health was checked by DNS resolution rather than HTTP status, because the HTTP status through this environment's proxy cannot distinguish a dead domain from a refused CONNECT -- the mistake that nearly put a wrong finding in the last commit message. Of 106 domains, 105 resolve. One does not, costing three resources. The difference between the two collections is worth recording: 68 dead links out of 179 in the older one, 3 out of 609 in the newer. That is what a maintained list looks like against an abandoned one.
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (12)
📝 WalkthroughWalkthroughThe resource catalog expands from six to nine pages, increases indexed content counts, reorganizes existing resource pages, adds strategy, growth, and legal pages, and updates generated navigation entries and README statistics. ChangesStartup resource catalog
Estimated code review effort: 3 (Moderate) | ~20 minutes Possibly related PRs
Suggested reviewers: ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The university collection arrived complete rather than in the fragments the Internet Archive had, so the index goes from 92 resources to 653 across nine pages. 561 are new, 37 appear in both source collections, 55 come only from the climate list.
Within each page, sub-headings are the topic the resource was filed under in its source, so a 132-entry page stays navigable rather than becoming a wall.
Classification is now topic-level rather than per-resource. The source files 41 topics under 7 sections; mapping those 41 topics onto the ten groups here is 41 reviewable decisions instead of 653 individual judgements, and it does not drift between one resource and the next. The mapping is in the generator.
Two things caught in the arithmetic rather than by eye:
The URL de-duplication keyed on host plus path and dropped the query string, which collapsed all 206 YouTube links onto a single key -- 205 resources would have vanished silently. The key now keeps identity-bearing query parameters, strips only decoration (timestamps, referrers, campaign tags), and canonicalises youtu.be/ID and watch?v=ID onto one key so the same video does not appear twice.
Link health was checked by DNS resolution rather than HTTP status, because the HTTP status through this environment's proxy cannot distinguish a dead domain from a refused CONNECT -- the mistake that nearly put a wrong finding in the last commit message. Of 106 domains, 105 resolve. One does not, costing three resources.
The difference between the two collections is worth recording: 68 dead links out of 179 in the older one, 3 out of 609 in the newer. That is what a maintained list looks like against an abandoned one.
Summary by CodeRabbit
New Features
Documentation