--- name: comment_mining_research description: Mine public comments for research: draw a defensible sample from video and post comments, code themes and repeated questions with a codebook, separate loud voices from common ones by counting unique authors next to comment counts, state sample limits in plain language, and turn findings into content and product decisions. Ships a tested CLI for sampling, first-pass coding, intercoder agreement and saturation checks, a coding sheet and report template, and a worked example where one author drives a whole theme. Use when someone asks what commenters are saying, wants themes or repeated questions from YouTube, TikTok, Reddit, or forum comments, asks for comment analysis, audience research from comments, or a FAQ or roadmap built from comments. For brand mention monitoring use social_listening, for support feedback use a feedback triage skill, for turning commenters into customers use a comment-to-customer skill. --- # Comment Mining Research ## Purpose Public comments are a cheap research corpus and a biased one. The people who comment are a small, self selected slice of the audience, platform sorting decides which comments a reader sees first, and a few repeat voices can make one opinion look like a consensus. This skill runs comment research that survives those problems: a stated sampling method, a written codebook, counts that separate how many comments say something from how many people say it, and a limits section that travels with the findings wherever they get quoted. The output feeds real decisions: which repeated questions become a FAQ or a follow-up video, which confusion becomes a product fix or a clearer tutorial step, which feature requests are broad and which are one person posting six times. ## Workflow ### 1. Frame the question before collecting anything - Write the decision the research has to inform, in one sentence: pick the next three tutorial topics, decide whether a setup step needs redesign, build a FAQ page. - Write the population in one sentence: comments on these 12 videos, on this channel, posted between these dates. If the scope is "comments about our product anywhere", say which platforms and which search terms define it. - Fix the unit of analysis: one top level comment, or one thread. Threads answer "what do people argue about", top level comments answer "what do people ask and report". Do not mix units in one count. - Set exclusions up front: creator and team replies, pinned comments, giveaway entry comments, and obvious copy paste spam usually come out. Record every exclusion in the report. ### 2. Collect through an official route and record the pull - Use the platform API or an export the requester already has. For YouTube, `commentThreads.list` on the YouTube Data API v3 returns up to 100 top level comments per page at 1 quota unit per call, ordered by `time` (the default) or by `relevance`. Record which order was used: relevance is a platform ranking, so a relevance pull is already filtered by likes and visibility before you touch it. - Record the pull in the collection log: platform, source posts or videos, pull date, date range covered, order used, total comments the platform reports, total comments retrieved. If retrieved is much lower than reported, say why if you know it (deleted comments, held comments, comments turned off on some posts) and say "reason unknown" if you do not. - Hash or pseudonymise author identifiers at collection time. The analysis needs to know that six comments came from one author, it does not need a handle, a real name, or a profile link. Store the mapping, if one is needed at all, in a separate file from the working data. - Save the raw export untouched. All cleaning happens on a copy, so any count in the report can be traced back to the raw pull. ### 3. Clean without quietly changing the answer - Remove exact duplicate texts first and report the count. Copy paste spam repeats word for word, genuine agreement almost never does. The bundled tool does this in `analyze` and `sample --dedupe`. - Flag, do not delete, near duplicates and repeat authors. A person posting the same complaint under five videos is a signal about intensity, and a bias in any frequency count. Both facts belong in the report. - Keep a cleaning log: rows in, rows removed per rule, rows out. A theme count that cannot be reconciled with the cleaning log does not ship. ### 4. Sample with a stated method - Small corpora need no sample: code everything up to about 500 comments. Above that, sample, and pick the method by the shape of the corpus: - Stratified random (default): sample inside each source post or video in proportion to its comment count, so one viral post cannot swamp nine normal ones. Run: `python3 scripts/comment_mine.py sample comments.csv --n 300 --strategy stratified --seed 42 --dedupe --out sample.csv` - Random: fine when the corpus is one post or a set of similar posts. - Systematic over time: sort by timestamp and take every kth comment when the question is about change over time, launches, or a controversy with a timeline. - Never sample from "top comments" sorting alone. Top is a ranking by likes and platform rules, not a sample. If stakeholders care about top comments, code them as a separate, labelled stratum and report them apart from the random sample. - Cover time on purpose: early comments under a post over-represent subscribers and fast reactors. A stratified sample across the full life of the post beats the first two pages of any sort order. - Report the seed and the strategy. Another coder with the same export, seed, and strategy must land on the same sample. ### 5. Build the codebook before coding the sample - Start from the decision, not from the data: draft 5 to 12 codes that map to actions (setup_confusion, price_concern, feature_request, bug_report, praise, off_topic). Add an explicit `other` code so coders never force a fit. - Give every code six fields, following standard content analysis practice: code name, short definition, full definition, when to use it, when not to use it, and one example comment (paraphrased). The when-not-to-use field does most of the work in keeping two coders aligned. A filled sheet ships in `assets/coding-sheet.csv`. - Allow multi-label coding for themes (a comment can be a question and a bug report) and pick one primary code per comment for the reliability check in step 7. - Pilot the codebook on 30 to 50 comments outside the main sample. Every code that two readers apply differently gets a sharper boundary or gets merged. Update the codebook version and note what changed. ### 6. Code in batches and track repeated questions separately - Follow the six phases of thematic analysis (Braun and Clarke, 2006) as a loop, not a ladder: familiarise yourself with the data, generate initial codes, group codes into candidate themes, review themes against the full data, define and name themes, then write the report. Expect to move back a phase when a theme does not hold up. - Code in batches of about 50. After each batch, run the saturation check on the coded rows so far: `python3 scripts/comment_mine.py saturation coded.csv --batch 50`. When the last two batches add no new codes, the code list is usually stable enough to finish the sample and move to theme review. If new codes keep appearing, the sample is not done, whatever the calendar says. - Use keyword first-pass coding only as an assist: `python3 scripts/comment_mine.py analyze sample.csv --codebook codebook.csv`. It overcounts (a praise comment mentioning "the alerts setup" picks up setup_confusion). A human or model coder checks every keyword hit before it becomes a finding, and the report says which pass produced the numbers. - Pull questions into their own list as you code. Repeated questions are the highest value output for most requesters: count each distinct question, count its unique askers, and rank by unique askers, not by total repeats. A question asked by 17 different people is a FAQ. A question posted 17 times by one account is a support case. ### 7. Check reliability before trusting the counts - Have a second coder (a second person, or a second model run with the same codebook and no access to the first pass) code the same 10 to 20 percent of the sample, primary code only. - Compare with: `python3 scripts/comment_mine.py agreement coder-a.csv coder-b.csv`. It prints percent agreement and Cohen kappa. - Read the result plainly: 0.80 and above is the common bar for drawing conclusions from coded data. Below it, the codebook is the problem more often than the coders are: sharpen definitions, add when-not-to-use cases from the disagreements, and recode a fresh pilot batch. Do not average the two coders and move on. - Report the agreement numbers in the findings. A theme table with no reliability figure is a draft, not a result. ### 8. Separate loud voices from common ones - Report every theme twice: share of comments and share of unique authors. The `analyze` report prints both, plus comments per author. - Treat comments per author above 2.0 as a flag, not a finding. Read who is repeating the theme before calling it widespread. One determined person is evidence about that person. - Remember why this step exists. Participation follows a steep curve, described by Nielsen in 2006 as the 90-9-1 rule: roughly 90 percent of users in an online community never contribute, 9 percent contribute a little, and 1 percent account for almost all the activity. Commenters are already the vocal slice of viewers, and repeat commenters are the vocal slice of that slice. - Cross-check any "everyone is asking for X" claim against the unique author count before it reaches a roadmap or a headline. If the claim survives on authors but not on comments, it is still real, and it should be stated in author terms. ### 9. Turn findings into decisions - Map each surviving theme to one of four actions, with an owner: answer it (FAQ, pinned reply, follow-up content), fix it (product, docs, setup flow), test it (feature request with broad author support goes to a prototype or a poll), or log it (real but small, revisit next cycle). - Rank actions by unique authors affected, then by how cheap the fix is. A setup confusion hitting 26 percent of commenters outranks a feature request from 6 people, however well written the request is. - Quote sparingly and paraphrase by default (see Operating Rules). One short paraphrased example per theme is enough to make a count feel real. ## Output Contract Deliver a report in the shape of `assets/report-template.md`, containing all of these: - The decision question and the population definition from step 1. - The collection log: sources, dates, pull order, reported vs retrieved totals, exclusions. - The cleaning log and the sampling method, size, and seed. - The theme table with, per theme: comments, percent of comments, unique authors, percent of authors, comments per author, and one paraphrased example. - The repeated questions list, ranked by unique askers, with the planned answer route for the top questions. - The reliability result: shared comments coded, percent agreement, Cohen kappa or Krippendorff alpha. - The limits section, written in full sentences a non-researcher can follow (see `references/limits-and-ethics.md` for the required points). - The decision list: theme, action, owner. ## Operating Rules - Every factual claim in a skill or report must be verified via web search or a primary source. Never invent numbers, dates, quotes, or events. Unverifiable means cut it or frame it as opinion. - Never present comment findings as audience opinion. Comments describe commenters. Write "26 percent of commenters in this sample" and never "26 percent of viewers think". - Paraphrase quoted comments by default and strip handles. A verbatim public comment can be pasted into a search engine and traced to its author in seconds. Verbatim quotes need a reason (the exact wording is the finding) and, for sensitive topics, consent. Publicly posted does not mean posted for researchers: internet research ethics guidance (AoIR guidelines 3.0, franzke et al. 2020) treats public access and consent as different things. - Stay inside platform terms and official access routes. No scraping setup that breaks a platform's terms, no buying comment dumps, no logging in as someone else to reach comments. - Do not profile individual commenters. No lists of named critics, no cross-platform identity matching, no demographic guesses about authors. The API does not provide commenter demographics, and anything inferred is a guess dressed as data. - Report bot and coordination suspicion honestly: automated activity cannot be identified with certainty from a comment export. Say "some comments showed copy paste patterns consistent with spam, N removed" rather than claiming a bot count. - Small samples get small claims. Under about 100 coded comments, report themes as a list with examples, not as percentages with decimal places. - If the sample, the codebook, or the reliability check is missing, label the output exploratory. Exploratory work can guide a hunch. It cannot headline a decision memo. ## Worked Example Synthetic data, built to show the loud voice trap. `assets/example-comments.csv` holds 49 comments under two tutorial videos for a fictional budgeting app. `assets/example-codebook.csv` holds five codes. Running: ``` python3 scripts/comment_mine.py analyze assets/example-comments.csv --codebook assets/example-codebook.csv ``` gives 49 comments in the export, 2 exact duplicates removed (a copy paste spam line posted three times), 47 comments analysed from 42 unique authors. The theme table: - price_concern: 6 comments, 12.8 percent of comments, 1 unique author, 2.4 percent of authors, 6.00 comments per author. - setup_confusion: 11 comments, 23.4 percent of comments, 11 unique authors, 26.2 percent of authors, 1.00 comments per author. - feature_request: 6 comments, 6 unique authors. - praise: 8 comments, 8 unique authors. - bug_report: 4 comments, 4 unique authors. - 17 comments (36.2 percent) are questions, from 17 unique authors. Read naively, price looks like a top theme. Read by authors, price is one unhappy subscriber posting under both videos, and setup confusion is the real headline: more than a quarter of commenters hit it, once each. The decision list writes itself: fix or re-film the setup step first, answer the repeated questions in a FAQ, log the price complaint as one case, not a pricing signal. Two coders coding the first 20 comments with one primary code each (`assets/example-coder-a.csv`, `assets/example-coder-b.csv`) agree on 90.0 percent with Cohen kappa 0.843, above the 0.80 bar, so the counts are usable. The setup count of 11 also carries a lesson: one praise comment about "the alerts setup" matched the setup keywords, which is why keyword coding is a first pass, never the published number.