corpus_linguistics v1.0.0

name: corpus_linguistics description: Do corpus linguistics on a real text collection: build and clean a corpus with documented sources and permissions, read frequency lists against a reference corpus, compute collocation statistics (mutual information, t-score and log-likelihood) and read them against each other, read KWIC concordances by hand, and write up findings that separate what the corpus shows from what it cannot. Triggers on corpus building, frequency lists, keyword or keyness analysis, collocation analysis, concordance or KWIC work, comparing a text collection against the BNC or another reference corpus, and questions about what a body of texts can and cannot prove.

Corpus Linguistics

Purpose

Study language from a collection of real texts, with the receipts attached. A corpus study stands or falls on three things: the corpus was built to a stated design from documented sources, the statistics were read for what they actually measure, and the write-up claims only what this collection can support.

The workflow runs from raw text to findings: design and document the corpus, clean it in small logged steps, count frequencies with dispersion, compare against a reference corpus for keywords, measure collocations with several statistics at once, read concordances by eye, then report with the limits in the same document as the results.

Bundled tool: bin/corpus.py, standard library only. It inventories a corpus with file hashes, cleans markup and punctuation variants with a change log, builds frequency lists with per-million rates and document range, computes log-likelihood keywords with log ratio effect sizes, computes MI, t-score and G2 for the collocates of one node word, and prints KWIC concordance lines sorted by left or right context. A complete worked study on bundled demo data, with every number reproducible, is in references/worked-study.md. The formulas and their reading are in references/statistics.md. Design, permissions and cleaning rules are in references/corpus-design-and-cleaning.md.

Workflow

1. State the question and the population before collecting anything. One sentence for the question, one for the population the corpus must stand for, with date and genre bounds. Biber (1993) defines representativeness as the extent to which a sample includes the full range of variability in a population, and that range cannot be checked against a population nobody defined. If the user wants a claim about a variety ("how support agents write", "what this journal sounds like"), the population sentence is what the claim will be tested against at the end.

1. Design the corpus and open the manifest. Copy assets/corpus-manifest-template.md and fill in the sampling frame, the corpus type (balanced, monitor, opportunistic or specialist), the balance plan and the selection criteria before the first text arrives. Select texts on external criteria such as date, genre, author and venue, never on the linguistic features under study. Record every text in the manifest with its source, date, genre, how it was obtained and the basis for using it: public domain, a licence that covers the use, permission, or the user's own text. A text with an unclear basis stays out.

1. Inventory what you actually have. Run python3 bin/corpus.py inventory <dir>. Read the output for duplicate hashes (the same text arriving twice is one observation, not two), replacement characters (encoding damage already in the source), and size spread across files. One very long text can steer every count in a small corpus. Report tokens and text count at this point, because every later claim inherits them.

1. Clean on a copy, in small logged steps. Run python3 bin/corpus.py clean <in_dir> <out_dir> on a copy and keep the raw files untouched. The cleaner strips markup, decodes entities, normalises punctuation variants that would split one word into two types, and logs counts per file. Never silently modernise spelling, expand contractions, fix dialect forms or drop messy passages. Each of those is a linguistic decision. If the study needs one, run it as a separate named step and state it in the report.

1. Fix the counting choices and write them down. State the tokenisation, case handling, lemma handling and any tagging before counting, following references/corpus-design-and-cleaning.md. The bundled tool lowercases and counts letter sequences with internal apostrophes as one token, drops numbers, and splits hyphenated forms. Word-form counts and lemma counts answer different questions. A reader who cannot tell which one a table shows cannot use the table.

1. Build the frequency list and read it with dispersion. Run python3 bin/corpus.py freq <dir> --min <n> --out freq.csv. Read each frequent word with three numbers: raw frequency, rate per million words, and range (in how many documents, out of how many). A word with a high count in one document is a fact about that document. A word with a moderate count in nearly every document is a fact about the corpus. Brezina (2018) is blunt on this point: frequency and dispersion together describe a word, neither one alone does.

1. Get keywords against a reference corpus. Run python3 bin/corpus.py keywords --target <dir> --reference <dir> --min <n> --out keywords.csv. The reference is half the method: a large general corpus such as the British National Corpus (100 million words of written and spoken British English, built 1991 to 1994) shows how the target differs from general language, while a neighbouring specialist corpus shows how it differs from its peers. Read the output in references/statistics.md terms: log-likelihood says how unlikely the difference is under chance (3.84 for p below 0.05, 6.63 for p below 0.01, 10.83 for p below 0.001), and log ratio says how big the difference is. Keep overuse and underuse apart, and sort the list into Scott's three kinds: proper nouns, aboutness words, and high-frequency style words. A keyword list read without that sort mixes topic and style into one misleading ranking.

1. Measure collocations with three statistics, never one. Pick the node word from the frequency or keyword results, then run python3 bin/corpus.py collocates <dir> <node> --window 5 --min-cooc <n> --out collocates.csv. Read MI as strength of association, t-score as strength backed by frequency, and G2 as confidence that the pairing is not chance. The three will disagree, and the disagreement is information: high MI with low pair frequency is a lead, high G2 with high pair frequency is a pattern. Set a minimum co-occurrence count before sorting by MI, because MI inflates for rare pairs. State the window in the report, because every score moves when the window moves, and scores from different tools or different windows do not share a scale.

1. Read the concordance by hand. Run python3 bin/corpus.py kwic <dir> <node> --sort right, then again with --sort left. Sorting lines up the patterns on each side of the node so repeated frames become visible. Read for what counts cannot mark: negated uses ("nobody talked about powerful tea"), mentioned rather than used forms, irony, names, and one-off metaphors. Pull the exact lines that support each finding into the report. If a pattern from the tables dissolves when its lines are read, the tables lose and the report says so.

1. Cross-check one number by hand. Pick the most load-bearing statistic in the study (usually the top keyword or the headline collocation) and recompute it from the raw counts with the formulas in references/statistics.md. The bundled functions reproduce a published check case exactly (the BNC pairing alphabetical order: expected 0.078, MI 10.12, t 9.32, G2 1084.84), so a mismatch on your own data points at the counts, not the arithmetic. If the hand check fails, fix the counts before writing anything.

1. Write the report with the limits attached. Use assets/study-report-template.md. Every finding carries its numbers, its dispersion and quoted concordance lines. A separate section names what this corpus cannot show: population limits, coverage gaps, features too rare here to judge, and findings that rest on one text. The demo study in references/worked-study.md shows the tone: its corpus is 622 tokens of invented text, so its findings are labelled as demonstrations of method, not discoveries about real language.

Output Contract

Deliver all of the following:

• Manifest: the filled corpus manifest, one row per text with source, date, genre and basis for use, plus totals (texts, tokens, smallest and largest text) and the duplicates removed.
• Counting statement: tokenisation, case, lemma and tagging choices in plain lines a reader can repeat.
• Cleaning log: each class of change with its per-file or total count, and the replacement character count from the sources.
• Frequency table: word, raw frequency, rate per million words, document count and range share, with the minimum frequency stated.
• Keyword table (when a reference corpus is used): word, frequencies in both corpora, rates per million in both, log-likelihood, direction (overuse or underuse) and log ratio, with zero-corrected rows flagged and the reference corpus named with its size.
• Collocation table (per node studied): collocate, pair frequency, node tokens with the collocate, collocate frequency, expected co-occurrences, MI, t-score and G2, with window and minimum co-occurrence stated.
• Concordance evidence: the sorted KWIC lines behind each reported pattern, quoted exactly, with file names.
• Report: findings in the template shape, each with numbers, dispersion and lines, followed by the section on what the corpus cannot show, and the exact commands needed to rerun every number.

Operating Rules

1. No manifest, no analysis. A corpus without documented sources and a stated basis for using each text does not get counted. Reachable is not the same as usable.
2. Raw files never change. Cleaning and every later transformation work on copies, and each step keeps its own output so any count can be traced back to the raw text.
3. Frequency without dispersion is half a result. Any claim built on a count states how many texts the count comes from. A pattern living in one text is reported as a pattern in that text.
4. Significance is not size. A log-likelihood value says how surprising a difference is, not how much it matters. Every keyword and collocation claim pairs its test value with an effect size or a raw frequency a reader can judge.
5. MI never travels alone. A collocate list sorted by MI without a minimum co-occurrence count and the pair frequencies printed beside the scores will mislead, because MI rewards rare pairs. Print the frequencies.
6. One window, one tool, one scale. Compare collocation scores only inside a single run with fixed settings. Scores from AntConc, LancsBox, Sketch Engine or this script differ by counting convention and are never placed side by side as if they matched.
7. The concordance outranks the table. When reading the lines contradicts a statistic, the lines win, and the report explains the contradiction (negation, mention rather than use, one prolific author) instead of burying it.
8. Small corpora get small claims. If the feature under study occurs a handful of times, say the count in the same sentence as the conclusion. A specialist corpus can be small and still sound, because its population is narrow. A general claim from a small opportunistic corpus is not sound at any size of enthusiasm.
9. Name the corpus type honestly. Most project corpora are opportunistic. The label is not a confession, it is the boundary of the claims, and dressing an opportunistic collection up as balanced is the one move that turns a decent study into a false one.
10. Do not import outside numbers. Frequencies, corpus sizes and statistics quoted in the report come from this run's counts or from a named primary source the run actually checked. A remembered figure about a famous corpus stays out until it is verified.