The third book in ABC Books is McGuffey's Second Eclectic Reader, Revised Edition (1879), all 71 lessons. It's the green book on the shelf at https://games.503mail.com/abcbooks/. Every lesson plays like the others: new words, listen, read aloud, trace in print and cursive, and the blackboard. This post covers what's special about this book and how to add it. The app itself is rebuilt in the Primer post. Every block has a Copy button.
The book
- McGuffey's Second Eclectic Reader, Revised Edition, copyright 1879, from Project Gutenberg #14668 (public domain).
- 71 lessons of real stories and poems, each with a title: "Evening at Home", "Bubbles", Willie's letter to Santa Claus, "Twinkle, Twinkle, Little Star", the bees and their hive, the tiger, the kingbird, "The Greedy Girl", Ralph Wick and the rose bush, "Alice's Supper", "Sweet and Low" and many more.
- The lessons are much longer than in the first two books (often 15 to 30 lines), and many new words have two or three syllables.
- Several lessons are letters or poems written in script; those become the cursive tracing pages.
- About 5,950 different things to say per speed, recorded in ETI-Eloquence: about 74 MB at "normal" speed and 61 MB at "faster".
Growing up through the books
The books follow the kids as they get older. On the shelf, each book shows the kid's own progress ("Lesson 3 of 63", or "★ Finished!"), and the book they're on now glows. Every book stays open, so an older child can start at the Second Reader without doing the Primer first. Stars are kept per kid and per book.
Because these lessons are long, the copy work is kept to a sensible size in every book: the letter sounds, the new words six to a page, up to six short sentences (70 characters or less), and up to six lines of the book's own script exercise. The blackboard asks for up to three short sentences.
What the converter had to clean up
On top of what the First Reader post lists, make-book.ps1 now handles:
- Numbered paragraphs and stanzas: "1. It is winter." reads as "It is winter.", and a verse number on its own line is dropped. One paragraph was scanned as "I." instead of "1."; that's handled too.
- Titles in normal letters: a short line like "The Little Star" right before the text becomes the lesson's title, like the capital-letter titles.
- Poems that aren't indented: a paragraph whose lines are all short and start with a capital is a poem, so each line stays a line. Wrapped prose goes on in lowercase, so it isn't mistaken for a poem. Up to one odd line is allowed in a long poem, because Gutenberg's text has a few line-break slips.
- "Mr." and "Mrs." don't end a sentence, and they don't stop a word list from being recognised as one.
- Old words that keep their apostrophe: o'clock, o'er, ne'er.
- Footnotes ("[Footnote: harebell: ...]") are left out of the reading.
- Two small slips are in Gutenberg's text itself and are kept as they are: Lesson 57's word list repeats Lesson 56's, and there are a couple of typos ("nealy", "seytle").
Add this book to ABC Books
You need the app from the Primer post and its C:\ABCBooks folder on the PC (with ETI-Eloquence installed). The app already lists this book on its shelf; it needs the lesson file and the recordings.
1. The converter and the upload script
If your copies in C:\ABCBooks\tools\ are older than this post, replace them. Proxmox host shell, then copy them to the PC the same way as in the Primer post (the scp and arrange-rebuild.ps1 steps):
mkdir -p /root/abcbooks-rebuildmake-book.ps1: turns a Gutenberg McGuffey Reader into the app's lesson file.
cat > /root/abcbooks-rebuild/make-book.ps1 <<'EOF'
# Builds public/books/<book>.json from a Project Gutenberg text of a McGuffey Reader.
# Usage: powershell -ExecutionPolicy Bypass -File ABCBooks\tools\make-book.ps1 -Book primer1881
# powershell -ExecutionPolicy Bypass -File ABCBooks\tools\make-book.ps1 -Book first1879
param([ValidateSet('primer1881', 'first1879', 'second1879', 'third1879', 'fourth1879', 'fifth1879', 'sixth1879', 'spell1867')] [string]$Book = 'primer1881')
$root = Split-Path -Parent (Split-Path -Parent $MyInvocation.MyCommand.Path)
$books = @{
# Spell & Read: the 1867 Pictorial Primer, hand-corrected from a scan into source/spell1867.txt (no Gutenberg text exists)
spell1867 = @{ hand = $true; file = 'spell1867.txt'; title = 'Spell & Read'; sub = 'Pictorial Primer, 1867'; color = '#5b6f3a'
source = 'McGuffey''s Pictorial Eclectic Primer, 1867 (archive.org mcguffeyspictori00mcgu, public domain), hand-corrected' }
primer1881 = @{ gutenberg = 14642; file = 'primer.txt'; title = "McGuffey's Eclectic Primer"; sub = 'Revised Edition, 1881'; color = '#b5562c'
end = 'SLATE EXERCISES'; pics = @{ 27 = '🏞️'; 46 = '✏️' } }
first1879 = @{ gutenberg = 14640; file = 'first.txt'; title = "McGuffey's First Eclectic Reader"; sub = 'Revised Edition, 1879'; color = '#2c6fb5'
end = 'PHONIC CHART.'; pics = @{} }
second1879 = @{ gutenberg = 14668; file = 'second.txt'; title = "McGuffey's Second Eclectic Reader"; sub = 'Revised Edition, 1879'; color = '#3f8f4f'
end = '*** END OF THE PROJECT GUTENBERG'; pics = @{} }
# prose = later Readers: no word lists in Gutenberg's copy (new words are worked out), printed page headers to remove,
# numbered paragraphs that run on without blank lines, plays ("Rose. ...") and an author line under poems
third1879 = @{ gutenberg = 14766; file = 'third.txt'; title = "McGuffey's Third Eclectic Reader"; sub = 'Revised Edition, 1879'; color = '#7b4fb5'
end = '*** END OF THE PROJECT GUTENBERG'; pics = @{}; prose = $true; pageRx = 'ECLECTIC SERIES|THIRD READER'
after = @('primer1881', 'first1879', 'second1879') }
# Fourth Reader: headings carry the title ("I. PERSEVERANCE. (25)"), and each lesson ends with DEFINITIONS (the book's
# own new words, with meanings), EXERCISES (questions to talk about) and sometimes NOTES
fourth1879 = @{ gutenberg = 14880; file = 'fourth.txt'; title = "McGuffey's Fourth Eclectic Reader"; sub = 'Revised Edition, 1879'; color = '#b5892c'
end = '*** END OF THE PROJECT GUTENBERG'; pics = @{}; prose = $true; pageRx = 'ECLECTIC SERIES|FOURTH READER'
start = 'I. PERSEVERANCE. (25)'; tocStart = 'SELECTIONS IN PROSE AND POETRY.'; tocEnd = 'LIST OF ILLUSTRATIONS'
headRx = '(?m)^([IVXLCl]+)[.:][ \t]+([A-Z][^a-z\n]*?(?:\((?:Concluded|Continued)\.?\))?[^a-z\n]*?)[ \t]*(?:\(\d+\))?[ \t]*$'
after = @('primer1881', 'first1879', 'second1879', 'third1879') }
# Fifth Reader: like the Fourth, but headings have no page number ("II. THE BLUEBELL.") and the first one is scanned as "1."
fifth1879 = @{ gutenberg = 15040; file = 'fifth.txt'; title = "McGuffey's Fifth Eclectic Reader"; sub = 'Revised Edition, 1879'; color = '#2c9bb5'
end = '*** END OF THE PROJECT GUTENBERG'; pics = @{}; prose = $true; pageRx = 'ECLECTIC SERIES|FIFTH READER'
start = '1. THE GOOD READER.'
headRx = '(?m)^(\d+|[IVXLCl]+)[.:][ \t]+(["\u201c]?[A-Z][^a-z\n]*?(?:\((?:Concluded|Continued)\.?\))?[^a-z\n]*?)[ \t]*(?:\(\d+\))?[ \t]*$'
tocEnd = 'LIST OF ILLUSTRATIONS'
after = @('primer1881', 'first1879', 'second1879', 'third1879', 'fourth1879') }
# Sixth Reader: headings with page numbers like the Fourth; the lessons start after the chapters on reading aloud
sixth1879 = @{ gutenberg = 16751; file = 'sixth.txt'; title = "McGuffey's Sixth Eclectic Reader"; sub = 'Revised Edition, 1879'; color = '#8f3f5f'
end = '*** END OF THE PROJECT GUTENBERG'; pics = @{}; prose = $true; pageRx = 'ECLECTIC SERIES|SIXTH READER'
start = 'I. ANECDOTE OF THE DUKE OF NEWCASTLE.'
headRx = '(?m)^(\d+|[IVXLCl]+)[.:][ \t]+(["\u201c]?[A-Z][^a-z\n]*?(?:\((?:Concluded|Continued)\.?\))?[^a-z\n]*?)[ \t]*(?:\(\d+\))?[ \t]*$'
tocEnd = 'LIST OF ILLUSTRATIONS'
after = @('primer1881', 'first1879', 'second1879', 'third1879', 'fourth1879', 'fifth1879') }
}
$cfg = $books[$Book]
$pics = [ordered]@{ 'rat'='🐀'; 'cat'='🐈'; 'kitten'='🐱'; 'dog'='🐕'; 'frog'='🐸'; 'chick'='🐤'; 'hen'='🐔'; 'fox'='🦊'; 'duck'='🦆'; 'owl'='🦉'; 'bird'='🐦'; 'cow'='🐄';
'rabbit'='🐇'; 'pig'='🐖'; 'lamb'='🐑'; 'kite'='🪁'; 'sled'='🛷'; 'top'='🎩'; 'ball'='⚽'; 'squirrel'='🐿️'; 'bee'='🐝'; 'boat'='⛵';
'ship'='⛵'; 'skat'='⛸️'; 'horse'='🐎'; 'train'='🚂'; 'goat'='🐐'; 'burning'='🔥'; 'school'='🏫'; 'quail'='🐦'; 'baby'='👶'; 'lighthouse'='🗼';
'sheep'='🐑'; 'snow'='☃️'; 'fishing'='🎣'; 'doll'='🪆'; 'sawing'='🪚'; 'lamp'='🪔'; 'seashore'='🏖️'; 'blind'='🦯'; 'mill'='🏞️';
'sunset'='🌅'; 'woman'='👩'; 'wagon'='🐎'; 'acorn'='🌰'; 'drum'='🥁'; 'lake'='🌊'; 'writing'='✏️'; 'man'='👴'; 'girl'='👧'; 'boy'='👦'; 'children'='👧' }
if ($cfg.hand) {
# Hand-corrected book (source/<file>): "= n" starts a lesson, "words:" is its spelling list, "?" lines are notes, "#" comments
$lessons = @(); $o = $null
foreach ($ln in [IO.File]::ReadAllLines("$root\source\$($cfg.file)")) {
$l = $ln.Trim()
if (-not $l -or $l.StartsWith('#')) { continue }
if ($l -match '^=\s*(\d+)') { if ($o) { $lessons += $o }; $o = [ordered]@{ n = [int]$Matches[1]; pic = '📖'; picText = ''; words = @(); letters = @(); lines = @() }; continue }
if ($l -match '^words:\s*(.*)$') { $o.words = @($Matches[1] -split '\s+' | Where-Object { $_ }); continue }
if ($l.StartsWith('?')) { $o.note = $l.TrimStart('?', ' '); continue }
$o.lines += $l
}
if ($o) { $lessons += $o }
foreach ($o in $lessons) {
if (-not $o.words.Count) { # no list printed: spell the lesson's own words
$w = @(); foreach ($m in [regex]::Matches(($o.lines -join ' '), "[A-Za-z]+(?:'[A-Za-z]+)?")) { $x = if ($m.Value -ceq 'I') { 'I' } else { $m.Value.ToLower() }; if ($w -notcontains $x) { $w += $x } }
$o.words = @($w | Select-Object -First 12)
}
$txt0 = $o.lines -join ' '
foreach ($key in $pics.Keys) { if ($txt0 -match "(?i)\b$key(?:s|es)?\b") { $o.pic = $pics[$key]; break } }
}
$out = [ordered]@{ title = $cfg.title; sub = $cfg.sub; color = $cfg.color; spell = $true; source = $cfg.source; lessons = $lessons }
[IO.File]::WriteAllText("$root\public\books\$Book.json", ($out | ConvertTo-Json -Depth 6), (New-Object Text.UTF8Encoding $false))
"$($lessons.Count) lessons"
$lessons | ForEach-Object { "{0,2} {1} w[{2}] lines={3} {4}" -f $_.n, $_.pic, ($_.words -join ' '), $_.lines.Count, $(if ($_.note) { '? ' + $_.note } else { '' }) }
return
}
if (-not (Test-Path "$root\source\$($cfg.file)")) {
New-Item -ItemType Directory -Force "$root\source" | Out-Null
Invoke-WebRequest "https://www.gutenberg.org/cache/epub/$($cfg.gutenberg)/pg$($cfg.gutenberg).txt" -OutFile "$root\source\$($cfg.file)" -UseBasicParsing
}
$txt = [IO.File]::ReadAllText("$root\source\$($cfg.file)") -replace "`r", ''
$headRx = '(?m)^(?:Lesson 1|LESSON [IVXLC]+\.{0,2}(?:\s*-*\s*REVIEW\.?)?)\s*$' # Third Reader has one "LESSON XXX.."
if ($cfg.headRx) { $headRx = $cfg.headRx }
$body = if ($cfg.start) { $txt.Substring($txt.LastIndexOf($cfg.start)) } else { $txt.Substring([regex]::Match($txt, $headRx).Index) }
$body = $body.Substring(0, $body.IndexOf($cfg.end))
$body = $body -replace '(?m)^SLATE WORK\.?\s*$', ''
if ($cfg.prose) {
# printed page headers ("18 ECLECTIC SERIES.") and page numbers, with the blank lines around them, so sentences join across pages
$body = [regex]::Replace($body, "\n[ \t]*\n?[ \t]*\d*[ \t]*(?:$($cfg.pageRx))\.?[ \t]*\d*[ \t]*\n(?:[ \t]*\n)*", "`n")
$body = $body -replace '(?m)^[ \t]*\(\d+\)[ \t]*\n', '' -replace '(?m)^[ \t]*\d+[,.]?(?:[ \t]*\d+[,.]?)*[ \t]*\n', ''
$body = $body -creplace "(?m)^[ \t]*\d*[ \t]*(?:$($cfg.pageRx))[.,]?[ \t]*\d*[ \t]*\n", '' # leftover headers ("62 ECLECTIC SERIES,")
$body = $body -creplace "(\w) n't\b", '$1n''t' -creplace "'T is\b", "'Tis" # old spelling: "do n't" -> "don't"
}
$heads = [regex]::Matches($body, $headRx)
# the text of each lesson runs from the end of its heading to the next heading (Regex.Split would also return captured groups)
$parts = @($body.Substring(0, $heads[0].Index))
for ($i = 0; $i -lt $heads.Count; $i++) { $s = $heads[$i].Index + $heads[$i].Length; $e = if ($i + 1 -lt $heads.Count) { $heads[$i + 1].Index } else { $body.Length }; $parts += $body.Substring($s, $e - $s) }
$sounds = 'th','sh','ch','ng','ew','oy','ow','wh','ou','oo','sk','qu','oi','gh','ck'
$romanVal = @{ I=1; V=5; X=10; L=50; C=100 }
# scanned "LVll." = LVII, so a lowercase l counts as I
function Roman($r) { if ($r -match '^\d+$') { return [int]$r }; $r = $r -creplace 'l', 'I'; $t = 0; for ($i = 0; $i -lt $r.Length; $i++) { $v = $romanVal["$($r[$i])"]; if ($i + 1 -lt $r.Length -and $romanVal["$($r[$i+1])"] -gt $v) { $t -= $v } else { $t += $v } }; $t }
function Sentences($para) {
# split a paragraph into kid-sized sentences; keep quotes and "--" asides with their sentence
$p = (($para -replace '\s+', ' ').Trim()) -creplace '^(?:\d+|I)\.\s+(?=["A-Z])', ''
$out = [regex]::Split($p, '(?<!\b(?:Mr|Mrs|Dr|St)\.)(?<=[.!?]"?)\s+(?=["A-Z])') | Where-Object { $_ }
# re-join pieces that split inside a quotation
$merged = @(); $buf = ''
foreach ($s in $out) { $buf = if ($buf) { "$buf $s" } else { $s }; if ((($buf -split '"').Count - 1) % 2 -eq 0) { $merged += $buf; $buf = '' } }
if ($buf) { $merged += $buf }
$merged | ForEach-Object { $s = $_.Trim(); if ($s -match '[A-Za-z]$') { $s += '.' }; $s }
}
# vocabulary words carry accent marks and syllable breaks in the later lessons: cun'ning, be cause', did n't
function IsPoem($para) {
$ls = @($para.Trim() -split '\n' | Where-Object { $_.Trim() })
# poem lines are short and each starts with a capital; wrapped prose goes on in lowercase
if ($para -match '\n\s+\S') { return $true }
if ($ls.Count -lt 2 -or ($ls | Measure-Object -Property Length -Maximum).Maximum -gt 55) { return $false }
$low = @($ls | Where-Object { $_.Trim() -cnotmatch '^["''\u2018\u201c(]{0,2}[A-Z]' }).Count
($low -eq 0) -or ($ls.Count -ge 4 -and $low -le 1)
}
function CleanWord($t) {
$t = $t -replace '\s+', ''
if ($t -match "(n't|'s|'ll|'ve|'re|'d|'m)$" -or $t -match "^'tis$") { return ($t -replace "'(?=.*')", '') } # keep a real contraction's apostrophe
$keep = $t.TrimEnd("'")
if ($keep -match "^(o'clock|o'er|ne'er|e'er|e'en)$") { return $keep } # old poetic words keep their apostrophe
$t -replace "'", ''
}
function VocabTokens($ln) {
$ln = $ln -replace '\s*\([^)]*\)', '' # pronunciation notes: (wuns), (a gen'), (as f)
$cols = if ($ln -match '\S\s{2,}\S') { $ln.Trim() -split '\s{2,}' } else { $ln.Trim() -split '\s+' }
foreach ($c in $cols) {
if (-not $c) { continue }
if ($c -match '\s') {
if ($c -match "'") { CleanWord $c } # syllables of one word: be cause'
elseif ($c -match '^(a|an|the)\s') { } # phrase like "a black hen": not a new word
else { $c -split '\s+' | ForEach-Object { CleanWord $_ } } # two words in one column: night owl
} else { CleanWord $c }
}
}
$nums = @()
if ($cfg.headRx) {
# A real lesson heading has a page number ("(74)") or is the next number in order. That skips a lesson's own
# numbered parts ("II. SUMMER.") and survives misprinted numerals ("XVV.", a second "LXX."): lessons are numbered in order.
$keep = @(); $prev = 0
# up to 2 lessons may be missing between two headings; a heading with a page number but a misprinted numeral is the next lesson
foreach ($m in $heads) {
$v = Roman $m.Groups[1].Value
if ($v -ge $prev + 1 -and $v -le $prev + 3) { $keep += $m; $prev = $v; $nums += $v }
# a misprinted numeral is still near its place (XVV for XXV); a lesson's own small parts ("III. A GOOD NAME. (95)" inside lesson 12) are not
elseif ($m.Value -match '\(\d+\)\s*$' -and $v -ge $prev - 5) { $keep += $m; $prev++; $nums += $prev }
}
$heads = $keep
$parts = @($body.Substring(0, $heads[0].Index))
for ($i = 0; $i -lt $heads.Count; $i++) { $s = $heads[$i].Index + $heads[$i].Length; $e = if ($i + 1 -lt $heads.Count) { $heads[$i + 1].Index } else { $body.Length }; $parts += $body.Substring($s, $e - $s) }
}
# authors from the table of contents: "87. The Best Capital Louisa M. Alcott 245"
$tocAuthors = @{}
# (page column optional: the Fifth Reader's contents have none); the first entry for a number wins, which is the contents page
$tocText = $txt.Substring(0, $txt.LastIndexOf($(if ($cfg.start) { $cfg.start } else { 'x' })) + 1)
if ($cfg.tocEnd) { $ci = $tocText.IndexOf($(if ($cfg.tocStart) { $cfg.tocStart } else { 'CONTENTS' })); $tocText = $tocText.Substring($ci, $tocText.IndexOf($cfg.tocEnd, $ci) - $ci) } # only the contents page, not the chapters on reading aloud
foreach ($m in [regex]::Matches($tocText, '(?m)^\s*(\d+)\.\s+(\S.*?\S)\s{2,}(\S.*?\S)(?:\s{2,}(\d+))?\s*$')) {
$a = (($m.Groups[3].Value -split '\s{2,}')[-1] -replace '\.?\s*\d+$', '').TrimEnd('.') # "Longfellow.290": page number stuck to the name
if ($a -notmatch '^\d' -and -not $tocAuthors.ContainsKey([int]$m.Groups[1].Value)) { $tocAuthors[[int]$m.Groups[1].Value] = $a } # skip a line that holds two entries
}
$lessons = @()
for ($k = 0; $k -lt $heads.Count; $k++) {
$h = $heads[$k].Value.Trim()
$n = if ($h -eq 'Lesson 1') { 1 } else { Roman(([regex]::Match($h, 'LESSON ([IVXLC]+)')).Groups[1].Value) }
$headTitle = ''
if ($cfg.headRx) {
$hm = [regex]::Match($h, $cfg.headRx); $n = $nums[$k]
$headTitle = (Get-Culture).TextInfo.ToTitleCase($hm.Groups[2].Value.Trim().TrimEnd('.').Trim('"', [char]0x201c, [char]0x201d).TrimEnd('.').ToLower()) -creplace "(\w) N't", '$1n''t' -replace '\(Concluded\.?\)', '(concluded)' -replace '\(Continued\.?\)', '(continued)'
}
$review = $h -match 'REVIEW'
$block = $parts[$k + 1]
$block = $block -replace '\[Footnote:[^\]]*\]', '' -replace '(?m)^\s*(\d+|[IVX]+)\.\s*$', ''
# a title line glued to its poem ("ALICE'S SUPPER." then the verses) gets its own paragraph
$block = $block -creplace '(?m)^([A-Z][A-Z .,''\u2019!-]{3,})$\n(?=\S)', "`$1`n`n"
$author = ''; $defs = @(); $questions = @()
# DEFINITIONS / EXERCISES / NOTES at the end of a lesson (Fourth Reader)
$tm = [regex]::Match($block, '(?m)^[ \t]*(?:DEFINITIONS|EXERCISES|NOTES?)\b')
if ($tm.Success) {
$tail = $block.Substring($tm.Index); $block = $block.Substring(0, $tm.Index)
foreach ($sec in [regex]::Matches($tail, '(?s)(DEFINITIONS|EXERCISES|NOTES?)\.?\s*-*(.*?)(?=(?:\n[ \t]*(?:DEFINITIONS|EXERCISES|NOTES?)\b)|\z)')) {
$txt2 = ($sec.Groups[2].Value -replace '\s+', ' ').Trim()
if ($sec.Groups[1].Value -eq 'DEFINITIONS') {
$txt2 = $txt2 -replace '^In defining words, that meaning is given which is appropriate to them in the connection in which they are used\.\s*', ''
foreach ($ln in (($txt2 -replace '(?:^|\s)\d+\.\s+', "`n") -split "`n")) {
foreach ($entry in [regex]::Split($ln.Trim(), '(?<=\.)\s+(?=[A-Z][\w''/\-]*(?: [\w''/\-]+){0,2},\s)')) {
$cm = [regex]::Match($entry, '^([A-Z][\w''/\-]*(?: [\w''/\-]+){0,2}),\s+(.+?)\.?\s*$')
if ($cm.Success) {
$w = $cm.Groups[1].Value -replace '[''/\-]', ''
$d = ($cm.Groups[2].Value -replace '(?<=[a-z])-(?=[a-z])', '').Trim()
$defs += ,@($w, $d)
}
}
}
} elseif ($sec.Groups[1].Value -eq 'EXERCISES') {
$questions += @([regex]::Split($txt2, '(?<=\?)\s+') | ForEach-Object { $_.Trim() } | Where-Object { $_ -match '\?$' })
}
}
}
if ($cfg.prose) {
# the author's name under a poem: "John G. Whittier.", "Mrs. Follen."
$am = [regex]::Match($block, '\n[ \t]*((?:Mrs?\.\s+)?(?:[A-Z]\.\s*|[A-Z][a-z]+\s+){0,3}[A-Z][a-z]+\.)\s*$')
if ($am.Success) { $author = $am.Groups[1].Value.TrimEnd('.'); $block = $block.Substring(0, $am.Index) }
$block = $block -creplace '(?m)^(?=[ \t]*\d+\.\s+\S)', "`n" # each numbered paragraph is its own paragraph
foreach ($sp in ([regex]::Matches($block, '(?m)^([A-Z][a-z]+)\.\s+(?=["A-Z])') | Group-Object { $_.Groups[1].Value } | Where-Object { $_.Count -ge 2 -and $_.Name -notmatch '^(Mr|Mrs|Dr|St)$' })) {
$block = $block -creplace "(?m)^$($sp.Name)\.\s+", "`n$($sp.Name): " # a play: "Rose. See how..." -> "Rose: See how..."
}
}
# pull out illustrations (may span lines); script exercises become the cursive copy work
$script = @(); $picText = ''
foreach ($m in [regex]::Matches($block, '\[Illustration:([^\]]*)\]')) {
$c = $m.Groups[1].Value
if ($c -match '^\s*Script [Ee]xercise:') {
$c = $c -replace '^\s*Script [Ee]xercise:', ''
$paras = [regex]::Split($c.Trim(), '\n\s*\n')
foreach ($pp in $paras) {
if (IsPoem $pp) { $script += ($pp -split '\n' | ForEach-Object { $_.Trim() } | Where-Object { $_ }) } # poem: keep lines
else { $script += Sentences $pp }
}
} elseif (-not $picText) { $picText = ($c -replace '\s+', ' ').Trim().TrimEnd('.') }
}
$block = [regex]::Replace($block, '\[Illustration:[^\]]*\]', "`n`n@@PIC@@`n`n")
$paras = [regex]::Split($block.Trim(), '\n\s*\n') | ForEach-Object { $_.TrimEnd() } | Where-Object { $_.Trim() }
$words = @(); $letters = @(); $lines = @(); $inText = $false; $first = $true; $title = ''; $about = ''
foreach ($p0 in $paras) {
if ($p0.Trim() -eq '@@PIC@@') { if ($words.Count -or $letters.Count) { $inText = $true }; continue }
$p = $p0 -replace '\s*\([^)]*\)', ''
$isVocab = -not $cfg.prose -and -not $inText -and (($p -replace '\b(Mr|Mrs|Dr|St)\.', '$1') -notmatch '[.!?;,"]') -and ($first -or $p -match '\S\s{2,}\S' -or $p -match '^\S+$')
# phrase rows like "a rat a cat" come after the picture, not before it
if ($isVocab -and $words.Count -and $p -match '^\s*(a|the|an)\s\w+\s{3,}') { $isVocab = $false }
if ($isVocab) {
foreach ($ln in ($p0 -split '\n')) {
foreach ($t in (VocabTokens $ln)) {
if (-not $t) { continue }
$isSound = ($sounds -contains $t) -or ($t.Length -eq 1 -and -not ($first -and ($t -ceq 'a' -or $t -cmatch '^[A-Z]$')) -and $t -cne 'I')
if ($isSound) { if ($letters -notcontains $t) { $letters += $t } } elseif ($words -notcontains $t) { $words += $t }
}
$first = $false
}
continue
}
$inText = $true
# a note about the author before the selection ("By Celia Thaxter, who was born...", "Hester Lynch Thrale. 1739--1821, ...")
if ($cfg.headRx -and -not $lines.Count -and -not $about -and $p -match '\bborn\b|\d{4}\s*-{1,2}\s*\d{2,4}') {
$ta = if ($tocAuthors.ContainsKey([int]$n)) { ($tocAuthors[[int]$n] -split '\s+')[-1] } else { '' }
if ($p -match '^\s*By\s' -or ($ta -and $p -match "\b$([regex]::Escape($ta))\b")) { $about = ($p -replace '\s+', ' ').Trim(); continue }
}
if ($p.Trim() -cmatch '^[A-Z][A-Za-z'' ]{2,40}$' -and -not $lines.Count -and $p.Trim() -cmatch '^(?:(?:[A-Z][a-z'']*|a|an|and|the|of|in|on|at|to|for|with|by)\s?)+$' -and $p.Trim() -match '\s') { $title = $p.Trim(); continue }
if ($p.Trim() -cmatch '^[A-Z][A-Z .,''\u2019!?-]+$' -and ($cfg.prose -or $p -notmatch '\?')) { $title = (Get-Culture).TextInfo.ToTitleCase($p.Trim().TrimEnd('.').ToLower()); continue } # story title: THE WHITE KITTEN.
if (IsPoem $p) { $lines += ($p -split '\n' | ForEach-Object { ($_.Trim()) -replace '^\d+\.\s+', '' } | Where-Object { $_ }) } # poem
elseif ($p -notmatch '[.!?;,"]' -and $p -match '\S\s{3,}\S') { $lines += ($p.Trim() -split '\s{3,}') } # phrase row
else { $lines += Sentences $p }
}
if (-not $lines.Count) { $lines = $script }
$pic = ''; foreach ($key in $pics.Keys) { if ($picText -match "(?i)$key") { $pic = $pics[$key]; break } }
if ($cfg.pics.ContainsKey([int]$n)) { $pic = $cfg.pics[[int]$n] }
if (-not $pic -and $cfg.prose) { foreach ($key in $pics.Keys) { if ("$title $(($lines | Select-Object -First 3) -join ' ')" -match "(?i)\b$key(?:s|es)?\b") { $pic = $pics[$key]; break } } } # no picture notes: guess from the title
if (-not $pic) { $pic = if ($review -or $cfg.prose) { '📖' } else { '🖼️' } }
$o = [ordered]@{ n = $n; pic = $pic; picText = $(if ($picText) { $picText } elseif ($review) { 'Review' } else { '' }); words = @($words); letters = @($letters); lines = @($lines) }
if ($review) { $o.review = $true }
if ($headTitle) { $title = $headTitle } # the heading's title wins over part titles inside the lesson ("IV. WINTER.")
if ($defs.Count) {
# the book's own new words: shown in lowercase unless the lesson only ever writes them with a capital (names)
$alltext = $lines -join ' '
$o.defs = [ordered]@{}
foreach ($dd in $defs) {
$w = $dd[0]; if ($alltext -cmatch "\b$([regex]::Escape($w.ToLower()))\b" -or -not ($alltext -cmatch "\b$([regex]::Escape($w))\b")) { $w = $w.ToLower() }
if (-not $o.defs.Contains($w)) { $o.defs[$w] = $dd[1] }
}
$o.words = @($o.defs.Keys | Select-Object -First 12)
}
if ($questions.Count) { $o.questions = @($questions) }
if ($title) { $o.title = $title }
if ($cfg.headRx -and $tocAuthors.ContainsKey([int]$n)) { $author = $tocAuthors[[int]$n] } # the contents page knows best
if ($author -match 'Reader|EXAMPLES') { $author = '' }
if ($author) { $o.author = $author }
if ($about) { $o.about = $about }
if ($script.Count) { $o.script = @($script) }
if (-not $lines.Count) { Write-Host "Lesson $n has no text in Gutenberg's copy ($title), skipped" -ForegroundColor Yellow; continue }
$lessons += $o
}
if ($cfg.prose) {
# New words = words not met in the earlier books or earlier lessons, in reading order, up to 10 per lesson
$seen = New-Object 'System.Collections.Generic.HashSet[string]' ([StringComparer]::OrdinalIgnoreCase)
$wordRx = "[A-Za-z]+(?:'[A-Za-z]+)?"
foreach ($prev in $cfg.after) {
$pj = Get-Content "$root\public\books\$prev.json" -Raw -Encoding UTF8 | ConvertFrom-Json
foreach ($PL in $pj.lessons) { foreach ($s in @($PL.lines) + @($PL.words) + @($PL.script)) { foreach ($m in [regex]::Matches("$s", $wordRx)) { [void]$seen.Add($m.Value) } } }
}
foreach ($o in $lessons) {
$text = (@($o.lines) + @($o.script)) -join ' '
$new = @()
foreach ($m in [regex]::Matches($text, $wordRx)) {
$w = $m.Value
if ($w.Length -lt 2 -or $w -match "^n'" -or -not $seen.Add($w)) { continue }
# names keep their capital (capitalized in the middle of a line); other words are shown in lowercase
if (-not ($text -cmatch "(?:[a-z]|\b(?:Mr|Mrs|Miss|Dr)\.)\s+$([regex]::Escape($w))\b") -or ($text -cmatch "\b$([regex]::Escape($w.ToLower()))\b")) { $w = $w.ToLower() }
$new += $w
}
if (-not @($o.words).Count) { $o.words = @($new | Select-Object -First 10) } # the book's own word list wins when it has one
}
}
# Last clean-up of every line: printed page headers that ended up inside a sentence ("... ECLECTIC SERIES. ..."),
# footnote/section marks (#, *, [ ]), and scanned digits ("l0" = 10, "I1" = 11)
function CleanLine($s) {
$s = $s -creplace '\d*\s*\b(?:ECLECTIC (?:SERIES|READER)|(?:FIRST|SECOND|THIRD|FOURTH|FIFTH|SIXTH) READER)\b[.,]?\s*\d*', ' ' -replace '[#*\[\]]', ''
$s = $s -creplace '\bI1\b', 'I' -creplace '(?<=\d)l|l(?=\d)', '1' -creplace '^\d+\s+(?=[a-z])', '' -creplace '(?<=[a-z]{3})\d\b', '' # footnote number on a word ("orthography1"); "I1 will" = I; "l0" = 10; a page number before the text
($s -replace '\s{2,}', ' ').Trim()
}
foreach ($o in $lessons) {
$o.lines = @($o.lines | ForEach-Object { CleanLine $_ } | Where-Object { $_ -match '[A-Za-z]' })
if ($o.script) { $o.script = @($o.script | ForEach-Object { CleanLine $_ } | Where-Object { $_ -match '[A-Za-z]' }) }
}
$out = [ordered]@{ title = $cfg.title; sub = $cfg.sub; color = $cfg.color; source = "Project Gutenberg #$($cfg.gutenberg) (public domain)"; lessons = $lessons }
$json = $out | ConvertTo-Json -Depth 6
New-Item -ItemType Directory -Force "$root\public\books" | Out-Null
[IO.File]::WriteAllText("$root\public\books\$Book.json", $json, (New-Object Text.UTF8Encoding $false))
"$($lessons.Count) lessons"
$lessons | ForEach-Object { "{0,2} {1} w[{2}] l[{3}] lines={4} script={5} | {6}" -f $_.n, $_.pic, ($_.words -join ' '), ($_.letters -join ' '), $_.lines.Count, $(if ($_.script) { $_.script.Count } else { 0 }), $_.lines[0] }
EOFpush-audio.ps1: sends only the recordings CT 306 doesn't have yet, 1,500 at a time. That keeps the container's disk from filling up with WAV files while they're converted.
cat > /root/abcbooks-rebuild/push-audio.ps1 <<'EOF'
# Sends recordings (tools/out/<book>/s<speed>/*.wav) that CT 306 doesn't have as MP3 yet, in batches,
# and converts each batch there (CT 306 has a small disk, so never more than one batch of WAVs at a time).
# Usage: powershell -ExecutionPolicy Bypass -File ABCBooks\tools\push-audio.ps1 [-Book fourth1879] (default: every book)
param([string]$Book = '*')
$root = Split-Path -Parent (Split-Path -Parent $MyInvocation.MyCommand.Path)
$batch = 1500
ssh -o BatchMode=yes proxmox "mkdir -p /root/abcbooks"
scp -o BatchMode=yes "$root\proxmox\convert-audio.sh" proxmox:/root/abcbooks/convert-audio.sh
ssh -o BatchMode=yes proxmox "pct push 306 /root/abcbooks/convert-audio.sh /root/convert-audio.sh"
foreach ($dir in Get-ChildItem "$root\tools\out\$Book\s*" -Directory) {
$rel = "$($dir.Parent.Name)/$($dir.Name)"
$have = @(ssh -o BatchMode=yes proxmox "pct exec 306 -- ls /var/www/abcbooks/audio/$rel" 2>$null) | ForEach-Object { $_ -replace '\.mp3$', '' }
$haveSet = New-Object 'System.Collections.Generic.HashSet[string]'
foreach ($h in $have) { [void]$haveSet.Add($h) }
$todo = @(Get-ChildItem $dir.FullName -Filter *.wav | Where-Object { -not $haveSet.Contains($_.BaseName) })
"$rel : $($todo.Count) to send"
for ($i = 0; $i -lt $todo.Count; $i += $batch) {
$part = $todo[$i..([Math]::Min($i + $batch, $todo.Count) - 1)]
$list = Join-Path $env:TEMP 'abcaudio-list.txt'
[IO.File]::WriteAllLines($list, [string[]]($part | ForEach-Object { "$rel/$($_.Name)" }))
$tar = Join-Path $env:TEMP 'abcaudio.tar'
tar -cf $tar -C "$root\tools\out" -T $list
scp -o BatchMode=yes $tar proxmox:/root/abcbooks/audio.tar
ssh -o BatchMode=yes proxmox "pct push 306 /root/abcbooks/audio.tar /tmp/abcaudio.tar"
ssh -o BatchMode=yes proxmox "unlink /root/abcbooks/audio.tar"
ssh -o BatchMode=yes proxmox "pct exec 306 -- mkdir -p /tmp/abcaudio"
ssh -o BatchMode=yes proxmox "pct exec 306 -- tar -xf /tmp/abcaudio.tar -C /tmp/abcaudio --no-same-owner"
ssh -o BatchMode=yes proxmox "pct exec 306 -- unlink /tmp/abcaudio.tar"
ssh -o BatchMode=yes proxmox "pct exec 306 -- bash /root/convert-audio.sh"
Remove-Item $tar, $list
}
}
ssh -o BatchMode=yes proxmox "pct exec 306 -- df -h /"
EOF2. Make the lesson file
PC (PowerShell). It prints "71 lessons" and one line per lesson:
powershell -ExecutionPolicy Bypass -File C:\ABCBooks\tools\make-book.ps1 -Book second18793. Record it with Eloquence
PC (PowerShell). About 11,900 recordings, about 7 minutes here:
powershell -ExecutionPolicy Bypass -File C:\ABCBooks\tools\make-audio.ps1 -Book second1879Send them to CT 306 (abc-web) and turn them into MP3s, about 8 batches. It ends by showing how full the container's disk is:
powershell -ExecutionPolicy Bypass -File C:\ABCBooks\tools\push-audio.ps14. Put it on the server
PC (PowerShell):
powershell -ExecutionPolicy Bypass -File C:\ABCBooks\deploy.ps1Open https://games.503mail.com/abcbooks/, pick a name, and the green McGuffey's Second Eclectic Reader opens a map of 71 lessons.
Troubleshooting
The logs and the Tablet check page are the same as in the Primer post. For this book:
The book doesn't open, or has no voice
- Open: PC (PowerShell). Both should print 200:
"/abcbooks/books/second1879.json","/abcbooks/audio/second1879.json" | % { (Invoke-WebRequest "https://games.503mail.com$_" -UseBasicParsing).StatusCode }CT 306 running out of disk
- Where: the root disk of CT 306 (abc-web), 8 GB.
- Open: Proxmox host shell.
pct exec 306 -- df -h /pct exec 306 -- du -sh /var/www/abcbooks/audio- Look for: "Use%" under 90%. The first version of the upload script unpacked every recording at once and filled the disk ("No space left on device" from
tar). If that ever happens, delete the half-unpacked folder and run push-audio again; it only sends what's missing:
pct exec 306 -- rm -r /tmp/abcaudioNotes
- Timing: making the lesson file takes a few seconds, recording about 7 minutes, uploading about 5 minutes.
- Every book gets its own post in the ABC Books section. Next are the Third to Sixth Readers, then Spell & Read from the 1867 Pictorial Primer.