📦 deps(thirdparty): update snapshots
This commit is contained in:
ci[bot]
+209
@@ -0,0 +1,209 @@
|
||||
---
|
||||
name: youtube-notetaker
|
||||
description: "Turn YouTube talks into local study notes with slides, transcripts, editable annotations, and a markdown-backed viewer."
|
||||
category: "video"
|
||||
risk: "safe"
|
||||
source: "official"
|
||||
source_repo: "dair-ai/dair-academy-plugins"
|
||||
source_type: "official"
|
||||
date_added: "2026-06-19"
|
||||
author: "DAIR.AI"
|
||||
license: "MIT"
|
||||
license_source: "https://github.com/dair-ai/dair-academy-plugins/blob/main/README.md#license"
|
||||
tags:
|
||||
- dair-academy
|
||||
- ai
|
||||
- workflow
|
||||
tools:
|
||||
- claude-code
|
||||
- codex-cli
|
||||
- cursor
|
||||
---
|
||||
|
||||
# YouTube Notetaker
|
||||
|
||||
## When to Use
|
||||
|
||||
Use when this workflow matches the user request: >
|
||||
|
||||
|
||||
_Source: [dair-ai/dair-academy-plugins](https://github.com/dair-ai/dair-academy-plugins) (MIT)._
|
||||
|
||||
Build a personal library of YouTube talks you study with. Each video becomes one **plain
|
||||
markdown file**: slide snapshots at their timestamps, a full timestamped transcript, and
|
||||
editable notes. A small bundled server renders the library as an interactive deep-dive in the
|
||||
browser. No database, no cloud service. Everything is files on disk you fully own.
|
||||
|
||||
## Architecture (read this first)
|
||||
|
||||
The **markdown library is the single source of truth**. The artifact is a thin HTML shell that
|
||||
fetches from the server and writes notes back. Never hardcode video data into the HTML.
|
||||
|
||||
- **Library:** a plain folder, set by `VIDEO_LIBRARY_DIR` (default `~/video-deepdives/`).
|
||||
- One markdown file per video, **filename slug = YouTube id** (e.g. `RtywqDFBYnQ.md`).
|
||||
- Frontmatter holds video metadata + a `slides` array.
|
||||
- Body holds the full transcript as `[HH:MM:SS] text` lines.
|
||||
- `_media/` holds slide images, **namespaced per video** as `<youtube_id>-slide-NN.jpg`
|
||||
to avoid collisions between videos.
|
||||
- **Server:** `scripts/serve.py`, a single stdlib + PyYAML file. Start it with:
|
||||
```
|
||||
python3 scripts/serve.py --dir ~/video-deepdives --port 8000
|
||||
```
|
||||
It serves the artifact at `/` and a small API the artifact talks to:
|
||||
- `GET /api/video-deepdives` (front page fetches this) lists every video.
|
||||
- `GET /api/video-deepdives/<id>` returns one video `{meta, body}`.
|
||||
- `GET /api/video-deepdives/_media/<file>` serves a slide image.
|
||||
- `PATCH /api/video-deepdives/<id>` with `{fields:{slides:[...]}}` writes notes back.
|
||||
- **It picks up new videos automatically** the moment a markdown file exists. Adding a video
|
||||
means writing a markdown file + media; you almost never touch the HTML.
|
||||
- The `/api/video-deepdives` URL namespace is local to the bundled server.
|
||||
- **Artifact:** `reference/artifact.html`, served by `serve.py` at `/`. A clean reference copy;
|
||||
only rewrite it if the user wants a UI change. For new videos, leave it alone.
|
||||
|
||||
## Requirements
|
||||
|
||||
- `yt-dlp` and `ffmpeg` on PATH (download + frame/scene extraction).
|
||||
- Python 3 with `Pillow` (contact sheet) and `PyYAML` (markdown file + server).
|
||||
```
|
||||
pip install yt-dlp pillow pyyaml # ffmpeg via your package manager
|
||||
```
|
||||
|
||||
## Adding a video — the pipeline
|
||||
|
||||
All helper scripts are in `scripts/`. Work in a scratch dir (e.g. `/tmp/ytnote-<id>/`), then
|
||||
copy final assets into the library. Set `VIDEO_LIBRARY_DIR` once per shell if you don't want the
|
||||
default. **Do not use em dashes (—) or arrows (→) in notes/titles.**
|
||||
|
||||
### 1. Resolve the id and check embeddability
|
||||
```
|
||||
scripts/setup.sh "<youtube_url_or_id>"
|
||||
```
|
||||
Prints the 11-char `YTID`, the scratch dir, the target library path, and whether YouTube
|
||||
**embedding is allowed** (oembed 200) or **blocked** (oembed 401, e.g. some university talks).
|
||||
If blocked, inline playback won't work but the artifact degrades gracefully to an "open at this
|
||||
moment on YouTube" link, so proceed normally.
|
||||
|
||||
### 2. Download video + subtitles
|
||||
```
|
||||
scripts/download.sh "<YTID>" /tmp/ytnote-<YTID>
|
||||
```
|
||||
Uses `yt-dlp` to grab the video (≤720p is plenty for slide frames) and the best available
|
||||
subtitles (manual if present, else auto-captions) as `.vtt`. Also fetches title/uploader.
|
||||
|
||||
### 3. Detect candidate slide timestamps
|
||||
```
|
||||
scripts/detect_slides.sh /tmp/ytnote-<YTID>/video.mp4 /tmp/ytnote-<YTID>
|
||||
```
|
||||
Runs ffmpeg scene detection (`select='gt(scene,0.3)'`) and writes `scene_times.txt` (seconds).
|
||||
0.3 is a good default; lower it (0.2) for subtle slide decks, raise it (0.4) for busy video.
|
||||
|
||||
### 4. Build a contact sheet and CURATE
|
||||
```
|
||||
python3 scripts/contact_sheet.py /tmp/ytnote-<YTID>/video.mp4 /tmp/ytnote-<YTID>/scene_times.txt /tmp/ytnote-<YTID>/contact.jpg
|
||||
```
|
||||
Read `contact.jpg` (labeled with index + timestamp). **This is the human-judgment step:** keep
|
||||
frames that are real content slides; **drop talking-head shots, transitions, duplicates, and
|
||||
blurry mid-animation frames.** Save the kept timestamps (seconds) to `/tmp/ytnote-<YTID>/keep.txt`,
|
||||
one per line. Typical talk yields 15-25 slides.
|
||||
|
||||
### 5. Extract the curated slides at full quality and install to _media
|
||||
```
|
||||
python3 scripts/extract_slides.py <YTID> /tmp/ytnote-<YTID>/video.mp4 /tmp/ytnote-<YTID>/keep.txt > /tmp/ytnote-<YTID>/slides.json
|
||||
```
|
||||
Extracts each kept timestamp at 1280px wide, JPEG, and copies them into
|
||||
`$VIDEO_LIBRARY_DIR/_media/` as `<YTID>-slide-01.jpg`, `-02.jpg`, … (numbered in time order).
|
||||
Progress goes to stderr; a clean `slides.json` scaffold prints to **stdout**, so redirect it to a
|
||||
file as shown, then fill in `title` and `note`.
|
||||
|
||||
Tip: talks are often a slide + speaker-cam composite, and speakers flip back and forth, so the
|
||||
same slide appears at several timestamps. Keep the cleanest instance of each, and re-anchor each
|
||||
slide's `t` to where it is actually discussed in the transcript (better "play from here" UX).
|
||||
|
||||
### 6. Build the transcript
|
||||
```
|
||||
python3 scripts/vtt_to_transcript.py /tmp/ytnote-<YTID>/*.vtt /tmp/ytnote-<YTID>/transcript.txt
|
||||
```
|
||||
Parses the VTT into clean, de-duplicated `[HH:MM:SS] text` lines (YouTube auto-captions repeat
|
||||
rolling text; the script collapses it). This becomes the markdown body.
|
||||
|
||||
### 7. Write notes and assemble the markdown file
|
||||
For each kept slide, write a 1-3 sentence `note` grounded in the transcript around that timestamp
|
||||
(don't invent claims). Then assemble:
|
||||
```
|
||||
python3 scripts/write_library_item.py \
|
||||
--id <YTID> \
|
||||
--title "Talk title" \
|
||||
--speaker "Name, Role, Org" \
|
||||
--tags tag1,tag2,tag3 \
|
||||
--slides /tmp/ytnote-<YTID>/slides.json \
|
||||
--transcript /tmp/ytnote-<YTID>/transcript.txt
|
||||
```
|
||||
Writes `$VIDEO_LIBRARY_DIR/<YTID>.md` with correct frontmatter + body.
|
||||
|
||||
### 8. Serve and verify (always do this)
|
||||
```
|
||||
python3 scripts/serve.py --dir "$VIDEO_LIBRARY_DIR" --port 8000 &
|
||||
scripts/verify.sh <YTID> # defaults to http://127.0.0.1:8000
|
||||
```
|
||||
`verify.sh` curls the collection list, the item, the first slide image, and the artifact,
|
||||
asserting HTTP 200 and that the new id appears in the index. Then open
|
||||
`http://127.0.0.1:8000/#/<YTID>` in a browser to confirm slides + transcript + notes render.
|
||||
|
||||
## Markdown file shape (reference)
|
||||
|
||||
```markdown
|
||||
---
|
||||
id: RtywqDFBYnQ
|
||||
title: Memory and dreaming for self-learning agents
|
||||
youtube_id: RtywqDFBYnQ
|
||||
speaker: Mahesh, Product Manager, Platform team at Anthropic
|
||||
source_url: https://www.youtube.com/watch?v=RtywqDFBYnQ
|
||||
slide_count: 19
|
||||
created: '2026-05-25'
|
||||
tags: [anthropic, memory, agents]
|
||||
slides:
|
||||
- idx: 1
|
||||
t: 55.7 # seconds (float ok), used for seeking
|
||||
mmss: 00:55 # display label
|
||||
title: Agent primitives have evolved
|
||||
note: One to three sentences grounded in the transcript at this timestamp.
|
||||
img: /api/video-deepdives/_media/RtywqDFBYnQ-slide-01.jpg
|
||||
# ... more slides
|
||||
---
|
||||
## Transcript
|
||||
[00:00:08] Hello, everyone...
|
||||
[00:00:11] ...
|
||||
```
|
||||
|
||||
Notes:
|
||||
- `idx` can be sparse/non-contiguous; the artifact sorts slides by `t`, so ordering is by
|
||||
timestamp, not idx.
|
||||
- `img` is always a `/api/video-deepdives/_media/<file>` URL (served by serve.py),
|
||||
never base64.
|
||||
- Slide `note` is what the user edits in the UI; PATCH writes the whole `slides` array back.
|
||||
|
||||
## Gotchas
|
||||
- **Embedding disabled** (oembed 401): inline player is blocked by the video owner. Not a bug;
|
||||
the artifact shows an "open at this moment on YouTube" link instead. Mention it to the user.
|
||||
- **Image collisions:** always namespace media `<YTID>-slide-NN.jpg`. Never reuse bare
|
||||
`slide-NN.jpg` for a new video.
|
||||
- **Auto-caption noise:** rolling YouTube captions duplicate text across cues; use the provided
|
||||
VTT parser, don't dump raw VTT into the body.
|
||||
- **Don't touch existing videos** when adding a new one. Each video is an independent file.
|
||||
- **Server not picking up a video:** confirm the `.md` file is directly inside `--dir` (not a
|
||||
subfolder) and the filename is `<YTID>.md`.
|
||||
|
||||
## What makes this portable
|
||||
- **No orchestrator / no database.** Storage is a plain folder of markdown + images.
|
||||
- **One env var** (`VIDEO_LIBRARY_DIR`) controls where the library lives.
|
||||
- **One small server file** (`serve.py`, stdlib + PyYAML) renders everything and handles
|
||||
note write-back. Drop it anywhere Python runs.
|
||||
- The markdown files are portable: readable in Obsidian or any editor, and the frontmatter is
|
||||
standard YAML.
|
||||
|
||||
|
||||
## Limitations
|
||||
|
||||
- Requires the upstream tool, account, API key, or local setup when the workflow names one.
|
||||
- Does not authorize destructive, production, paid, or external-message actions without explicit user approval.
|
||||
- Validate generated artifacts or recommendations against the user's real sources before treating them as final.
|
||||
+269
@@ -0,0 +1,269 @@
|
||||
<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
|
||||
<title>Video Deep-Dives</title>
|
||||
<link href="https://fonts.googleapis.com/css2?family=Playfair+Display:wght@700;900&family=DM+Sans:wght@300;400;500;600&display=swap" rel="stylesheet">
|
||||
<style>
|
||||
*{box-sizing:border-box;margin:0;padding:0}
|
||||
html,body{height:100%}
|
||||
body{font-family:'DM Sans',system-ui,sans-serif;background:#f4f1eb;color:#1c1a17;line-height:1.55;display:flex;flex-direction:column;height:100vh;overflow:hidden}
|
||||
:root{--slide-max:560px;--vid-max:100%}
|
||||
header{padding:11px 18px;border-bottom:1px solid #e2dccf;background:#fbf9f4;flex:0 0 auto}
|
||||
.eyebrow{font-size:10px;letter-spacing:2px;text-transform:uppercase;color:#8a7e6e;font-weight:600}
|
||||
.backlink{display:none;font-size:11.5px;color:#2a5cbf;text-decoration:none;font-weight:600;cursor:pointer}
|
||||
.backlink:hover{text-decoration:underline}
|
||||
h1{font-family:'Playfair Display',serif;font-size:clamp(17px,2.3vw,25px);font-weight:900;color:#111;line-height:1.05;margin:2px 0}
|
||||
.speaker{color:#4a4236;font-size:12.5px}.speaker a{color:#2a5cbf;text-decoration:none}
|
||||
/* ---------- HOME (index) ---------- */
|
||||
#home{flex:1 1 auto;overflow:auto;padding:22px 26px}
|
||||
.home-intro{max-width:760px;margin:0 auto 22px}
|
||||
.home-intro .lede{font-family:'Playfair Display',serif;font-size:clamp(22px,3vw,34px);font-weight:900;color:#111;line-height:1.08}
|
||||
.home-intro p{color:#5a5145;font-size:13.5px;margin-top:8px;max-width:640px}
|
||||
.grid{display:grid;grid-template-columns:repeat(auto-fill,minmax(300px,1fr));gap:18px;max-width:1100px;margin:0 auto}
|
||||
.card{background:#fff;border:1px solid #e2dccf;border-radius:14px;overflow:hidden;cursor:pointer;text-decoration:none;color:inherit;display:flex;flex-direction:column;transition:transform .14s,box-shadow .14s,border-color .14s}
|
||||
.card:hover{transform:translateY(-3px);box-shadow:0 10px 26px rgba(0,0,0,.10);border-color:#c8920a}
|
||||
.card .thumb{position:relative;aspect-ratio:16/9;background:#000;overflow:hidden}
|
||||
.card .thumb img{width:100%;height:100%;object-fit:cover;display:block}
|
||||
.card .badge{position:absolute;bottom:8px;right:8px;background:rgba(0,0,0,.74);color:#fff;font-size:11px;font-weight:600;padding:2px 8px;border-radius:6px}
|
||||
.card .play{position:absolute;inset:0;margin:auto;width:54px;height:54px;display:flex;align-items:center;justify-content:center;background:rgba(0,0,0,.5);color:#fff;border-radius:50%;font-size:20px;opacity:0;transition:.2s}
|
||||
.card:hover .play{opacity:1}
|
||||
.card .body{padding:13px 15px 15px}
|
||||
.card .ct{font-family:'Playfair Display',serif;font-size:16.5px;font-weight:700;color:#111;line-height:1.22}
|
||||
.card .cs{color:#7a6f5d;font-size:12px;margin-top:6px}
|
||||
.card .tags{margin-top:10px;display:flex;flex-wrap:wrap;gap:6px}
|
||||
.card .tag{font-size:10px;letter-spacing:.6px;text-transform:uppercase;color:#7a5010;background:rgba(180,130,20,.14);border-radius:5px;padding:2px 7px;font-weight:600}
|
||||
#homeErr{display:none;max-width:760px;margin:0 auto;padding:10px 14px;background:#fff0ec;border:1px solid #c87060;border-radius:8px;color:#8a1a1a;font-size:13px}
|
||||
/* ---------- DEEP-DIVE ---------- */
|
||||
#deepdive{flex:1 1 auto;display:none;min-height:0;overflow:hidden}
|
||||
#split{flex:1 1 auto;display:flex;min-height:0;overflow:hidden;width:100%}
|
||||
#left{flex:0 0 58%;min-width:240px;overflow:auto;padding:14px 18px;display:flex;flex-direction:column}
|
||||
#divider{flex:0 0 8px;cursor:col-resize;background:linear-gradient(#ddd6c8,#cfc7b6);position:relative}
|
||||
#divider::after{content:"⋮⋮";position:absolute;top:50%;left:50%;transform:translate(-50%,-50%) rotate(90deg);color:#8a7e6e;font-size:11px;letter-spacing:-2px}
|
||||
#divider:hover,#divider.drag{background:#c8920a}
|
||||
/* RIGHT pane: fixed video region + independently scrolling transcript */
|
||||
#right{flex:1 1 0;min-width:280px;display:flex;flex-direction:column;overflow:hidden;padding:14px 16px;background:#fbf9f4;border-left:1px solid #e2dccf}
|
||||
.rtop{flex:0 0 auto}
|
||||
.toolbar{display:flex;align-items:center;gap:14px;flex-wrap:wrap;background:#fff;border:1px solid #ddd8d0;border-radius:10px;padding:8px 12px;margin-bottom:12px;font-size:12px;color:#4a4236}
|
||||
.toolbar label{display:flex;align-items:center;gap:7px}.toolbar input[type=range]{accent-color:#c8920a}
|
||||
.deck-h{font-family:'Playfair Display',serif;font-size:18px;margin-bottom:3px}
|
||||
.deck-sub{color:#8a7e6e;font-size:12px;margin-bottom:12px}
|
||||
.slide{background:#fff;border:1px solid #ddd8d0;border-radius:12px;box-shadow:0 2px 8px rgba(0,0,0,.05);padding:11px;margin:0 auto 16px;width:100%;max-width:var(--slide-max);transition:border-color .15s,box-shadow .15s}
|
||||
.slide.active{border-color:#b0357a;box-shadow:0 0 0 3px rgba(176,53,122,.16)}
|
||||
.slide-img{position:relative;cursor:pointer;border-radius:9px;overflow:hidden;background:#000}
|
||||
.slide-img img{width:100%;display:block;aspect-ratio:16/9;object-fit:contain;background:#000}
|
||||
.slide-img:hover img{opacity:.85}
|
||||
.play-badge{position:absolute;inset:0;margin:auto;width:54px;height:54px;display:flex;align-items:center;justify-content:center;background:rgba(0,0,0,.55);color:#fff;border-radius:50%;font-size:20px;opacity:0;transition:.2s}
|
||||
.slide-img:hover .play-badge{opacity:1}
|
||||
.slide-t{position:absolute;bottom:8px;right:8px;background:rgba(0,0,0,.72);color:#fff;font-size:11px;font-weight:600;padding:2px 7px;border-radius:5px}
|
||||
.slide-meta{display:flex;align-items:center;justify-content:space-between;gap:10px;margin:10px 2px 8px}
|
||||
.slide-meta h3{font-family:'Playfair Display',serif;font-size:15.5px;color:#111;font-weight:700;line-height:1.25}
|
||||
.btn{cursor:pointer;border:0;background:#2a5cbf;color:#fff;font-size:11.5px;font-weight:600;padding:5px 11px;border-radius:6px;white-space:nowrap;font-family:'DM Sans'}
|
||||
.btn:hover{background:#1d4699}
|
||||
.note-lbl{display:block;font-size:10px;letter-spacing:1.4px;text-transform:uppercase;color:#8a7e6e;font-weight:600;margin:2px 2px 4px}
|
||||
.saved{color:#2e8b57;letter-spacing:0;text-transform:none;font-weight:500;margin-left:6px}
|
||||
.note-area{width:100%;min-height:70px;resize:vertical;border:1px solid #e3dccb;border-radius:8px;background:#fdfcf8;padding:9px 11px;font-family:'DM Sans';font-size:13px;color:#3a342b;line-height:1.55}
|
||||
.note-area:focus{outline:none;border-color:#c8920a;background:#fffdf6}
|
||||
.vidwrap{max-width:var(--vid-max);margin:0 auto}
|
||||
.vid{position:relative;width:100%;aspect-ratio:16/9;border-radius:10px;overflow:hidden;background:#000}
|
||||
.vid iframe{position:absolute;inset:0;width:100%;height:100%;border:0}
|
||||
.now{margin-top:10px;background:#fdf9f0;border:1px solid #e6dcc4;border-radius:9px;padding:10px 13px}
|
||||
.now .lbl{font-size:10px;letter-spacing:1.5px;text-transform:uppercase;color:#7a5010;font-weight:600;display:flex;gap:8px;align-items:center}
|
||||
.now .lbl b{background:rgba(180,130,20,.14);color:#7a5010;border-radius:5px;padding:2px 7px;font-size:11px}
|
||||
.now p{margin-top:6px;font-size:13px;color:#3a342b;max-height:96px;overflow:auto}
|
||||
.tr-h{font-family:'Playfair Display',serif;font-size:15px;margin:12px 2px 6px}
|
||||
.tsearch{width:100%;padding:8px 11px;border:1px solid #ddd8d0;border-radius:8px;font-size:12.5px;margin-bottom:8px;font-family:'DM Sans'}
|
||||
#transcript{flex:1 1 0;overflow:auto;border:1px solid #eee4d4;border-radius:9px;background:#fffdf8}
|
||||
.trow{display:flex;gap:9px;padding:6px 11px;cursor:pointer;border-bottom:1px solid #f1ead9;font-size:12.5px}
|
||||
.trow:hover{background:#fdf3df}.trow.hl{background:#fce9bd}
|
||||
.tt{color:#2a5cbf;font-weight:600;font-size:11.5px;min-width:54px;flex-shrink:0}
|
||||
#err{display:none;padding:10px 14px;background:#fff0ec;border:1px solid #c87060;border-radius:8px;color:#8a1a1a;font-size:13px;margin:10px 0}
|
||||
@media(max-width:760px){#split{flex-direction:column}#left,#right{flex:1 1 auto}#divider{display:none}}
|
||||
</style></head><body>
|
||||
<header>
|
||||
<a class="backlink" id="backlink" href="#/">← All videos</a>
|
||||
<div class="eyebrow">Video deep-dives · markdown-backed</div>
|
||||
<h1 id="title">Video Deep-Dives</h1>
|
||||
<div class="speaker" id="speaker">A growing library of talks I'm studying. Slides, transcripts, and editable notes, all backed by markdown files.</div>
|
||||
</header>
|
||||
|
||||
<!-- ===================== HOME / INDEX ===================== -->
|
||||
<div id="home">
|
||||
<div id="homeErr"></div>
|
||||
<div class="grid" id="grid"></div>
|
||||
</div>
|
||||
|
||||
<!-- ===================== DEEP-DIVE ===================== -->
|
||||
<div id="deepdive">
|
||||
<div id="split">
|
||||
<div id="left">
|
||||
<div class="toolbar">
|
||||
<label>Slide size <input type="range" id="slideSize" min="320" max="900" value="560" oninput="document.documentElement.style.setProperty('--slide-max',this.value+'px')"></label>
|
||||
<span style="color:#cfc7b6">|</span><span class="deck-sub" style="margin:0" id="deckcount"></span>
|
||||
</div>
|
||||
<div class="deck-h">Slide deck</div>
|
||||
<div class="deck-sub">Click a slide (or ▶) to play the video from that moment. Notes are editable and saved to markdown.</div>
|
||||
<div id="err"></div>
|
||||
<div id="deck"></div>
|
||||
</div>
|
||||
<div id="divider" title="Drag to resize"></div>
|
||||
<div id="right">
|
||||
<div class="rtop">
|
||||
<div class="toolbar"><label>Video size <input type="range" id="vidSize" min="40" max="100" value="100" oninput="document.documentElement.style.setProperty('--vid-max',this.value+'%')"></label></div>
|
||||
<div class="vidwrap"><div class="vid"><iframe id="ytplayer" src="" allow="autoplay; encrypted-media; fullscreen" allowfullscreen></iframe></div></div>
|
||||
<div class="now"><div class="lbl">Now playing <b id="now-t">--:--</b> <a id="yt-jump" target="_blank" rel="noopener" style="margin-left:auto;color:#2a5cbf;text-decoration:none;font-weight:600;letter-spacing:0;text-transform:none;display:none">open at this moment on YouTube ↗</a></div><p id="now-tx">Click any slide to play the video from that point.</p></div>
|
||||
<div class="tr-h">Full transcript</div>
|
||||
<input class="tsearch" id="tsearch" placeholder="Search transcript…" oninput="filt(this.value)">
|
||||
</div>
|
||||
<div id="transcript"></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<script src="https://www.youtube.com/iframe_api"></script>
|
||||
<script>
|
||||
var API_URL='/api/video-deepdives';
|
||||
var player,ready=false,pending=null,DATA=null,SLIDES=[],SEGS=[];
|
||||
var CURRENT_ID=null, YTID=null, INDEX=null;
|
||||
|
||||
function onYouTubeIframeAPIReady(){player=new YT.Player('ytplayer',{events:{'onReady':function(){ready=true;if(pending!=null){doPlay(pending);pending=null;}}}});}
|
||||
function fmt(t){t=Math.floor(t);return String(Math.floor(t/60)).padStart(2,'0')+':'+String(t%60).padStart(2,'0');}
|
||||
function esc(s){var d=document.createElement('div');d.textContent=s==null?'':s;return d.innerHTML;}
|
||||
|
||||
/* ---------------- Router ---------------- */
|
||||
function route(){
|
||||
var h=(location.hash||'').replace(/^#\/?/,'').trim();
|
||||
if(h){showVideo(h);} else {showHome();}
|
||||
}
|
||||
function showHome(){
|
||||
document.getElementById('deepdive').style.display='none';
|
||||
document.getElementById('home').style.display='block';
|
||||
document.getElementById('backlink').style.display='none';
|
||||
document.getElementById('title').textContent='Video Deep-Dives';
|
||||
document.getElementById('speaker').textContent="A growing library of talks I'm studying. Slides, transcripts, and editable notes, all backed by markdown files.";
|
||||
document.title='Video Deep-Dives';
|
||||
if(INDEX===null) loadIndex();
|
||||
}
|
||||
function showVideo(id){
|
||||
document.getElementById('home').style.display='none';
|
||||
document.getElementById('deepdive').style.display='flex';
|
||||
document.getElementById('backlink').style.display='inline-block';
|
||||
loadVideo(id);
|
||||
}
|
||||
|
||||
/* ---------------- Home / index ---------------- */
|
||||
async function loadIndex(){
|
||||
try{
|
||||
var r=await fetch(API_URL); if(!r.ok) throw new Error('HTTP '+r.status);
|
||||
var d=await r.json();
|
||||
INDEX=(d.items||[]).filter(function(it){return it.youtube_id;});
|
||||
var g=document.getElementById('grid'); g.innerHTML='';
|
||||
if(!INDEX.length){g.innerHTML='<p style="color:#7a6f5d">No videos in the library yet.</p>';return;}
|
||||
INDEX.forEach(function(it){
|
||||
var slides=it.slides||[]; var thumb=(slides[0]&&slides[0].img)||'';
|
||||
var tags=(it.tags||[]).slice(0,3).map(function(t){return '<span class="tag">'+esc(t)+'</span>';}).join('');
|
||||
var a=document.createElement('a'); a.className='card'; a.href='#/'+encodeURIComponent(it.id);
|
||||
a.innerHTML='<div class="thumb">'+(thumb?'<img src="'+esc(thumb)+'" alt="">':'')+'<span class="play">▶</span><span class="badge">'+(it.slide_count||slides.length)+' slides</span></div>'
|
||||
+'<div class="body"><div class="ct">'+esc(it.title||it.id)+'</div>'
|
||||
+'<div class="cs">'+esc(it.speaker||'')+'</div>'
|
||||
+(tags?'<div class="tags">'+tags+'</div>':'')+'</div>';
|
||||
g.appendChild(a);
|
||||
});
|
||||
}catch(e){var el=document.getElementById('homeErr');el.style.display='block';el.textContent='Could not load the video library: '+e.message+'. Is the backend running?';}
|
||||
}
|
||||
|
||||
/* ---------------- Deep-dive ---------------- */
|
||||
async function loadVideo(id){
|
||||
if(CURRENT_ID===id && DATA){return;} // already loaded
|
||||
CURRENT_ID=id;
|
||||
document.getElementById('err').style.display='none';
|
||||
document.getElementById('deck').innerHTML='';
|
||||
document.getElementById('transcript').innerHTML='';
|
||||
try{
|
||||
var r=await fetch(API_URL+'/'+encodeURIComponent(id)); if(!r.ok) throw new Error('HTTP '+r.status);
|
||||
DATA=await r.json(); var m=DATA.meta||{};
|
||||
YTID=m.youtube_id||id;
|
||||
SLIDES=(m.slides||[]).slice().sort(function(a,b){return a.t-b.t;});
|
||||
document.title=m.title||'Video deep-dive';
|
||||
document.getElementById('title').textContent=m.title||'';
|
||||
document.getElementById('speaker').innerHTML=esc(m.speaker||'')+' · <a target="_blank" href="'+(m.source_url||'#')+'">watch on YouTube ↗</a>';
|
||||
document.getElementById('deckcount').textContent=SLIDES.length+' slides · drag the divider ⋮⋮ to resize';
|
||||
document.getElementById('now-t').textContent='--:--';
|
||||
document.getElementById('now-tx').textContent='Click any slide to play the video from that point.';
|
||||
document.getElementById('ytplayer').src='https://www.youtube.com/embed/'+YTID+'?enablejsapi=1&rel=0&playsinline=1';
|
||||
SEGS=parseTranscript(DATA.body||'');
|
||||
renderDeck(); renderTranscript();
|
||||
}catch(e){var el=document.getElementById('err');el.style.display='block';el.textContent='Could not load library data: '+e.message+'. Is the server running?';}
|
||||
}
|
||||
function parseTranscript(body){
|
||||
var out=[],re=/^\[(\d{2}):(\d{2}):(\d{2})\]\s*(.*)$/;
|
||||
body.split('\n').forEach(function(line){var mm=line.match(re);if(mm){var sec=(+mm[1])*3600+(+mm[2])*60+(+mm[3]);out.push({t:sec,text:mm[4]});}});
|
||||
return out;
|
||||
}
|
||||
function renderDeck(){
|
||||
var deck=document.getElementById('deck');deck.innerHTML='';
|
||||
SLIDES.forEach(function(s,i){
|
||||
var d=document.createElement('div');d.className='slide';d.id='slide-'+i;d.dataset.t=s.t;
|
||||
d.innerHTML='<div class="slide-img"><img src="'+esc(s.img)+'" alt="'+esc(s.title)+'"><span class="play-badge">▶</span><span class="slide-t">'+esc(s.mmss||fmt(s.t))+'</span></div>'
|
||||
+'<div class="slide-meta"><h3>'+esc(s.title)+'</h3><button class="btn">▶ Play '+esc(s.mmss||fmt(s.t))+'</button></div>'
|
||||
+'<label class="note-lbl">Notes <span class="saved" id="saved-'+i+'"></span></label>'
|
||||
+'<textarea class="note-area" id="note-'+i+'"></textarea>';
|
||||
deck.appendChild(d);
|
||||
d.querySelector('textarea').value=s.note||'';
|
||||
d.querySelector('.slide-img').onclick=function(){play(i);};
|
||||
d.querySelector('.btn').onclick=function(){play(i);};
|
||||
d.querySelector('textarea').addEventListener('input',function(){onNote(i,this.value);});
|
||||
});
|
||||
}
|
||||
function renderTranscript(){
|
||||
var c=document.getElementById('transcript');c.innerHTML='';
|
||||
SEGS.forEach(function(seg){
|
||||
var r=document.createElement('div');r.className='trow';r.dataset.t=seg.t;r.dataset.text=seg.text.toLowerCase();
|
||||
r.innerHTML='<span class="tt">'+fmt(seg.t)+'</span><span class="tx">'+esc(seg.text)+'</span>';
|
||||
r.onclick=function(){seekOnly(seg.t);};c.appendChild(r);
|
||||
});
|
||||
}
|
||||
function loadAt(t){
|
||||
// Robust across video switches: use the JS API to load the right video at t.
|
||||
var vd=player.getVideoData?player.getVideoData():null;
|
||||
if(vd && vd.video_id===YTID){player.seekTo(t,true);player.playVideo();}
|
||||
else{player.loadVideoById({videoId:YTID,startSeconds:Math.floor(t)});}
|
||||
}
|
||||
function doPlay(t){loadAt(t);}
|
||||
function srcFallback(t){document.getElementById('ytplayer').src='https://www.youtube.com/embed/'+YTID+'?enablejsapi=1&rel=0&playsinline=1&autoplay=1&start='+Math.floor(t);}
|
||||
function setJump(t){var a=document.getElementById('yt-jump');if(a){a.href='https://www.youtube.com/watch?v='+YTID+'&t='+Math.floor(t)+'s';a.style.display='inline';}}
|
||||
function play(i){
|
||||
var s=SLIDES[i];
|
||||
document.querySelectorAll('.slide.active').forEach(function(x){x.classList.remove('active')});
|
||||
var card=document.getElementById('slide-'+i);if(card)card.classList.add('active');
|
||||
document.getElementById('now-t').textContent=s.mmss||fmt(s.t);
|
||||
document.getElementById('now-tx').textContent=transcriptAt(s.t)||'(no transcript here)';
|
||||
setJump(s.t);
|
||||
if(ready&&player&&player.loadVideoById)doPlay(s.t);else{pending=s.t;srcFallback(s.t);}
|
||||
hlRow(s.t);
|
||||
}
|
||||
function seekOnly(t){document.getElementById('now-t').textContent=fmt(t);document.getElementById('now-tx').textContent=transcriptAt(t);setJump(t);if(ready&&player&&player.loadVideoById)doPlay(t);else{pending=t;srcFallback(t);}hlRow(t);}
|
||||
function transcriptAt(t){var out=[];SEGS.forEach(function(s){if(s.t>=t-1&&s.t<=t+10)out.push(s.text);});return out.join(' ');}
|
||||
function hlRow(t){var rows=document.querySelectorAll('.trow'),best=null;rows.forEach(function(r){if(parseFloat(r.dataset.t)<=t+0.5)best=r;});document.querySelectorAll('.trow.hl').forEach(function(r){r.classList.remove('hl')});if(best){best.classList.add('hl');best.scrollIntoView({block:'nearest'});}}
|
||||
function filt(q){q=q.toLowerCase().trim();document.querySelectorAll('.trow').forEach(function(r){r.style.display=(!q||r.dataset.text.indexOf(q)>-1)?'flex':'none';});}
|
||||
// note write-back to markdown via PATCH
|
||||
var timers={};
|
||||
function onNote(i,val){
|
||||
SLIDES[i].note=val;
|
||||
var s=document.getElementById('saved-'+i);s.textContent='saving…';
|
||||
clearTimeout(timers[i]);
|
||||
timers[i]=setTimeout(function(){saveNotes(i,s);},700);
|
||||
}
|
||||
async function saveNotes(i,badge){
|
||||
try{
|
||||
var payload={fields:{slides:SLIDES.map(function(s){return {idx:s.idx,t:s.t,mmss:s.mmss,title:s.title,note:s.note,img:s.img};})}};
|
||||
var r=await fetch(API_URL+'/'+encodeURIComponent(CURRENT_ID),{method:'PATCH',headers:{'Content-Type':'application/json'},body:JSON.stringify(payload)});
|
||||
badge.textContent=r.ok?'✓ saved':'save failed';
|
||||
}catch(e){badge.textContent='save failed';}
|
||||
setTimeout(function(){badge.textContent='';},1500);
|
||||
}
|
||||
(function(){var dv=document.getElementById('divider'),sp=document.getElementById('split'),lf=document.getElementById('left'),drag=false;
|
||||
dv.addEventListener('mousedown',function(e){drag=true;dv.classList.add('drag');e.preventDefault();});
|
||||
window.addEventListener('mousemove',function(e){if(!drag)return;var r=sp.getBoundingClientRect();var pct=(e.clientX-r.left)/r.width*100;pct=Math.max(25,Math.min(80,pct));lf.style.flexBasis=pct+'%';});
|
||||
window.addEventListener('mouseup',function(){drag=false;dv.classList.remove('drag');});})();
|
||||
|
||||
window.addEventListener('hashchange',route);
|
||||
route();
|
||||
</script></body></html>
|
||||
+53
@@ -0,0 +1,53 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Build a labeled contact sheet of candidate slide frames for human curation.
|
||||
|
||||
Usage: contact_sheet.py <video.mp4> <scene_times.txt> <out.jpg> [--cols 5] [--thumb 360]
|
||||
|
||||
Reads timestamps (seconds, one per line), grabs a frame at each, lays them out in a
|
||||
grid labeled "<index> | <mm:ss>". Read the output image, then write the timestamps you
|
||||
want to KEEP (real content slides, not talking-head/transition frames) to a keep.txt,
|
||||
one per line. The index labels make it easy to call out which to drop.
|
||||
"""
|
||||
import subprocess, sys, tempfile, os, argparse
|
||||
from PIL import Image, ImageDraw, ImageFont
|
||||
|
||||
def grab(video, t, path, w=360):
|
||||
subprocess.run(["ffmpeg","-hide_banner","-loglevel","error","-ss",str(t),
|
||||
"-i",video,"-frames:v","1","-vf",f"scale={w}:-1","-y",path], check=True)
|
||||
|
||||
def mmss(t):
|
||||
t=int(float(t)); return f"{t//60:02d}:{t%60:02d}"
|
||||
|
||||
def main():
|
||||
ap=argparse.ArgumentParser()
|
||||
ap.add_argument("video"); ap.add_argument("times"); ap.add_argument("out")
|
||||
ap.add_argument("--cols",type=int,default=5); ap.add_argument("--thumb",type=int,default=360)
|
||||
a=ap.parse_args()
|
||||
times=[l.strip() for l in open(a.times) if l.strip()]
|
||||
if not times: sys.exit("no timestamps")
|
||||
tmp=tempfile.mkdtemp()
|
||||
thumbs=[]
|
||||
for i,t in enumerate(times):
|
||||
p=os.path.join(tmp,f"f{i:03d}.jpg")
|
||||
try:
|
||||
grab(a.video,t,p,a.thumb); thumbs.append((i,t,p))
|
||||
except subprocess.CalledProcessError:
|
||||
pass
|
||||
if not thumbs: sys.exit("could not grab any frames")
|
||||
tw=a.thumb; th=int(tw*9/16); lab=22; pad=6
|
||||
cols=a.cols; rows=(len(thumbs)+cols-1)//cols
|
||||
cw=tw+pad*2; ch=th+lab+pad*2
|
||||
sheet=Image.new("RGB",(cols*cw,rows*ch),(20,20,20))
|
||||
d=ImageDraw.Draw(sheet)
|
||||
try: font=ImageFont.truetype("/System/Library/Fonts/Supplemental/Arial Bold.ttf",15)
|
||||
except Exception: font=ImageFont.load_default()
|
||||
for n,(idx,t,p) in enumerate(thumbs):
|
||||
r,c=divmod(n,cols); x=c*cw+pad; y=r*ch+pad
|
||||
im=Image.open(p).convert("RGB").resize((tw,th))
|
||||
sheet.paste(im,(x,y+lab))
|
||||
d.text((x+2,y+2),f"{idx} | {mmss(t)} ({float(t):.1f}s)",fill=(255,210,90),font=font)
|
||||
sheet.save(a.out,quality=85)
|
||||
print(f"contact sheet: {a.out} ({len(thumbs)} frames, {cols}x{rows})")
|
||||
print("Read it, then write the timestamps (seconds) to keep -> keep.txt (one per line).")
|
||||
|
||||
if __name__=="__main__": main()
|
||||
+19
@@ -0,0 +1,19 @@
|
||||
#!/usr/bin/env bash
|
||||
# Scene-detect candidate slide-change timestamps with ffmpeg.
|
||||
# Usage: detect_slides.sh <video.mp4> <out_dir> [threshold]
|
||||
# threshold default 0.3 (lower=more frames for subtle decks, higher=fewer for busy video).
|
||||
set -euo pipefail
|
||||
VIDEO="${1:?usage: detect_slides.sh <video.mp4> <out_dir> [threshold]}"
|
||||
OUT="${2:?usage: detect_slides.sh <video.mp4> <out_dir> [threshold]}"
|
||||
THRESH="${3:-0.3}"
|
||||
mkdir -p "$OUT"
|
||||
|
||||
# showinfo on the scene-selected frames prints pts_time per cut.
|
||||
ffmpeg -hide_banner -i "$VIDEO" \
|
||||
-vf "select='gt(scene,$THRESH)',showinfo" -vsync vfr -f null - 2>"$OUT/ffinfo.log" || true
|
||||
|
||||
grep -oE 'pts_time:[0-9.]+' "$OUT/ffinfo.log" | sed 's/pts_time://' | sort -n -u > "$OUT/scene_times.txt"
|
||||
|
||||
N=$(wc -l < "$OUT/scene_times.txt" | tr -d ' ')
|
||||
echo "Detected $N candidate scene changes (threshold=$THRESH) -> $OUT/scene_times.txt"
|
||||
echo "Next: build a contact sheet and curate which are real content slides."
|
||||
+24
@@ -0,0 +1,24 @@
|
||||
#!/usr/bin/env bash
|
||||
# Download video (<=720p) + best subtitles for slide/transcript extraction.
|
||||
# Usage: download.sh "<YTID>" "<scratch_dir>"
|
||||
set -euo pipefail
|
||||
YTID="${1:?usage: download.sh <YTID> <scratch_dir>}"
|
||||
OUT="${2:?usage: download.sh <YTID> <scratch_dir>}"
|
||||
mkdir -p "$OUT"
|
||||
URL="https://www.youtube.com/watch?v=$YTID"
|
||||
|
||||
# Video: 720p mp4 is plenty for 1280px slide frames; merge to a single file.
|
||||
yt-dlp -f "bestvideo[height<=720][ext=mp4]+bestaudio[ext=m4a]/best[height<=720]" \
|
||||
--merge-output-format mp4 -o "$OUT/video.%(ext)s" "$URL"
|
||||
|
||||
# Subtitles: prefer human captions, fall back to auto. English variants.
|
||||
yt-dlp --skip-download --write-subs --write-auto-subs \
|
||||
--sub-langs "en.*,en" --sub-format vtt -o "$OUT/subs.%(ext)s" "$URL" || true
|
||||
|
||||
# Metadata for title/uploader.
|
||||
yt-dlp --skip-download --print "%(title)s\n%(uploader)s\n%(duration)s" "$URL" \
|
||||
> "$OUT/meta.txt" 2>/dev/null || true
|
||||
|
||||
echo "--- downloaded to $OUT ---"
|
||||
ls -la "$OUT"
|
||||
echo "title/uploader/duration:"; cat "$OUT/meta.txt" 2>/dev/null || true
|
||||
+43
@@ -0,0 +1,43 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Extract curated slide frames at full quality and install them into the library _media dir.
|
||||
|
||||
Usage: extract_slides.py <YTID> <video.mp4> <keep.txt>
|
||||
|
||||
keep.txt: one timestamp (seconds) per line, the frames you chose from the contact sheet.
|
||||
Frames are extracted at 1280px wide, JPEG, numbered in time order, and copied to
|
||||
$VIDEO_LIBRARY_DIR/_media/<YTID>-slide-NN.jpg (default ~/video-deepdives/_media)
|
||||
|
||||
Prints a slides scaffold (idx,t,mmss,img) you can paste into slides.json and then fill
|
||||
in title + note for each. idx here is just the sequence number; ordering is by time.
|
||||
|
||||
The img URL is served by serve.py at /api/video-deepdives/_media/<file>.
|
||||
"""
|
||||
import subprocess, sys, os, json
|
||||
|
||||
LIB = os.path.expanduser(os.environ.get("VIDEO_LIBRARY_DIR", "~/video-deepdives"))
|
||||
MEDIA = os.path.join(LIB, "_media")
|
||||
IMG_PREFIX = "/api/video-deepdives/_media" # served by serve.py
|
||||
|
||||
def mmss(t):
|
||||
t=int(round(float(t))); return f"{t//60:02d}:{t%60:02d}"
|
||||
|
||||
def main():
|
||||
if len(sys.argv)!=4: sys.exit("usage: extract_slides.py <YTID> <video.mp4> <keep.txt>")
|
||||
ytid,video,keep=sys.argv[1],sys.argv[2],sys.argv[3]
|
||||
times=sorted({float(l.strip()) for l in open(keep) if l.strip()})
|
||||
if not times: sys.exit("keep.txt is empty")
|
||||
os.makedirs(MEDIA,exist_ok=True)
|
||||
scaffold=[]
|
||||
for i,t in enumerate(times,1):
|
||||
fn=f"{ytid}-slide-{i:02d}.jpg"
|
||||
out=os.path.join(MEDIA,fn)
|
||||
subprocess.run(["ffmpeg","-hide_banner","-loglevel","error","-ss",f"{t}",
|
||||
"-i",video,"-frames:v","1","-vf","scale=1280:-1","-q:v","3","-y",out],check=True)
|
||||
scaffold.append({"idx":i,"t":round(t,1),"mmss":mmss(t),"title":"","note":"",
|
||||
"img":f"{IMG_PREFIX}/{fn}"})
|
||||
print(f" wrote {fn} @ {mmss(t)}",file=sys.stderr)
|
||||
print(f"\nInstalled {len(scaffold)} slides to {MEDIA}",file=sys.stderr)
|
||||
print("--- slides.json scaffold on stdout; redirect to a file, then fill in title + note ---",file=sys.stderr)
|
||||
print(json.dumps(scaffold,indent=2))
|
||||
|
||||
if __name__=="__main__": main()
|
||||
+222
@@ -0,0 +1,222 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Standalone viewer + API server for a YouTube deep-dive library.
|
||||
|
||||
Zero framework dependencies (Python stdlib + PyYAML). It serves the interactive
|
||||
artifact and a small read/write API over a plain folder of markdown files, so the
|
||||
whole thing runs anywhere with no custom backend.
|
||||
|
||||
python3 serve.py [--dir LIBRARY] [--port 8000] [--artifact path/to/artifact.html]
|
||||
|
||||
LIBRARY defaults to $VIDEO_LIBRARY_DIR or ~/video-deepdives. Layout:
|
||||
LIBRARY/<YTID>.md one markdown file per video (frontmatter + transcript)
|
||||
LIBRARY/_media/<YTID>-slide-NN.jpg slide images
|
||||
|
||||
Routes (the artifact talks to these; the /api/video-deepdives namespace is
|
||||
arbitrary and kept only so the same artifact HTML works unmodified):
|
||||
GET / the artifact (single-page app)
|
||||
GET /api/video-deepdives list every video (flattened frontmatter)
|
||||
GET /api/video-deepdives/<id> one video: {meta, body}
|
||||
GET /api/video-deepdives/_media/<f> a slide image
|
||||
PATCH /api/video-deepdives/<id> merge {fields:{...}} into frontmatter, rewrite
|
||||
"""
|
||||
import argparse, json, os, sys, re, mimetypes, posixpath
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
from pathlib import Path
|
||||
from tempfile import TemporaryDirectory
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError:
|
||||
sys.exit("pip install pyyaml")
|
||||
|
||||
API = "/api/video-deepdives"
|
||||
FM_RE = re.compile(r"^---\n(.*?)\n---\n?(.*)$", re.DOTALL)
|
||||
SAFE_SLUG_RE = re.compile(r"^[A-Za-z0-9_-]+$")
|
||||
SAFE_MEDIA_RE = re.compile(r"^[A-Za-z0-9_.-]+$")
|
||||
SAFE_CTYPE_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9!#$&^_.+-]*/[A-Za-z0-9][A-Za-z0-9!#$&^_.+-]*(?:; charset=[A-Za-z0-9._-]+)?$")
|
||||
|
||||
|
||||
def split_frontmatter(text):
|
||||
"""Return (meta_dict, body_str) from a markdown file with YAML frontmatter."""
|
||||
m = FM_RE.match(text)
|
||||
if not m:
|
||||
return {}, text
|
||||
meta = yaml.safe_load(m.group(1)) or {}
|
||||
return meta, m.group(2)
|
||||
|
||||
|
||||
def dump_file(meta, body):
|
||||
out = "---\n" + yaml.safe_dump(meta, sort_keys=False, allow_unicode=True, width=100) + "---\n"
|
||||
return out + body
|
||||
|
||||
|
||||
def library_path(lib, *parts):
|
||||
root = Path(lib).resolve()
|
||||
candidate = root.joinpath(*parts).resolve()
|
||||
try:
|
||||
candidate.relative_to(root)
|
||||
except ValueError:
|
||||
return None
|
||||
return candidate
|
||||
|
||||
|
||||
def safe_content_type(ctype):
|
||||
return ctype if isinstance(ctype, str) and SAFE_CTYPE_RE.match(ctype) else "application/octet-stream"
|
||||
|
||||
|
||||
def load_item(lib, slug):
|
||||
if not SAFE_SLUG_RE.match(slug):
|
||||
return None
|
||||
path = library_path(lib, slug + ".md")
|
||||
if not path or not path.is_file():
|
||||
return None
|
||||
meta, body = split_frontmatter(path.read_text(encoding="utf-8"))
|
||||
return path, meta, body
|
||||
|
||||
|
||||
def list_items(lib):
|
||||
items = []
|
||||
for path in sorted(Path(lib).iterdir()):
|
||||
fn = path.name
|
||||
if not path.is_file() or not fn.endswith(".md") or fn.startswith("_"):
|
||||
continue
|
||||
slug = path.stem
|
||||
loaded = load_item(lib, slug)
|
||||
if not loaded:
|
||||
continue
|
||||
_, meta, body = loaded
|
||||
it = dict(meta)
|
||||
it["slug"] = slug
|
||||
it["file"] = fn
|
||||
it["preview"] = body.strip()[:160]
|
||||
items.append(it)
|
||||
return items
|
||||
|
||||
|
||||
class Handler(BaseHTTPRequestHandler):
|
||||
lib = None
|
||||
artifact = None
|
||||
write_token = None
|
||||
|
||||
def log_message(self, *a):
|
||||
pass # quiet
|
||||
|
||||
def _send(self, code, body, ctype="application/json"):
|
||||
ctype = safe_content_type(ctype)
|
||||
if isinstance(body, (dict, list)):
|
||||
body = json.dumps(body).encode()
|
||||
elif isinstance(body, str):
|
||||
body = body.encode()
|
||||
self.send_response(code)
|
||||
self.send_header("Content-Type", ctype)
|
||||
self.send_header("Content-Length", str(len(body)))
|
||||
self.send_header("Access-Control-Allow-Origin", "*")
|
||||
self.send_header("Access-Control-Allow-Methods", "GET, OPTIONS")
|
||||
self.send_header("Access-Control-Allow-Headers", "Content-Type, X-Video-Library-Token")
|
||||
self.end_headers()
|
||||
if self.command != "HEAD":
|
||||
self.wfile.write(body)
|
||||
|
||||
def do_OPTIONS(self):
|
||||
self._send(204, b"")
|
||||
|
||||
def do_GET(self):
|
||||
path = self.path.split("?", 1)[0].rstrip("/") or "/"
|
||||
if path in ("/", "/index.html"):
|
||||
try:
|
||||
return self._send(200, open(self.artifact, encoding="utf-8").read(), "text/html; charset=utf-8")
|
||||
except OSError:
|
||||
return self._send(500, {"error": "artifact not found: " + self.artifact})
|
||||
|
||||
if path == API:
|
||||
items = list_items(self.lib)
|
||||
return self._send(200, {"collection": "video-deepdives", "total": len(items), "items": items})
|
||||
|
||||
if path.startswith(API + "/_media/"):
|
||||
fn = posixpath.basename(path) # strip any traversal
|
||||
if not SAFE_MEDIA_RE.match(fn):
|
||||
return self._send(400, {"error": "bad media name"})
|
||||
fp = library_path(self.lib, "_media", fn)
|
||||
if not fp or not fp.is_file():
|
||||
return self._send(404, {"error": "no such media"})
|
||||
ctype = mimetypes.guess_type(str(fp))[0] or "application/octet-stream"
|
||||
return self._send(200, fp.read_bytes(), ctype)
|
||||
|
||||
if path.startswith(API + "/"):
|
||||
slug = posixpath.basename(path)
|
||||
loaded = load_item(self.lib, slug)
|
||||
if not loaded:
|
||||
return self._send(404, {"error": "no such item"})
|
||||
_, meta, body = loaded
|
||||
return self._send(200, {"slug": slug, "type": "video-deepdive", "meta": meta, "body": body.rstrip("\n")})
|
||||
|
||||
return self._send(404, {"error": "not found"})
|
||||
|
||||
def do_PATCH(self):
|
||||
if not self.write_token:
|
||||
return self._send(403, {"error": "writes disabled"})
|
||||
if self.headers.get("X-Video-Library-Token") != self.write_token:
|
||||
return self._send(403, {"error": "bad write token"})
|
||||
path = self.path.split("?", 1)[0].rstrip("/")
|
||||
if not path.startswith(API + "/"):
|
||||
return self._send(404, {"error": "not found"})
|
||||
slug = posixpath.basename(path)
|
||||
loaded = load_item(self.lib, slug)
|
||||
if not loaded:
|
||||
return self._send(404, {"error": "no such item"})
|
||||
fp, meta, body = loaded
|
||||
try:
|
||||
n = int(self.headers.get("Content-Length", 0))
|
||||
payload = json.loads(self.rfile.read(n) or b"{}")
|
||||
except (ValueError, json.JSONDecodeError):
|
||||
return self._send(400, {"error": "bad json"})
|
||||
fields = payload.get("fields", payload) # accept {fields:{...}} or a bare dict
|
||||
if not isinstance(fields, dict):
|
||||
return self._send(400, {"error": "fields must be an object"})
|
||||
meta.update(fields)
|
||||
fp.write_text(dump_file(meta, body), encoding="utf-8")
|
||||
return self._send(200, {"ok": True, "slug": slug, "updated": list(fields.keys())})
|
||||
|
||||
|
||||
def self_test():
|
||||
with TemporaryDirectory() as tmp:
|
||||
root = Path(tmp)
|
||||
(root / "video_1.md").write_text("---\ntitle: Demo\n---\nBody", encoding="utf-8")
|
||||
(root / "_media").mkdir()
|
||||
(root / "_media" / "video_1-slide-01.jpg").write_bytes(b"x")
|
||||
assert load_item(str(root), "video_1")
|
||||
assert load_item(str(root), "../secret") is None
|
||||
assert library_path(str(root), "_media", "../video_1.md") == root.resolve() / "video_1.md"
|
||||
assert safe_content_type("text/html; charset=utf-8") == "text/html; charset=utf-8"
|
||||
assert safe_content_type("text/html\r\nX-Bad: 1") == "application/octet-stream"
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--self-test", action="store_true")
|
||||
ap.add_argument("--dir", default=os.path.expanduser(os.environ.get("VIDEO_LIBRARY_DIR", "~/video-deepdives")))
|
||||
ap.add_argument("--port", type=int, default=int(os.environ.get("VIDEO_LIBRARY_PORT", "8000")))
|
||||
ap.add_argument("--host", default="127.0.0.1")
|
||||
ap.add_argument("--write-token", default=os.environ.get("VIDEO_LIBRARY_WRITE_TOKEN"))
|
||||
here = os.path.dirname(os.path.abspath(__file__))
|
||||
ap.add_argument("--artifact", default=os.path.join(here, "..", "reference", "artifact.html"))
|
||||
a = ap.parse_args()
|
||||
if a.self_test:
|
||||
self_test()
|
||||
return
|
||||
|
||||
lib = os.path.abspath(os.path.expanduser(a.dir))
|
||||
os.makedirs(lib, exist_ok=True)
|
||||
Handler.lib = lib
|
||||
Handler.artifact = os.path.abspath(a.artifact)
|
||||
Handler.write_token = a.write_token
|
||||
n = len([f for f in os.listdir(lib) if f.endswith(".md") and not f.startswith("_")])
|
||||
print(f"Library: {lib} ({n} videos)")
|
||||
print(f"Artifact: {Handler.artifact}")
|
||||
print("Writes: " + ("enabled with X-Video-Library-Token" if Handler.write_token else "disabled (set VIDEO_LIBRARY_WRITE_TOKEN to enable PATCH)"))
|
||||
print(f"Serving on http://{a.host}:{a.port}/ (Ctrl-C to stop)")
|
||||
ThreadingHTTPServer((a.host, a.port), Handler).serve_forever()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
+27
@@ -0,0 +1,27 @@
|
||||
#!/usr/bin/env bash
|
||||
# Resolve a YouTube id from a URL/id, print the scratch dir, and report embeddability.
|
||||
# Usage: setup.sh "<youtube_url_or_id>"
|
||||
#
|
||||
# Library location is configurable via the VIDEO_LIBRARY_DIR env var
|
||||
# (default: ~/video-deepdives). One markdown file per video lives there.
|
||||
set -euo pipefail
|
||||
IN="${1:?usage: setup.sh <youtube_url_or_id>}"
|
||||
LIB="${VIDEO_LIBRARY_DIR:-$HOME/video-deepdives}"
|
||||
|
||||
# Extract 11-char id from common URL shapes, or accept a bare id.
|
||||
YTID="$(printf '%s' "$IN" | sed -nE 's#.*(youtu\.be/|v=|/embed/|/shorts/)([A-Za-z0-9_-]{11}).*#\2#p')"
|
||||
[ -z "$YTID" ] && [ "${#IN}" -eq 11 ] && YTID="$IN"
|
||||
[ -z "$YTID" ] && { echo "Could not parse a YouTube id from: $IN" >&2; exit 1; }
|
||||
|
||||
SCRATCH="/tmp/ytnote-$YTID"
|
||||
mkdir -p "$SCRATCH"
|
||||
|
||||
# Embeddability: oembed returns 200 if embedding allowed, 401 if the owner disabled it.
|
||||
CODE="$(curl -s -o /dev/null -w '%{http_code}' \
|
||||
"https://www.youtube.com/oembed?url=https://www.youtube.com/watch?v=$YTID&format=json" || echo "000")"
|
||||
if [ "$CODE" = "200" ]; then EMBED="allowed"; else EMBED="BLOCKED (oembed $CODE) — inline player disabled, artifact falls back to YouTube link"; fi
|
||||
|
||||
echo "YTID: $YTID"
|
||||
echo "SCRATCH: $SCRATCH"
|
||||
echo "EMBED: $EMBED"
|
||||
echo "LIBRARY: $LIB/$YTID.md"
|
||||
+31
@@ -0,0 +1,31 @@
|
||||
#!/usr/bin/env bash
|
||||
# Verify a video is correctly served by the standalone server + appears in the index.
|
||||
# Usage: verify.sh <YTID> [base_url]
|
||||
# Start the server first: python3 scripts/serve.py --dir <LIBRARY> --port 8000
|
||||
set -uo pipefail
|
||||
YTID="${1:?usage: verify.sh <YTID> [base_url]}"
|
||||
BASE="${2:-http://127.0.0.1:8000}" # standalone serve.py default port
|
||||
COLL="$BASE/api/video-deepdives"
|
||||
fail=0
|
||||
|
||||
code(){ curl -s -o /dev/null -w '%{http_code}' "$1"; }
|
||||
|
||||
echo "1) collection list:"
|
||||
C=$(code "$COLL"); echo " GET $COLL -> $C"; [ "$C" = 200 ] || fail=1
|
||||
if curl -s "$COLL" | grep -q "\"$YTID\""; then echo " ✓ $YTID present in index"; else echo " ✗ $YTID NOT in index"; fail=1; fi
|
||||
|
||||
echo "2) item:"
|
||||
C=$(code "$COLL/$YTID"); echo " GET $COLL/$YTID -> $C"; [ "$C" = 200 ] || fail=1
|
||||
|
||||
echo "3) first slide image:"
|
||||
C=$(code "$COLL/_media/$YTID-slide-01.jpg"); echo " GET .../_media/$YTID-slide-01.jpg -> $C"; [ "$C" = 200 ] || fail=1
|
||||
|
||||
echo "4) artifact shell:"
|
||||
C=$(code "$BASE/"); echo " GET / -> $C"; [ "$C" = 200 ] || fail=1
|
||||
|
||||
if [ "$fail" = 0 ]; then
|
||||
echo "ALL GOOD. Open: $BASE/#/$YTID"
|
||||
else
|
||||
echo "SOME CHECKS FAILED — is serve.py running and pointed at the library that contains $YTID?"
|
||||
fi
|
||||
exit $fail
|
||||
+59
@@ -0,0 +1,59 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Convert a YouTube .vtt (manual or auto-captions) into clean [HH:MM:SS] transcript lines.
|
||||
|
||||
Usage: vtt_to_transcript.py <input.vtt> <output.txt>
|
||||
|
||||
Handles the rolling-duplicate problem in auto-captions: each cue repeats the tail of the
|
||||
previous cue, so we keep only newly-added words per cue and emit one line per cue start
|
||||
time. Strips inline <00:00:00.000> word-timing tags and HTML tags.
|
||||
"""
|
||||
import sys, re, html
|
||||
|
||||
TS=re.compile(r'(\d{2}):(\d{2}):(\d{2})\.\d{3}\s*-->\s*(\d{2}):(\d{2}):(\d{2})')
|
||||
INLINE=re.compile(r'<[^>]+>')
|
||||
|
||||
def hhmmss(h,m,s): return f"[{int(h):02d}:{int(m):02d}:{int(s):02d}]"
|
||||
|
||||
def clean(text):
|
||||
text=INLINE.sub('',text)
|
||||
text=html.unescape(text)
|
||||
return re.sub(r'\s+',' ',text).strip()
|
||||
|
||||
def main():
|
||||
if len(sys.argv)!=3: sys.exit("usage: vtt_to_transcript.py <in.vtt> <out.txt>")
|
||||
raw=open(sys.argv[1],encoding='utf-8',errors='replace').read().splitlines()
|
||||
cues=[] # (start_label, text)
|
||||
i=0; cur=None
|
||||
while i<len(raw):
|
||||
m=TS.search(raw[i])
|
||||
if m:
|
||||
if cur: cues.append(cur)
|
||||
cur=[hhmmss(*m.groups()[:3]),[]]
|
||||
i+=1
|
||||
while i<len(raw) and not TS.search(raw[i]) and raw[i].strip()!='':
|
||||
if raw[i].strip() and not raw[i].strip().isdigit():
|
||||
cur[1].append(clean(raw[i]))
|
||||
i+=1
|
||||
else:
|
||||
i+=1
|
||||
if cur: cues.append(cur)
|
||||
|
||||
# De-duplicate rolling captions: keep only the suffix not already seen.
|
||||
out=[]; seen_words=[]
|
||||
for label,parts in cues:
|
||||
text=clean(' '.join(parts))
|
||||
if not text: continue
|
||||
words=text.split()
|
||||
# find longest overlap of seen tail with this cue's head
|
||||
overlap=0; maxk=min(len(words),len(seen_words))
|
||||
for k in range(maxk,0,-1):
|
||||
if seen_words[-k:]==words[:k]: overlap=k; break
|
||||
new=words[overlap:]
|
||||
if new:
|
||||
out.append(f"{label} {' '.join(new)}")
|
||||
seen_words=(seen_words+new)[-40:] # bounded window
|
||||
with open(sys.argv[2],'w',encoding='utf-8') as f:
|
||||
f.write('\n'.join(out)+'\n')
|
||||
print(f"wrote {len(out)} transcript lines -> {sys.argv[2]}")
|
||||
|
||||
if __name__=="__main__": main()
|
||||
+69
@@ -0,0 +1,69 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Assemble the library markdown file for a video deep-dive.
|
||||
|
||||
Usage:
|
||||
write_library_item.py --id <YTID> --title "..." --speaker "..." \
|
||||
--tags a,b,c --slides slides.json --transcript transcript.txt [--created YYYY-MM-DD]
|
||||
|
||||
slides.json: a JSON array of slide objects. Each:
|
||||
{
|
||||
"idx": 1, # sequence/original frame number (display only; sorted by t)
|
||||
"t": 55.7, # seconds (float ok) — used for video seeking
|
||||
"mmss": "00:55", # display label
|
||||
"title": "Slide title", # short headline
|
||||
"note": "1-3 sentences grounded in the transcript at this timestamp.",
|
||||
"img": "/api/video-deepdives/_media/<YTID>-slide-01.jpg"
|
||||
}
|
||||
|
||||
Writes $VIDEO_LIBRARY_DIR/<YTID>.md (default ~/video-deepdives/<YTID>.md)
|
||||
with YAML frontmatter + transcript body. No em dashes or arrows in titles/notes.
|
||||
"""
|
||||
import argparse, json, os, sys, datetime
|
||||
try:
|
||||
import yaml
|
||||
except ImportError:
|
||||
sys.exit("pip install pyyaml")
|
||||
|
||||
LIB = os.path.expanduser(os.environ.get("VIDEO_LIBRARY_DIR", "~/video-deepdives"))
|
||||
|
||||
def main():
|
||||
ap=argparse.ArgumentParser()
|
||||
ap.add_argument("--id",required=True)
|
||||
ap.add_argument("--title",required=True)
|
||||
ap.add_argument("--speaker",default="")
|
||||
ap.add_argument("--tags",default="")
|
||||
ap.add_argument("--slides",required=True)
|
||||
ap.add_argument("--transcript",required=True)
|
||||
ap.add_argument("--created",default=datetime.date.today().isoformat())
|
||||
a=ap.parse_args()
|
||||
|
||||
slides=json.load(open(a.slides))
|
||||
slides=sorted(slides,key=lambda s:s["t"])
|
||||
for bad in ("—","→"):
|
||||
for s in slides:
|
||||
if bad in (s.get("title") or "")+(s.get("note") or ""):
|
||||
sys.exit(f"Found forbidden char {bad!r} in slide notes/titles; remove it.")
|
||||
|
||||
fm={
|
||||
"id":a.id,
|
||||
"title":a.title,
|
||||
"youtube_id":a.id,
|
||||
"speaker":a.speaker,
|
||||
"source_url":f"https://www.youtube.com/watch?v={a.id}",
|
||||
"slide_count":len(slides),
|
||||
"created":a.created,
|
||||
"tags":[t.strip() for t in a.tags.split(",") if t.strip()],
|
||||
"slides":slides,
|
||||
}
|
||||
body=open(a.transcript,encoding="utf-8").read().strip()
|
||||
os.makedirs(LIB,exist_ok=True)
|
||||
path=os.path.join(LIB,f"{a.id}.md")
|
||||
with open(path,"w",encoding="utf-8") as f:
|
||||
f.write("---\n")
|
||||
yaml.safe_dump(fm,f,sort_keys=False,allow_unicode=True,width=100)
|
||||
f.write("---\n## Transcript\n")
|
||||
f.write(body+"\n")
|
||||
print(f"wrote {path} ({len(slides)} slides, {len(body.splitlines())} transcript lines)")
|
||||
print("Verify with: scripts/verify.sh "+a.id)
|
||||
|
||||
if __name__=="__main__": main()
|
||||
Reference in New Issue
Block a user