Frame-Analyzer/.github/copilot-instructions.md

The repository is a browser-only local video key-frame extractor (suggested Vite + React + TypeScript).

Be concise and only change files relevant to this feature-area unless asked to expand the scope.

Quick context (what this project is and why)

• Goal: let a user pick a local video file, scan for visually significant frames, and copy/download a small set (4–10) of full-resolution stills. No backend; everything runs in the browser.
• Primary design choices: use HTMLVideoElement + Canvas API for V1 sampling; avoid heavy WASM tools (ffmpeg.wasm, OpenCV.js) for initial implementation. Move analysis to a Worker/OffscreenCanvas in V2.

Where to look (key files/dirs)

• design-document.md — authoritative implementation notes and algorithms (frame difference, histogram, phash), default settings, and suggested file layout.
• Suggested source layout (not yet implemented): src/app, src/components, src/media, src/analysis, src/workers, src/types — use these paths if adding code.

High-level architecture

• File picker → createObjectURL → hidden <video> element → sampling loop (seek → draw to small analysis canvas) → compare to last accepted key-frame → collect candidate timestamps → rank/time-bucket → render full-res frames only for chosen candidates → Copy/Download.

What a PR should do (examples of correct changes)

• Add src/media/seekVideo.ts which implements the seek+promise pattern (listen for seeked and error). See design-document.md for example.
• Add src/analysis/frameDifference.ts that implements luminance-based diff with the pixelDeltaThreshold and changedPixelRatioThreshold defaults from design-document.md.
• UI work should use React + TypeScript; keep state minimal (video meta, settings, candidates, progress). Prefer small focused components listed in the suggested file structure.

Project-specific patterns and conventions

• Use a small analysis canvas for pixel comparisons (defaults: 160×90). Do not store or compare full-resolution frames during scanning.
• Compare each sampled frame against the last accepted key-frame, not just the immediately previous sample. Enforce a minimum time gap (default 2s) between accepted frames.
• Keep candidate blobs count small (4–10 final images). Avoid persisting many full-res canvases in memory.
• Clipboard writes must be triggered by user gestures and require secure contexts (HTTPS / localhost). Provide a download fallback on clipboard failure.

Important defaults and constants (from design-document.md)

• ANALYSIS_W = 160, ANALYSIS_H = 90
• SAMPLE_INTERVAL = 0.5 (seconds)
• PIXEL_DELTA_THRESHOLD = 32
• CHANGED_PIXEL_RATIO_THRESHOLD = 0.18
• MIN_SECONDS_BETWEEN_ACCEPTED = 2.0

Build / dev / test workflows

• Recommended dev setup: Vite + React + TypeScript. Example commands to put into README when scaffolding (these are the expected scripts if project is bootstrapped):
  • npm install
  • npm run dev (Vite dev server)
  • npm run build
  • npm run preview

Debugging tips specific to this repo

• The video decoding/seek behavior is the primary cause of flakiness: verify video.duration and video.currentTime after seeked events. Use conservative bounds (Math.min(time, duration)).
• If no candidates are found, test with a lower threshold or ensure the sampling interval covers the video's content.
• For clipboard issues, test locally over localhost or via an HTTPS preview server (Vite's dev server is fine for local testing).

Integration points and possible pitfalls

• Browser codec limitations: prefer MP4/H.264 for broad compatibility. WebCodecs can be added later but is optional.
• Worker migration: initial implementation may keep seek and drawImage on main thread; move only pixel analysis to the worker (send ImageData buffers). See the Worker message shapes in design-document.md.

Small concrete examples to copy into PRs

• Seek helper (expected file: src/media/seekVideo.ts): the repo expects a promise-based seekVideo(video, time) which resolves on seeked or rejects on error.
• Analysis function (expected file: src/analysis/frameDifference.ts): implement frameDifferenceRatio(a: ImageData, b: ImageData, pixelDeltaThreshold) returning the fraction of pixels changed.

What to avoid

• Do not read the entire File into memory with FileReader.readAsDataURL. Use URL.createObjectURL(file) and URL.revokeObjectURL().
• Do not generate or buffer many full-resolution canvases during scanning.
• Avoid adding ffmpeg.wasm / OpenCV.js as a V1 dependency — explain the tradeoffs in the PR description if proposing them.

Testing guidance

• Unit-test pure analysis code (frameDifference, histogramDiff, ranking). These are deterministic and fast.
• For integration/e2e, run the app in Vite dev and manually test with representative videos (MP4/H.264). Document repro steps for flakiness (seek failures, large files).

When in doubt

• Prefer minimal, well-scoped changes that mirror the design-doc algorithms. Reference specific sections of design-document.md in PR descriptions.

Ask for feedback

• If any section of this file is unclear or you want more examples (component props, state types, worker message shapes), tell me which area and I will expand with code snippets or tests.