ben/SmartSpeaker
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
SmartSpeaker/.agents/AGENTS.md
T

9.2 KiB
Raw Permalink Blame History

Project Rules for ESP32-S3 Audio Board Development

This workspace is for the ESP32-S3 AI Smart Speaker Development Board. All agents interacting with this project must follow these guidelines:

  • Target Hardware: UeeKKoo (Waveshare) ESP32-S3-AUDIO-Board-EN (No Battery version).

    • CPU: Dual-core Xtensa 32-bit LX7 CPU, running at 240 MHz (ESP32-S3R8).
    • Memory: 16 MB Flash, 8 MB Octal PSRAM (0.01 GB RAM total).
    • OS: FreeRTOS.
    • Connectivity: Wi-Fi & Bluetooth.
    • USB: 1x USB port.
    • Audio Codec Hardware:
      • Playback (DAC): ES8311 low-power mono audio codec chip.
      • Recording (ADC): ES7210 audio encoder chip.
    • I2S / I2C Pin Mapping:
      • I2S MCLK: GPIO 12
      • I2S BCLK: GPIO 13
      • I2S LRCK: GPIO 14
      • I2S DIN (Microphone): GPIO 15
      • I2S DOUT (Speaker): GPIO 16
      • I2C SDA: GPIO 11
      • I2C SCL: GPIO 10
      • WS2812 RGB LEDs: GPIO 38
    • Key Features & Peripherals:
      • Support for AI Speech Interaction with Dual Digital Microphone Array.
      • Hardware/Software Noise Reduction and Echo Cancellation.
      • Support for External Displays (1.47", 2", 2.8", 3.5" LCDs) and Cameras (OV2640, OV5640).
      • 7-ring programmable RGB LEDs (WS2812B, GRB/RGB order, connected to GPIO 38).
      • PCF85063 RTC chip.

  • Development Environment & Toolchain Details:

    • ESP-IDF Version: v6.0.1 (Target: esp32s3)
    • ESP-IDF Directory: C:\esp\v6.0.1\esp-idf
    • ESP-IDF Tools Path: C:\Espressif\tools
    • Build Command: eim run "idf.py build" or execute build.bat
    • Flash Command: eim run "idf.py flash" or execute flash.bat
    • Serial Monitor Command: eim run "idf.py monitor" or execute monitor.bat

  • Helper Scripts Folder (/scripts):

    • build.bat / build.ps1: Triggers standard incremental project build.
    • flash.bat / flash.ps1: Automatically rebuilds and flashes project to COM5 at 460800 baud.
    • monitor.bat / monitor.ps1: Launches target serial logging monitor on COM5 (press Ctrl + ] to exit).
    • clean.bat / clean.ps1: Deletes the /build folder to enforce clean target configs.

  • Implemented Wi-Fi Functionality:

    • Credentials Storage: Wi-Fi credentials are read from the wifi-credentials file in the project root. This file is parsed by CMake during the configure step, and dynamically generates wifi_credentials.h inside the build directory so credentials are never checked into Git.
    • Station Connection: Initializes Wi-Fi in Station mode and handles auto-reconnect on disconnect (up to 5 retries). Implemented in wifi_connect.c and started in main.c.

  • Implemented Audio Functionality:

    • Startup Sound: Plays startup.mp3 immediately on boot.
    • Chime Assets: The audio files in SPIFFS were generated using agentctl tts:
      • startup.mp3: "Ding Dong, I'm turned on!"
      • connecting_wifi.mp3: "Connecting to the wifi"
      • connected_wifi.mp3: "I'm now connected to the wifi"
      • connected.mp3: "I'm connected!"
      • connect_fail.mp3: "I could not connect"
      • Packaged into internal SPIFFS partition model on the device's flash.
    • Playback Management: Implemented Audio_Play_Music_To_End in audio_driver.c to perform blocking playback, preventing overlapping audio playback during initialization.
    • Codec Init: Initializes ES8311 DAC via I2C at sample rate 16000Hz, stereo (2 channels), 16-bit depth.
    • PA Enable: Drivers enable the speaker power amplifier (PA_CTRL) on pin 8 of the TCA9555 IO expander.
    • Volume Adjust: Set to 85 (0 to 100 range scale) via Volume_Adjustment(85) inside main/main.c.

  • Implemented LED Ring Functionality:

    • State-driven Display: Background FreeRTOS task led_rainbow_task drives the 7 WS2812B LEDs connected to GPIO 38 based on the global connection state:
      • LED_STATE_IDLE: LEDs remain off during initial startup sounds.
      • LED_STATE_CONNECTING: LEDs flash orange (approx. 500ms intervals) during connection.
      • LED_STATE_FAILED: LEDs glow solid red at 20% brightness on failure.
      • LED_STATE_CONNECTED: LEDs run a vibrant, moving color fade using HSV-to-RGB translation rotating around the ring (speed set to approx. 2.25 seconds per cycle at 20% brightness).
    • Dependency: Linked and integrated using the espressif/led_strip (^3.0.1) component.

  • Build & Partition Configurations:

    • Partition Table: Defined in partitions.csv with custom model (SPIFFS, 5900K size) and flash_test (FATFS, 528K size) partitions.
    • SPIFFS Generation: The model partition image is automatically built from folder main/spiffs and flashed using CMake macro spiffs_create_partition_image(model spiffs FLASH_IN_PROJECT) declared inside main/CMakeLists.txt.
    • Flash/PSRAM config: Configured for 16MB flash (CONFIG_ESPTOOLPY_FLASHSIZE_16MB=y) and 8MB Octal SPIRAM (CONFIG_SPIRAM=y & CONFIG_SPIRAM_MODE_OCT=y) via pre-configured sdkconfig imported from Waveshare's demo.

  • Troubleshooting & Porting to ESP-IDF v6.x:

    • I2S Types: ESP-IDF v6 removes the legacy I2S driver definitions. Any parameter defined as legacy i2s_port_t in BSP libraries (e.g., bsp_board.c) must be changed to int.
    • Serial Monitoring: The standard serial monitor idf.py monitor requires an attached interactive TTY. In sandboxed running environments, raw serial read scripts should be used instead.

  • Development Socket Server & Cloudflare Tunneling:

    • Dev Server Folder (/dev-server): Node.js/TypeScript Express & WebSockets server listening on port 8901.
    • Auto-Reload: Configured via nodemon and ts-node using npm run dev in /dev-server.
    • Logging: Console logs when clients connect, send messages, disconnect, or trigger errors.
    • Cloudflare Tunnel Routing: Local configuration config.yml is mapped to forward public traffic for example.com and dashboard.example.com directly to loopback http://127.0.0.1:8901.
    • DNS Mappings: Configure your public hostnames to point to your deployment tunnel or reverse proxy.
    • Command Routing: Monitor clients can broadcast control_device command payloads (volume level adjustments, voice text-to-speech output, LED strip animations, or chime prompts) targeting specific client IDs, which the dev-server routes to standard client device sockets.

  • Real-Time Monitor Dashboard Server (/monitor-server):

    • Service Port: Port 8902 (running via Express). Starts with npm start.
    • Layout & Structure: Built with an enterprise-grade dark theme layout featuring a collapsible left sidebar, header operations cards, and responsive page views (Overview, Device Fleet Manager, and Telemetry Terminal Console).
    • Dynamic Elements: Includes dynamic search boxes, severity filter button groups, CSV/JSON log exporters, device diagnostic modals, and volume/LED controllers that dispatch control instructions directly to connected ESP32-S3 boards.

  • Hardware Model Variants (DEV-1 and PROTO-1):

    • Model Configuration: Defined in the device-config file in the project root. Edit this file to switch the active hardware profile: 
      MODEL=DEV-1    # Standard Waveshare development board (default)
MODEL=PROTO-1  # Prototype board with additional peripherals
    • How it works: CMake reads device-config at configure-time, generates device_config.h in the build directory, and defines either MODEL_DEV_1=1 or MODEL_PROTO_1=1 as a compiler macro on the ${COMPONENT_LIB} target.
    • ⚠️ After changing device-config: Always run scripts/clean.bat followed by scripts/build.bat to force CMake to regenerate device_config.h with the new model value. An incremental build alone will NOT pick up the change.
    • DEV-1 Variant: Includes all core firmware functionalities (Wi-Fi, audio playback, WS2812B LEDs on GPIO 38, WebSocket telemetry).
    • PROTO-1 Variant: Inherits DEV-1 baseline and introduces:
      • Audio I2C/I2S Interface: Same as DEV-1 (GPIO 10–16, EXIO 8 PA_EN), but DIN/DOUT are physically swapped on the PROTO board.
      • GPIO 9: WS2812 RGB LED strip signal (replaces GPIO 38 used on DEV-1)
      • GPIO 5: Haptics motor — PWM output via LEDC driver (1kHz, 10-bit duty cycle)
      • GPIO 4: Ambient light sensor (LDR) — ADC1 Channel 3, analog oneshot read, scaled 0–4095 → 0–100%
      • GPIO 7: Capacitive touch sensor — digital input, active HIGH when touched
      • Background FreeRTOS task proto1_sensor_task polls GPIO 4 and GPIO 7 every 100ms and updates device_state_t fields: ambient_light (int) and touch_active (bool).
      • Haptic intensity is downlink-controllable via set_haptics WebSocket command (value 0–100).

  • Protocol-First Development Rules:

    • API Specification Priority: When implementing any new client/device functionality or modifying messaging structures, you MUST always update API.md FIRST as the absolute source of truth before editing any application or driver code.
    • Implementation Sequence:
      1. Update/define the JSON schema, commands, and fields in API.md.
      2. Write the corresponding host (gateway/server) and client (firmware) code changes matching the specification.
Reference in New Issue View Git Blame Copy Permalink