SmartSpeaker/.agents/AGENTS.md

Project Rules for ESP32-S3 Audio Board Development

This workspace is for the ESP32-S3 AI Smart Speaker Development Board. All agents interacting with this project must follow these guidelines:

• Target Hardware: UeeKKoo (Waveshare) ESP32-S3-AUDIO-Board-EN (No Battery version).

  • CPU: Dual-core Xtensa 32-bit LX7 CPU, running at 240 MHz (ESP32-S3R8).
  • Memory: 16 MB Flash, 8 MB Octal PSRAM (0.01 GB RAM total).
  • OS: FreeRTOS.
  • Connectivity: Wi-Fi & Bluetooth.
  • USB: 1x USB port.
  • Audio Codec Hardware:
    • Playback (DAC): ES8311 low-power mono audio codec chip.
    • Recording (ADC): ES7210 audio encoder chip.
  • I2S / I2C Pin Mapping:
    • I2S MCLK: GPIO 12
    • I2S BCLK: GPIO 13
    • I2S LRCK: GPIO 14
    • I2S DIN (Microphone): GPIO 15
    • I2S DOUT (Speaker): GPIO 16
    • I2C SDA: GPIO 11
    • I2C SCL: GPIO 10
    • WS2812 RGB LEDs: GPIO 38
  • Key Features & Peripherals:
    • Support for AI Speech Interaction with Dual Digital Microphone Array.
    • Hardware/Software Noise Reduction and Echo Cancellation.
    • Support for External Displays (1.47", 2", 2.8", 3.5" LCDs) and Cameras (OV2640, OV5640).
    • 7-ring programmable RGB LEDs (WS2812B, GRB/RGB order, connected to GPIO 38).
    • PCF85063 RTC chip.

• Development Environment & Toolchain Details:

  • ESP-IDF Version: v6.0.1 (Target: esp32s3)
  • ESP-IDF Directory: C:\esp\v6.0.1\esp-idf
  • ESP-IDF Tools Path: C:\Espressif\tools
  • Build Command: eim run "idf.py build" or execute build.bat
  • Flash Command: eim run "idf.py flash" or execute flash.bat
  • Serial Monitor Command: eim run "idf.py monitor" or execute monitor.bat

• Helper Scripts Folder (/scripts):

  • build.bat / build.ps1: Triggers standard incremental project build.
  • flash.bat / flash.ps1: Automatically rebuilds and flashes project to COM5 at 460800 baud.
  • monitor.bat / monitor.ps1: Launches target serial logging monitor on COM5 (press Ctrl + ] to exit).
  • clean.bat / clean.ps1: Deletes the /build folder to enforce clean target configs.

• Implemented Wi-Fi Functionality:

  • Credentials Storage: Wi-Fi credentials are read from the wifi-credentials file in the project root. This file is parsed by CMake during the configure step, and dynamically generates wifi_credentials.h inside the build directory so credentials are never checked into Git.
  • Station Connection: Initializes Wi-Fi in Station mode and handles auto-reconnect on disconnect (up to 5 retries). Implemented in wifi_connect.c and started in main.c.

• Implemented Audio Functionality:

  • Startup Sound: Plays startup.mp3 immediately on boot.
  • Chime Assets: The audio files in SPIFFS were generated using agentctl tts:
    • startup.mp3: "Ding Dong, I'm turned on!"
    • connecting_wifi.mp3: "Connecting to the wifi"
    • connected_wifi.mp3: "I'm now connected to the wifi"
    • connected.mp3: "I'm connected!"
    • connect_fail.mp3: "I could not connect"
    • Packaged into internal SPIFFS partition model on the device's flash.
  • Playback Management: Implemented Audio_Play_Music_To_End in audio_driver.c to perform blocking playback, preventing overlapping audio playback during initialization.
  • Codec Init: Initializes ES8311 DAC via I2C at sample rate 16000Hz, stereo (2 channels), 16-bit depth.
  • PA Enable: Drivers enable the speaker power amplifier (PA_CTRL) on pin 8 of the TCA9555 IO expander.
  • Volume Adjust: Set to 85 (0 to 100 range scale) via Volume_Adjustment(85) inside main/main.c.

• Implemented LED Ring Functionality:

  • State-driven Display: Background FreeRTOS task led_rainbow_task drives the 7 WS2812B LEDs connected to GPIO 38 based on the global connection state:
    • LED_STATE_IDLE: LEDs remain off during initial startup sounds.
    • LED_STATE_CONNECTING: LEDs flash orange (approx. 500ms intervals) during connection.
    • LED_STATE_FAILED: LEDs glow solid red at 20% brightness on failure.
    • LED_STATE_CONNECTED: LEDs run a vibrant, moving color fade using HSV-to-RGB translation rotating around the ring (speed set to approx. 2.25 seconds per cycle at 20% brightness).
  • Dependency: Linked and integrated using the espressif/led_strip (^3.0.1) component.

• Build & Partition Configurations:

  • Partition Table: Defined in partitions.csv with custom model (SPIFFS, 5900K size) and flash_test (FATFS, 528K size) partitions.
  • SPIFFS Generation: The model partition image is automatically built from folder main/spiffs and flashed using CMake macro spiffs_create_partition_image(model spiffs FLASH_IN_PROJECT) declared inside main/CMakeLists.txt.
  • Flash/PSRAM config: Configured for 16MB flash (CONFIG_ESPTOOLPY_FLASHSIZE_16MB=y) and 8MB Octal SPIRAM (CONFIG_SPIRAM=y & CONFIG_SPIRAM_MODE_OCT=y) via pre-configured sdkconfig imported from Waveshare's demo.

• Troubleshooting & Porting to ESP-IDF v6.x:

  • I2S Types: ESP-IDF v6 removes the legacy I2S driver definitions. Any parameter defined as legacy i2s_port_t in BSP libraries (e.g., bsp_board.c) must be changed to int.
  • Serial Monitoring: The standard serial monitor idf.py monitor requires an attached interactive TTY. In sandboxed running environments, raw serial read scripts should be used instead.

• Development Socket Server & Cloudflare Tunneling:

  • Dev Server Folder (/dev-server): Node.js/TypeScript Express & WebSockets server listening on port 8901.
  • Auto-Reload: Configured via nodemon and ts-node using npm run dev in /dev-server.
  • Logging: Console logs when clients connect, send messages, disconnect, or trigger errors.
  • Cloudflare Tunnel Routing: Local configuration config.yml is mapped to forward public traffic for example.com and dashboard.example.com directly to loopback http://127.0.0.1:8901.
  • DNS Mappings: Configure your public hostnames to point to your deployment tunnel or reverse proxy.
  • Command Routing: Monitor clients can broadcast control_device command payloads (volume level adjustments, voice text-to-speech output, LED strip animations, or chime prompts) targeting specific client IDs, which the dev-server routes to standard client device sockets.

• Real-Time Monitor Dashboard Server (/monitor-server):

  • Service Port: Port 8902 (running via Express). Starts with npm start.
  • Layout & Structure: Built with an enterprise-grade dark theme layout featuring a collapsible left sidebar, header operations cards, and responsive page views (Overview, Device Fleet Manager, and Telemetry Terminal Console).
  • Dynamic Elements: Includes dynamic search boxes, severity filter button groups, CSV/JSON log exporters, device diagnostic modals, and volume/LED controllers that dispatch control instructions directly to connected ESP32-S3 boards.

• Hardware Model Variants (DEV-1 and PROTO-1):

  • Model Configuration: Defined in the device-config file in the project root. Edit this file to switch the active hardware profile:

    MODEL=DEV-1    # Standard Waveshare development board (default)
    MODEL=PROTO-1  # Prototype board with additional peripherals
    

  • How it works: CMake reads device-config at configure-time, generates device_config.h in the build directory, and defines either MODEL_DEV_1=1 or MODEL_PROTO_1=1 as a compiler macro on the ${COMPONENT_LIB} target.
  • ⚠️ After changing device-config: Always run scripts/clean.bat followed by scripts/build.bat to force CMake to regenerate device_config.h with the new model value. An incremental build alone will NOT pick up the change.
  • DEV-1 Variant: Includes all core firmware functionalities (Wi-Fi, audio playback, WS2812B LEDs on GPIO 38, WebSocket telemetry).
  • PROTO-1 Variant: Inherits DEV-1 baseline and introduces:
    • Audio I2C/I2S Interface: Same as DEV-1 (GPIO 10–16, EXIO 8 PA_EN), but DIN/DOUT are physically swapped on the PROTO board.
    • GPIO 9: WS2812 RGB LED strip signal (replaces GPIO 38 used on DEV-1)
    • GPIO 5: Haptics motor — PWM output via LEDC driver (1kHz, 10-bit duty cycle)
    • GPIO 4: Ambient light sensor (LDR) — ADC1 Channel 3, analog oneshot read, scaled 0–4095 → 0–100%
    • GPIO 7: Capacitive touch sensor — digital input, active HIGH when touched
    • Background FreeRTOS task proto1_sensor_task polls GPIO 4 and GPIO 7 every 100ms and updates device_state_t fields: ambient_light (int) and touch_active (bool).
    • Haptic intensity is downlink-controllable via set_haptics WebSocket command (value 0–100).

• Protocol-First Development Rules:

  • API Specification Priority: When implementing any new client/device functionality or modifying messaging structures, you MUST always update API.md FIRST as the absolute source of truth before editing any application or driver code.
  • Implementation Sequence:
    1. Update/define the JSON schema, commands, and fields in API.md.
    2. Write the corresponding host (gateway/server) and client (firmware) code changes matching the specification.