Trying AI-Trader on local LLMs

David Pickett

04 Nov 2025 — 1 min read

People

David

Idea

Joe wanted to know what the capabilities of https://github.com/HKUDS/AI-Trader look like in the context for local LLMs

Details

Looks like the assumption of the repo is that you can test multiple AI in agentic fashion against historical stock market data over a period of time (for the current official leaderboard see https://hkuds.github.io/AI-Trader/)
Since local LLMs are typically a little slower than cloud deployments, I modified the code to allow for streaming updates to the benchmark user in the agent action flows
Running a few models over the time period of Oct 1st to Oct 28th was interesting - Qwen3-Next-80B-AWQ-4bit made $986.26 profit and gpt-oss-120b-mxfp4 made $1,306.38 (one of the super tiny models just didn't trade anything at all and stayed at $0 profit but and $0 loss)

Testing Qwen3-Omni Audio Inputs

People David Idea Part of our work in the Kumubot cluster involves being able to work on both text as well as audio recordings - the idea was the figure out the capabilities of Qwen3-Omni on our Blackwell hardware Details * Blackwell software support (at least for the RTX 6000 Pro)

4bit Quant Showdown: Finding the Sweet Spot for Qwen3 Models

People: David Idea: To setup our Kumubot cluster, I went down a rabbit hole benchmarking different quantization methods for Qwen3 8B and 32B models to see which ones actually deliver the best accuracy-to-speed tradeoff in real-world use Details: • ExLlamaV3-4bpw surprisingly beat even BF16 on LiveBench accuracy (60.0 vs 58.

Benchmarking multi-lingual open-source LLMs

People: David Idea: Based on conversations with Keao about machine-translation benchmarking, I ran a bunch of LLM models through MMLU-ProX (lite) French benchmark tests (biology section + full 14-topic suite) to see which ones actually deliver the best mix of speed and accuracy in French. This started because we hypothesized that

Local hardware fine-tuning LLMs for Hawaiian-English translation benchmarking

People * David Idea Exploring memory-efficient fine-tuning techniques for improving Hawaiian-to-English translation using Apple's MLX framework, comparing multiple approaches and optimizing for Mac hardware. Details * Successfully fine-tuned gemma-3-4b-it-4bit on Mac M1 Ultra (128GB RAM) achieving 0.8296 semantic similarity score, a 3.6% improvement over the base model * Discovered