artem@raspberrypi:~ $ cd /mnt/nvme/models/nibble-phi3/packed-q8k128-layer0-qkv
artem@raspberrypi:/mnt/nvme/models/nibble-phi3/packed-q8k128-layer0-qkv $ taskset -c 0-3 env RAYON_NUM_THREADS=4 \
    /mnt/nvme/nibble/target/release/phi3-mixed-quant \
    --stats detailed \
    ./packed-shard1.safetensors \
    ./packed-shard2.safetensors
Using config.json: ./config.json
Loading Phi-3 Q8K model (layers=32, hidden=3072)...
Loading from 2 shards...
  Shard 1: ./packed-shard1.safetensors
  Shard 2: ./packed-shard2.safetensors
Runtime: arch=aarch64 | dotprod=yes | rayon=4
Quantized weights:
  Q8K128  matrices=1 weights=28.3M disk=31.2 MiB runtime=27.8 MiB bits/weight=8.25 kernel=compact-sdot-i8xi8
  Q8K256  matrices=96 weights=2.889B disk=3.07 GiB runtime=2.73 GiB bits/weight=8.12 kernel=compact-sdot-i8xi8
  Q4K     matrices=32 weights=805.3M disk=432.0 MiB runtime=432.0 MiB bits/weight=4.50 kernel=candle-k-quants
Storage: linear-F32=13.87 GiB | disk=3.52 GiB | runtime=3.18 GiB | saved=9.6% | compression=4.36x | effective=7.34 bits/weight
EOS token ids: [2, 32000, 32007]

Phi-3 Mini Q8K Conversational AI
Commands: 'exit' to quit | 'reset' to clear history


You: What do You know about Rasberry pi?
Assistant: Rasberry Pi is a small, affordable, single-board computer developed by the Raspberry Pi Foundation in the UK. It was first released in 2012 with the aim of promoting the teaching of basic computer science in schools and developing countries. The Raspberry Pi has since become popular among hobbyists, educators, and professionals for various applications, including home automation, robotics, media centers, and more.

The Raspberry Pi features a Broadcom system-on-a-chip (SoC) with a 700 MHz ARM processor, 256 MB RAM, and 4 GB of storage. It runs on a Linux-based operating system called Raspberry Pi OS (formerly Raspbian), which is optimized for the device. The Raspberry Pi has a 4-inch HDMI display, USB ports, Ethernet, and audio output.

The Raspberry Pi Foundation offers several models, including the Raspberry Pi 3 Model B, Raspberry Pi 4 Model B, and Raspberry Pi Zero W. The Raspberry Pi 4 Model B, released in 2019, features a faster processor, more RAM, and improved connectivity options.

The Raspberry Pi is widely used in educational settings, where it helps students learn programming, electronics, and computer science. It is also used in various DIY projects, such as home automation, robotics, and media centers. The Raspberry Pi' composition of a single-board computer, its affordability, and its versatility make it a popular choice for many enthusiasts and professionals alike.
[Perf: in=32 tok | out=361 tok | TTFT=8.22s | prefill=3.9 t/s | decode=2.97 t/s | p50/p95=332/382 ms | logical BW=9.5 GiB/s]
[State: ctx=393/4096 (9.6%) | KV=393 tok (294.8 MiB) | RSS=3.90 GiB | stop=eos | hist=3 msgs]
[Breakdown/decode: model=336.2 ms/tok | Q8K128=2.4 | Q8K256=233.1 | Q4K=40.3 | non-quant=60.4 | sample=0.04 | render=0.19 ms/tok]
[Breakdown/prefill: model=8.224s | quant=7.978s | non-quant=0.246s]
[System: RSS=3.90 GiB | peak RSS=7.26 GiB | CPU=2400 MHz | temp=66.1 C]

You: And in which year it was appeared or developed, if You know?
Assistant: The Raspberry Pi was developed and first released in 2012 by the Raspberry Pi Foundation in the UK.
[Perf: in=18 tok | out=27 tok | TTFT=5.06s | prefill=3.6 t/s | decode=2.61 t/s | p50/p95=383/389 ms | logical BW=8.3 GiB/s]
[State: ctx=438/4096 (10.7%) | KV=438 tok (328.5 MiB) | RSS=3.96 GiB | stop=eos | hist=5 msgs]
[Breakdown/decode: model=383.1 ms/tok | Q8K128=2.4 | Q8K256=230.6 | Q4K=39.8 | non-quant=110.4 | sample=0.04 | render=0.05 ms/tok]
[Breakdown/prefill: model=5.058s | quant=4.529s | non-quant=0.529s]
[System: RSS=3.96 GiB | peak RSS=7.26 GiB | CPU=2400 MHz | temp=59.5 C]

You: exit

Goodbye!