Fix Word Salad (Tensor Corruption) on ARM Devices for i2_s Quantization#551
Fix Word Salad (Tensor Corruption) on ARM Devices for i2_s Quantization#551uno-km wants to merge 5 commits intomicrosoft:mainfrom
Conversation
Changes: Unified QK Standard: Strictly enforced QK_I2_S = 128 across NEON and Scalar paths to match the standard GGUF packing layout. Refactored Loop Logic: Removed legacy group32_num and la_num chunks. Replaced with a clean, block-level loop to prevent pointer corruption. NEON Optimization: Implemented a dual 16-byte chunk load strategy within the 32-byte weight block to maximize SIMD register utilization. Mathematical Alignment: Synchronized bit-unpacking order (MSB to LSB) with the AVX2 reference. Implemented 32-stride interleaved memory fetching for activations (y). Removed redundant (-1) offset mapping to leverage zero-mean distribution properties, matching the high-performance AVX2 kernel behavior. Result: Completely resolved the word salad issue on Exynos/Snapdragon chips. Validated logical consistency across AVX2, NEON, and Pure C++ Scalar fallback paths.
…g QK layout to 128 - Standardized `QK_I2_S` to 128 for `__ARM_NEON` to match the x86 GGUF packing standard. - Fixed memory misalignment in `quantize_i2_s` by updating the packing stride to 32. - Refactored `ggml_vec_dot_i2_i8_s` NEON kernels (1x1, 1xN, Nx1) to use a dynamic block-level loop (`nb = n / QK`) instead of hardcoded 64-stride loop unrolling. - Aligned interleaved memory fetching (`vld1q_s8`) with the AVX2 logic. - Upgraded accumulator horizontal sum to `vaddlvq_s32` (64-bit) to prevent potential 32-bit integer overflow in extended context scenarios. Tested on Exynos 1380 (Android PRoot) with `-t 8`. Output generation is now 100% stable without word salad.
|
@microsoft-github-policy-service agree |
|
Hi @betovildoza, Thank you for reviewing! After reading through Issue #468, I can confidently say that this PR is the exact antidote for Bug 3 (GGGG garbage) and Bug 4 (Word Salad) you mentioned. I noticed you suspected act_sums or scale offsets for Bug 4. However, the true root cause was much deeper in the memory layout phase. By synchronizing the NEON packing logic to a 32-stride and replacing the hardcoded 64-block loop with a dynamic 128-block logic, both the GGGG issue and the Word Salad are completely resolved. Also, changing the horizontal sum to vaddlvq_s32 prevented the accumulator overflow. It's running perfectly on my Exynos environment now. I'm highly confident this will bring your Oracle Ampere A1 back to life. Let me know if you need any help testing it on your end! |
2 participants
Hello,
The code has been updated to resolve ARM architecture build failures. The entire process of identifying the root causes, applying fixes, and running final tests to ensure stability has been completed.
While Gemini was used for the initial draft, the final solution is based on my own technical validation and experience. For more details on how these ARM-specific issues were tackled, please check my blog post:
https://uno-kim.tistory.com/462
Best regards,
🐛 The Problem
When running models quantized to
i2_s(1.58-bit) on non-Apple ARM devices (e.g., Android devices with Exynos/Snapdragon via Termux/PRoot), the model produces severe "Word Salad" (meaningless token generation).🔍 Root Cause Analysis
The root cause was a memory layout mismatch between the x86 packing phase and the ARM NEON unpacking phase:
QK_I2_Swas hardcoded to64for__ARM_NEONin the macro definition, while the GGUF models packed on x86 strictly enforceQK=128.quantize_i2_sfor NEON used a 16-stride jump, whereas AVX2 uses a 32-stride.ggml_vec_dot_i2_i8_s_*) had hardcoded loop unrolling that assumed a 64-block size, causing out-of-bounds padding reads and complete accumulator corruption.🛠️ Changes Implemented
#define QK_I2_S 128across all architectures.quantize_i2_sfor NEON to use the exact same 32-stride packing layout as AVX2.1x1,1xN, andNx1NEON kernels. Removed the legacy QK=64 loop unrolling and replaced it with a dynamic block loop (nb = n / 128)._mm256_srli_epi16logic.vaddlvq_s32(64-bit) to ensure absolute safety against overflow during massive multi-threaded prompt evaluations.🧪 Testing
-t 8). The DotProd (__ARM_FEATURE_DOTPROD) hardware acceleration and FMA fallbacks both work flawlessly.