Decoding Language Switching in AI Assistants: A Step-by-Step Analysis Guide

By ● min read

Introduction

Have you ever been typing in Chinese to your AI coding assistant, only to have it start replying in Korean? This puzzling behavior isn't random—it stems from how embeddings work under the hood. When code vocabulary mixes with natural language, the assistant's internal representation can drift, leading to unexpected language switches. In this guide, you'll learn how to investigate this phenomenon step by step, from setting up your environment to analyzing embedding spaces.

Decoding Language Switching in AI Assistants: A Step-by-Step Analysis Guide — Source: towardsdatascience.com

What You Need

A computer with Python 3.8+ installed
Access to an AI coding assistant API (e.g., OpenAI, Anthropic, or a local model)
Python libraries: numpy, scikit-learn, matplotlib, transformers (Hugging Face)
Sample prompts in Chinese and English, including code snippets
Basic knowledge of embedding vectors and cosine similarity
A text editor or Jupyter Notebook

Step-by-Step Guide

Step 1: Choose Your Testing Prompts

Select a set of prompts that mirror real-world usage. You'll want:

Pure Chinese prompts (no code): e.g., "写一个函数来计算斐波那契数列" (Write a function to compute Fibonacci)
Pure English prompts (no code): e.g., "Write a function to compute the Fibonacci sequence"
Mixed prompts: Chinese with embedded code keywords like def, return, if

Record the assistant's responses. Note any language shifts.

Step 2: Extract Embeddings from the Assistant

Most coding assistants allow you to access internal embeddings or you can use a separate embedding model. For example, using OpenAI's text-embedding-ada-002 or Hugging Face's sentence-transformers:

from sentence_transformers import SentenceTransformer
model = SentenceTransformer('all-MiniLM-L6-v2')
embedding = model.encode("你的提示")

Create embeddings for both your prompts and the assistant's responses.

Step 3: Analyze Embedding Similarity

Use cosine similarity to compare embeddings. The unexpected language switch often occurs when code vocabulary pulls the Chinese prompt closer to Korean-language embeddings in the model's space.

from sklearn.metrics.pairwise import cosine_similarity
import numpy as np

# Example: compare Chinese prompt with its response
prompt_emb = model.encode("写一个函数")  # Chinese
response_emb = model.encode("함수를 작성하세요")  # Korean
sim = cosine_similarity([prompt_emb], [response_emb])
print(sim)

Key insight: High similarity between a Chinese+code prompt and a Korean response suggests the code vocabulary has bridged the language gap.

Step 4: Visualize the Embedding Space

Reduce dimensionality using PCA or t-SNE to plot embeddings. Color-code by language (Chinese, English, Korean). You'll often see a cluster where code-related terms mix languages.

from sklearn.decomposition import PCA
import matplotlib.pyplot as plt

# Assume you have a list of embeddings and labels
pca = PCA(n_components=2)
reduced = pca.fit_transform(all_embeddings)

for lang, color in [('Chinese', 'red'), ('English', 'blue'), ('Korean', 'green')]:
    idx = [i for i, l in enumerate(labels) if l == lang]
    plt.scatter(reduced[idx,0], reduced[idx,1], c=color, label=lang)
plt.legend()
plt.show()

Step 5: Isolate Code Vocabulary Effect

Create a controlled test: Take a pure Chinese prompt and a pure English prompt about the same task. Then add identical code keywords (like for, while, import) to both. Compare the embeddings before and after adding code. If the Chinese+code embedding moves toward the Korean region more than the English+code does, you've found the culprit.

Step 6: Document and Repeat

Run your tests multiple times with different models (GPT-3.5, GPT-4, Claude, etc.). Note that each model's training data and tokenizer affect how code vocabulary reshapes language. Some models might switch to Japanese or other languages, not just Korean.

Tips & Best Practices

Use consistent tokenization: Different tokenizers can break Chinese and Korean characters differently. Check token counts to ensure fair comparisons.
Watch for false positives: A Korean response might sometimes be due to training data imbalance, not just code vocabulary. Cross-check with multiple prompts.
Leverage existing research: Search for papers on multilingual embedding shifts in code assistants—this guide builds on that work.
Adapt for prevention: Once you understand the pattern, you can ask the assistant to explicitly state its language, or use system prompts like "Always respond in the language of the user's input."
Explore open-source models: If your assistant is a black box, test with alternatives like CodeBERT or StarCoder for deeper embedding access.

By following these steps, you'll not only decode why your assistant switched to Korean—you'll gain a practical method for analyzing any language drift in AI systems. Happy embedding!

Tags: