Changyu Lee

NLP & Prompt Engineering

Prompt
Search
Summary

Seq2Seq

•
Difference from Sequence Labeling
1.
The length of Y can be different from the length of X
2.
The space of Y is often much larger

Statistic Machine Translation (SMT)

IBM Translation Models

Seq2Seq & Neural Machine Translation
Lecture Summary
•
RNNs are essential for sequence labeling tasks like POS tagging and NER, where output tags depend on context and neighboring predictions.
•
Simple RNNs suffer from vanishing/exploding gradients, lack of future context, and limited memory capacity.
•
LSTMs address gradient problems through cell states and gating mechanisms (input, forget, output gates), enabling long-term dependency learning.
•
Bidirectional RNNs capture both past and future context by processing sequences in both directions.
•
Character-level CNNs combined with BiLSTMs improve representation by capturing morphological features.
•
Structured models like CRFs explicitly model dependencies between output tags and capture global sequence consistency.
•
CRFs use global normalization over entire sequences, while MEMMs use local normalization, making CRFs more effective for sequence labeling.
•
BiLSTM-CNN-CRF architecture combines neural feature extraction with structured prediction for state-of-the-art sequence labeling performance.

Structured Prediction & Sequence Labeling

Structured Prediction

•
Y consists of multiple components Y = {y1, y2, y3}
Recurrent Nerual Networks for Sequence Labeling
Lecture Summary
•
2-layer neural networks consist of a hidden layer (multiple logistic regression classifiers) and a final layer (linear for binary, softmax for multi-class)
•
Non-linear activation functions (sigmoid, tanh, ReLU) are essential for networks to learn complex patterns; without them, stacked layers collapse into one linear equation
•
Deep networks (more layers) generally outperform wide networks (more units per layer) in practice
•
CNNs apply convolutional filters over fixed-size windows, creating learned n-gram features that generalize to unseen word sequences via word embeddings
•
SGD partitions training data into batches; larger batches yield more accurate gradients but slower updates, while smaller batches are faster but noisier
•
Proper weight initialization (He, Xavier, or PyTorch default uniform) prevents symmetry and ensures effective training
•
Learning rate is critical: too high causes divergence, too low leads to slow convergence; learning rate schedules can improve optimization
•
Momentum and adaptive learning rates (e.g., Adam) help SGD overcome local minima and navigate non-convex loss surfaces more effectively

Neural Nets

MLP

•
2-layer NN for Binary Classification
◦
Hidden Layer = A bunch of Logistic regression classifiers
The logistic regression equation is:
P(y=1∣x)=σ(wTx+b)=11+e−(wTx+b)P(y=1|x) = \sigma(w^T x + b) = \frac{1}{1 + e^{-(w^T x + b)}}
where:
▪
σ is the sigmoid function
▪
w is the weight vector
▪
x is the input feature vector
▪
b is the bias term
◦
Final Layer = linear Model
•
What about 2-layer NN for Multi-Class Classification?
◦
Final Layer is changed to Softmax Regression (Only final layer changes when changing to a different task)
For multi-class classification, we use the softmax function to convert the output logits into probability distributions:
P(y=k∣x)=ezk∑j=1KezjP(y=k|x) = \frac{e^{z_k}}{\sum_{j=1}^{K} e^{z_j}}
where:
▪
z_k = w_k^T x + b_k is the logit for class k
▪
K is the total number of classes
▪
The softmax function ensures that all probabilities sum to 1
The predicted class is: ŷ = argmax P(y=k|x)
In Matrix Form, Hidden Layer (Layer 1):
h=σ(W1x+b1)h = \sigma(W_1 x + b_1)
where:
▪
h is the hidden layer activation vector
▪
W_1 is the weight matrix for the first layer
▪
x is the input feature vector
▪
b_1 is the bias vector for the first layer
▪
σ is the activation function (e.g., sigmoid, ReLU)
Output Layer (Layer 2):
z=W2h+b2z = W_2 h + b_2
where:
▪
z is the output logit vector
▪
W_2 is the weight matrix for the second layer
▪
h is the hidden layer activation from Layer 1
▪
b_2 is the bias vector for the second layer
Final Prediction:
For binary classification:
y^=σ(z)=11+e−z\hat{y} = \sigma(z) = \frac{1}{1 + e^{-z}}
For multi-class classification:
P(y=k∣x)=softmax(z)k=ezk∑j=1KezjP(y=k|x) = \text{softmax}(z)_k = \frac{e^{z_k}}{\sum_{j=1}^{K} e^{z_j}}
Neural Nets
Lecture Summary
•
Word vectors represent words as numerical vectors that capture semantic meaning
•
Similar words have similar vectors based on the distributional hypothesis—words appearing in similar contexts have similar meanings
•
Word2Vec learns dense word vectors by predicting co-occurrence patterns within sliding windows, using positive examples (real co-occurrences) and negative samples
•
The model optimizes both word vectors (features) and context vectors (classifier weights) simultaneously through gradient descent
•
Word vectors enable solving analogies through vector arithmetic (e.g., king - man + woman ≈ queen) and measuring similarity via cosine similarity
•
For text classification, variable-length documents can be converted to fixed-size vectors using mean pooling, though this loses word order and phrase information
•
Key limitations include inability to handle polysemy (multiple word meanings) and word sense disambiguation without broader sentence context
•
Word vectors can reflect biases present in training data, capturing historical correlations rather than purely semantic relationships

Word Vectors

Another View: A Word Vector Layer as a Function

Word Vector
Lecture Summary
Summary

Pytorch Basics

Transformer (Part 1)

Transformer
Lecture Summary
•
NLP systems consist of feature extraction and machine learning components, with deep learning replacing manual feature engineering
•
Text classification can be approached through generative methods (Naive Bayes) or discriminative methods (Logistic Regression, SVM)
•
Naive Bayes assumes conditional independence of words given labels and uses Bayes Rule to compute P(y|x)
•
Laplace smoothing addresses zero probability problems by adding pseudocounts; log space prevents numerical underflow
•
Logistic Regression learns weights for features to compute scores, converting them to probabilities via sigmoid function
•
Training optimizes parameters by maximizing log-likelihood (minimizing cross-entropy loss) using gradient descent
•
Feature extraction methods include unigram counts, which are simple but lose word order information
•
SVM provides an alternative discriminative approach for training linear classifiers

NLP System Pipeline

Feature Extraction

NLP Pipelines & Text Classification Methods
Lecture Summary
Summary

Pytorch Basics

Transformer (Part 1)

Pytorch Basics
Lecture Summary

Project Overview

This project implements a sentiment classification pipeline using Amazon reviews. The task is binary classification (positive vs. negative) based on star ratings. I built a full pipeline from dataset preparation and text cleaning to feature extraction (bigram count vectors) and model training/evaluation using four classical classifiers: Perceptron, SVM, Logistic Regression, and Multinomial Naive Bayes.
Key design choices:
•
Labels: Positive = 4–5 stars (y = 1), Negative = 1–2 stars (y = 0), Neutral (3 stars) ignored.
•
Sampling: 100,000 positive + 100,000 negative reviews, shuffled (total 200,000).
•
Text processing: lowercasing, URL/HTML removal, non-alphabetic removal, contraction expansion, stopword removal, lemmatization.
•
Feature representation: bigram frequency vectors stored as a sparse matrix (CSR) to handle high dimensionality.
•
Evaluation: Accuracy, Precision, Recall, F1 for both training and testing splits (80/20).

Results

3 Analysis of Dataset

Relabeling and Sampling

•
Totally, from the dataset,
Average length before cleaning: 317.2775
Average length after cleaning: 301.1736
•
How I relabeled and sampled
# ensure star ratings are numeric df['star_rating'] = pd.to_numeric(df['star_rating'], errors='coerce') # drop rows with missing values df = df.dropna(subset=['review_body', 'star_rating']) # create 2 classes: positive (4-5), negative (1-2) - ignore class 0 df['label'] = df['star_rating'].apply( lambda r: '1' if r > 3 else ('0' if r <= 2 else np.nan) ) # print number of samples before cleaning print(f"Positive reviews:", df['label'].value_counts().get('1', 0)) print(f"Negative reviews:", df['label'].value_counts().get('0', 0)) print(f"Neutral reviews:", df['label'].isna().sum()) # sample 100,000 from positive and 100,000 from negative pos_sample = df[df['label'] == '1'].sample(n=100000, random_state=42) neg_sample = df[df['label'] == '0'].sample(n=100000, random_state=42) # combine and shuffle df = pd.concat([pos_sample, neg_sample]).sample(frac=1, random_state=42).reset_index(drop=True)
Python
복사
◦
Created binary labels: positive (1) for 4–5 stars, negative (0) for 1–2 stars, ignoring neutral (3 stars)
◦
Sampled 100,000 reviews from each class to create a balanced dataset of 200,000 total reviews
◦
Used random_state=42 for reproducibility
◦
Shuffled the combined dataset to prevent any ordering bias during training
◦
This balanced sampling ensures that the model doesn't develop a bias toward the majority class and allows for fair evaluation of classification performance

Pre-processing

A Report of Hands-on Project of Text Representations and the Use of Text Classification for Sentiment Analysis
Project
Lecture Summary

Ollama 특정 GPU만 사용하게 하기

•
기본 원리는 간단하다. ollama 서비스 실행 시 CUDA_VISIBLE_DEVICES를 명시해주면 된다.
•
이 글은 linux 기반의 서버를 기준으로 설명한다.
1.
시스템 서비스로 Ollama 관리:
•
터미널을 열고 다음 명령어를 입력하여 Ollama 서비스를 재시작하고,
bash 코드 복사 sudo systemctl restart ollama
Shell
복사
•
이 명령어는 Ollama 서비스를 중지한 후 다시 시작한다.
2.
수동으로 Ollama 서버 실행:
•
만약 시스템 서비스로 관리하지 않는 경우, 터미널에서 다음 명령어를 사용하여 서버를 종료하고 다시 실행할 수 있음
bash 코드 복사 # 서버 종료 pkill ollama # 서버 실행 ollama serve
Shell
복사

이후, CUDA_VISIBLE_DEVICES 옵션 설정

Ollama 서버를 특정 GPU에서 실행하려면 CUDA_VISIBLE_DEVICES 환경 변수를 설정하여 사용할 GPU를 지정할 수 있다. 이렇게 하면 Ollama가 지정된 GPU에서만 실행된다.
•
CUDA_VISIBLE_DEVICES 설정:
bash 코드 복사 sudo nano /etc/systemd/system/ollama.service
Shell
복사
파일이 열리면 [Service] 섹션에 다음 줄을 추가한다
Environment="CUDA_VISIBLE_DEVICES=1,2"
Makefile
복사
•
서비스 파일 리로드 및 Ollama 서버 재시작: 변경 사항을 저장하고 나서, 다음 명령어를 입력하여 시스템 데몬을 리로드하고 Ollama 서버를 재시작.
bash 코드 복사 sudo systemctl daemon-reload sudo systemctl start ollama
Shell
복사
nano 편집기에서 파일을 저장하면 끝
Ollama 특정 GPU만 사용하기 설정 방법
Log
RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval
Retrieval-augmented language models can better adapt to changes in world state and incorporate long-tail knowledge. However, most existing methods retrieve only short contiguous chunks from a retrieval corpus, limiting holistic understanding of the overall document context. We introduce the novel approach of recursively embedding, clustering, and summarizing chunks of text, constructing a tree with differing levels of summarization from the bottom up. At inference time, our RAPTOR model retrieves from this tree, integrating information across lengthy documents at different levels of abstraction. Controlled experiments show that retrieval with recursive summaries offers significant improvements over traditional retrieval-augmented LMs on several tasks. On question-answering tasks that involve complex, multi-step reasoning, we show state-of-the-art results; for example, by coupling RAPTOR retrieval with the use of GPT-4, we can improve the best performance on the QuALITY benchmark by 20% in absolute accuracy.
https://arxiv.org/html/2401.18059v1

Abstract

•
Problems of existing retrieval-augmented approaches
◦
most existing methods retrieve only a few short, contiguous text chunks
▪
Authors designed an indexing and retrieval system that uses a tree structure to capture both high-level and low-level details about a text.
•
selecting the most relevant information for knowledge-intensive tasks is still crucial.

Background

•
Recursive Summarization as Context
◦
The recursive-abstractive summarization model by Wu et al. (2021) employs task decomposition to summarize smaller text chunks, which are later integrated to form summaries of larger sections.
◦
While this method is effective for capturing broader themes, it can miss granular details. LlamaIndex (Liu, 2022) mitigates this issue by similarly summarizing adjacent text chunks but also retaining intermediate nodes thus storing varying levels of detail, keeping granular details. 
◦
may still overlook distant interdependencies within the text
•
 long texts often present subtopics and hierarchical structures (Cao & Wang, 2022; Dong et al., 2023b)

Method

Tree Contrection Process

•
RAPTOR recursively clusters chunks of text based on their vector embeddings and generates text summaries of those clusters, constructing a tree from the bottom up. Nodes clustered together are siblings; a parent node contains the text summary of that cluster.
The clustering approach in tree construction includes a few interesting ideas.
GMM (Gaussian Mixture Model)
•
Model the distribution of data points across different clusters
RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval
Paper

How Does Vector Similarity Work

Summary by AI:
Vector similarity search converts data items into vectors in a high-dimensional space, with the distance between vectors indicating similarity. Techniques like Word Embeddings and Convolutional Neural Networks are used for text and non-text data respectively. Similarity is calculated using metrics like Euclidean Distance or Cosine Similarity, with Nearest Neighbor and Approximate Nearest Neighbor algorithms used for large datasets. Indexing structures and algorithms are used for efficient search. Applications of vector similarity search include recommendation systems, information retrieval, natural language processing, computer vision, bioinformatics, fraud detection, and customer support.
Vector Similarity Search works by converting data items (such as text documents, images, or audio clips) into vectors in a high-dimensional space. Each vector represents the features or characteristics of its corresponding item in a way that captures semantic or contextual meaning. The process of determining how "similar" two items are involves measuring the distance between their vectors in this space. Let's break down the key components and steps involved in vector similarity search:

1. Vector Representation

•
Text Data: Techniques like Word Embeddings (Word2Vec, GloVe), Sentence Embeddings (BERT, Sentence-BERT), or Document Embeddings are used to convert words, sentences, or entire documents into vectors. These embeddings capture semantic meaning, where similar words or sentences have vectors that are close to each other in the vector space.
•
Non-Text Data: For images, audio, and other non-textual data, deep learning models (such as Convolutional Neural Networks for images) are used to extract features and encode them as vectors.
•
Vector Embedding
Vector Embedding Example
Vector embedding involves transforming high-dimensional data (like text, images, or sounds) into vectors of fixed size in a low-dimensional space.
Image Embedding Process in Computer Vision
Unlike raw data, which can be complex, unstructured, and challenging to work with directly, these vectors encapsulate the essential features and relationships of the data in a form that computers can efficiently process and analyze.
◦
Dimensionality Reduction: Embeddings convert data from high-dimensional (often sparse) spaces to lower-dimensional (dense) vector spaces. This process helps in revealing and preserving the intrinsic patterns, relationships, and structures within the data, despite the reduction in dimensions.
◦
Semantic Meaning: In the context of text data, embeddings are designed so that semantically similar words or phrases are mapped to points that are close to each other in the vector space. This property enables algorithms to understand and work with the meaning of words or sentences.
◦
Generalization: Embeddings capture the underlying essence of data items, allowing models to generalize better to unseen data by understanding its position and relation in the vector space.

2. Similarity Metrics

What is Vector Similarity Search? : 2. How Does Vector Similarity Work
Theory

Vector Similarity Search

Vector Similarity Search refers to the process of finding vectors (arrays of numbers) that are closest to a given query vector in a vector space.
This process is fundamental in various applications such as recommendation systems, search engines, image retrieval, and natural language processing tasks. The idea is to represent data (like text, images, or user preferences) as vectors in a high-dimensional space where the similarity between these data points can be calculated based on their vector representations.
The similarity between vectors is often measured using metrics such as Euclidean distance, cosine similarity, or Manhattan distance, among others. The choice of metric depends on the specific application and the nature of the data.
•
Euclidean Distance measures the straight-line distance between two points in Euclidean space, which is suitable for numerical data.
•
Cosine Similarity measures the cosine of the angle between two vectors, which is particularly useful for text data represented in vector space models (like TF-IDF or word embeddings) because it captures the orientation of the vectors and not the magnitude, making it sensitive to the similarity in direction of the vectors.
•
Manhattan Distance (or L1 distance) measures the sum of the absolute differences of their coordinates, which can be useful for certain types of data distributions or applications where the grid-like distance matters.
Vector Similarity Search can be computationally expensive, especially as the size of the dataset grows. To address this, various indexing and search optimization techniques such as k-d trees, locality-sensitive hashing (LSH), and approximate nearest neighbor (ANN) algorithms are employed to speed up the search process without significantly compromising accuracy.
These techniques enable efficient retrieval of high-dimensional data by approximating the nearest neighbors to a query vector, making Vector Similarity Search a critical component in handling and making sense of large datasets in machine learning and data science.

Prob. to Solve

What is Vector Similarity Search? : 1. Introduction

Concept of Autonomous Agent Sys. [1]

•
Planning
◦
Subgoal and decomposition: The agent breaks down large tasks into smaller, manageable subgoals, enabling efficient handling of complex tasks.
◦
Reflection and refinement/critic: The agent can do self-criticism and self-reflection over past actions, learn from mistakes and refine them for future steps, thereby improving the quality of final results.
LLM Agent Overview

Pydantic

Pydantic은 Python에서 데이터 유효성 검사와 구조화를 위한 라이브러리입니다. Pydantic을 사용하면 데이터 모델을 정의하고 입력 데이터를 검증할 수 있습니다. 이를 통해 코드의 안정성을 높이고 유지 보수를 용이하게 할 수 있습니다.
Pydantic은 간단하고 직관적인 구문을 제공하여 사용하기 쉽습니다. 데이터 모델은 일반적으로 Python 클래스로 정의되며 필드와 필드 유효성 검사 규칙을 포함합니다. 이를 통해 입력 데이터가 예상한 형식과 구조를 따르는지 확인할 수 있습니다.
Pydantic은 다양한 유효성 검사 규칙을 지원하며, 예를 들어 필드의 타입, 최소 및 최대 값, 길이 등을 검사할 수 있습니다. 또한, Pydantic은 JSON 직렬화 및 역직렬화, 데이터 파싱 및 변환 등의 유용한 기능도 제공합니다.
Pydantic은 웹 애플리케이션, 데이터베이스, API 등 다양한 영역에서 사용될 수 있습니다. 문서에서 제공하는 많은 예제와 자세한 설명을 통해 Pydantic의 다양한 기능과 활용 방법을 학습할 수 있습니다.
더 자세한 내용은 Pydantic 문서를 참고할 수 있습니다.

기본 구조

Model

Models are simply classes which inherit from pydantic.BaseModel and define fields as annotated attributes.

Field

The Field function is used to customize and add metadata to fields of models.
Langchain: Pydantic
Basic
Home
Projects
Blog
Contact