> | Resource | Info |
> | :---        |:--- |
> | Paper       | https://arxiv.org/abs/2109.05687 |
> | Code & Data | / |
> | Public      | EMNLP |
> | Date        | 2024.12.14 |


# Summary Overview

最近预处理的语言模型从数百万到数十亿个参数延伸。因此，需要在各种下游任务中进行有限的训练语料库微调一个非常大的审计模型。在本文中，我们提出了一种直接而有效的微调技术，即$Child-Tuning$，该技术通过策略性地掩盖了反向传播的过程，从而更新了 PLM 的部分参数（称为child network）通过策略性地掩盖了 non-child network 的梯度。

# Main Content

作者提出了一种有效的微调技术，即 $Child-Tuning$，它可以直接通过在反向传播的过程中策略性的掩盖 non-child network 的梯度更新 child network 的参数。

![image.png](/static/img/042269609563bb45851085f4c24f066f.image.png)

Child-tuning 进一步根据是否与任务有关分为 $Child-Tuning_F$ 和 $Child-Tuning_D$。$Child-Tuning_F$在没有任务数据的情况下通过Bernoulli分布选择child network，它将噪声引入梯度中，因此防止了模型在小型数据集上的过拟合，具有更好的泛化性。$Child-Tuning_D$利用下游任务数据来检测与任务相关的参数。

原本的模型参数更新：

$$
\mathbf{w}_{t+1} = \mathbf{w}_t - \eta \frac{\partial \mathcal{L}(\mathbf{w}_t)}{\partial \mathbf{w}_t}
$$

$Child-Tuning$

$$
M_t^{(i)} =
\begin{cases}
1, & \mathbf{w}_t^{(i)} \in \mathcal{C}_t \\
0, & \mathbf{w}_t^{(i)} \notin \mathcal{C}_t
\end{cases}
$$

$$
\mathbf{w}_{t+1} = \mathbf{w}_t - \eta \frac{\partial \mathcal{L}(\mathbf{w}_t)}{\partial \mathbf{w}_t} \odot M_t
$$

**Task-Free Variant:** $Child-Tuning_F$

具体而言，$Child-Tuning_F$在从伯诺利分布中绘制的t-th迭代中生成0-1蒙版$M_t$，其概率$p_F$：

$$
M_t \sim \text{Bernoulli}(p_F)
$$

$p_F$越高，子网络越大，因此更新更多的参数。

**Task-Driven Variant:** $Child-Tuning_D$

具体而言，我们采用Fisher信息估计，以找到特定下游任务的参数的高度相关子集。 Fisher信息是提供估计随机变量有关分布参数的信息的好方法。

$$
\mathbf{F}(\mathbf{w}) = \mathbb{E} \left[ 
\left( \frac{\partial \log p(\mathbf{y} \mid \mathbf{x}; \mathbf{w})}{\partial \mathbf{w}} \right) 
\left( \frac{\partial \log p(\mathbf{y} \mid \mathbf{x}; \mathbf{w})}{\partial \mathbf{w}} \right)^\top
\right]
$$

正式地，我们为$i$-th参数提供了Fisher信息，如下所示：

$$
\mathbf{F}^{(i)}(\mathbf{w}) = \frac{1}{|D|} \sum_{j=1}^{|D|} 
\left( 
\frac{\partial \log p(y_j \mid \mathbf{x}_j; \mathbf{w})}{\partial \mathbf{w}^{(i)}} 
\right)^2
$$

# Experiments

## Datasets

- GLUE benchmark
- NLI datasets

## Results

![image.png](/static/img/e862683a0ce2f53b2ba01076d27ec011.image.png)

![image.png](/static/img/988726ae257f838fa885531ce65e4b13.image.png)

![image.png](/static/img/68d90c42e3a2a6de30a7257f40d2af81.image.png)


![image.png](/static/img/a742e72be385fceecf6b7250e1a9d430.image.png)

**Results in Low-resource Scenarios**

由于过度拟合的风险升高，因此在极小的数据集上进行微调的大型模型可能非常具有挑战。因此，在本节中，我们仅通过几个培训示例探讨了$Child-Tuning$的效果。为此，我们将所有数据集都置于GLUE中为1K培训示例，并在其中进行微调。

![image.png](/static/img/9f6fa718f8e8749218173523e964a239.image.png)

**Is the Task-Driven Child Network Really that Important to the Target Task?**

![image.png](/static/img/cd5d7a17d4a3a33ba61e04bf0f5fba17.image.png)

**What is the Relationship among Child Networks for Different Tasks?**

随着任务驱动的child network与任务相关，我们进一步探讨了child network之间的关系以进行不同的任务。为此，我们可视化不同任务驱动的child network之间的重叠率，在那里我们使用jaccard相似性系数$\frac{|C^i \cap C^j|}{|C^i \cup C^j|}$，计算任务$i$和$j$之间的重叠率。

![image.png](/static/img/7b25581377ed32b93e8340409e132cd8.image.png)

## Case Study

# 🤖ChatGPT

:::info{title="ChapGPT"}
1. 论文的创新之处与独特性：
   - **创新点1：提出CHILD-TUNING方法**  
     论文提出了一种新的微调技术CHILD-TUNING，通过在反向传播过程中对非子网络的梯度进行屏蔽，仅更新子网络的参数。这种方法避免了对整个模型参数的过度调整，同时保留了预训练模型的知识，尤其在小数据集上表现出色。
   - **创新点2：任务无关与任务驱动两种子网络选择策略**  
     提出两种子网络选择策略：任务无关的CHILD-TUNINGF（基于伯努利分布随机选择子网络）和任务驱动的CHILD-TUNINGD（基于Fisher信息矩阵选择最相关的子网络）。任务驱动方法更具针对性，能够有效降低模型假设空间。
   - **创新点3：广泛的实验验证与性能提升**  
     通过在GLUE基准测试和跨任务迁移实验中验证，CHILD-TUNING在多种预训练语言模型（如BERT、RoBERTa、ELECTRA等）上均表现出显著的性能提升，并在任务泛化和领域迁移上优于其他微调方法。

2. 论文中存在的问题及改进建议：
   - **问题1：任务驱动子网络选择的计算开销较高**  
     任务驱动的CHILD-TUNINGD需要在微调前计算Fisher信息矩阵，这在大规模预训练模型上可能导致较高的计算成本。
     - **改进建议**：探索更高效的子网络选择方法，例如通过稀疏正则化或梯度重要性评分动态选择子网络。
   - **问题2：对不同任务的子网络重叠性缺乏深入分析**  
     论文中虽然讨论了子网络的重叠性，但未进一步分析这种重叠性对模型性能和迁移能力的具体影响。
     - **改进建议**：引入更细粒度的分析，例如子网络在不同任务上的共享参数比例对性能的具体贡献。
   - **问题3：对低资源场景的优化仍有改进空间**  
     虽然CHILD-TUNING在低资源场景下表现较好，但其在极端低资源数据下（例如仅百条样本）仍可能面临过拟合。
     - **改进建议**：结合数据增强技术或对抗训练方法，进一步提升CHILD-TUNING在极低资源场景下的鲁棒性。

3. 基于论文的内容和研究结果，提出的创新点或研究路径：
   - **创新点1：动态子网络选择策略**  
     探索一种动态子网络选择方法，在训练过程中根据梯度变化动态调整子网络，从而平衡模型的适应性与泛化能力。
   - **创新点2：跨任务共享子网络的优化**  
     针对不同任务的子网络重叠性，研究如何设计一个共享子网络框架，使得多个任务能够共享部分参数，同时保留任务特定的能力。
   - **创新点3：基于CHILD-TUNING的多模态模型微调**  
     将CHILD-TUNING扩展到多模态模型（如视觉-语言模型）中，研究如何在多模态场景下选择子网络以提升微调效率和性能。

4. 为新的研究路径制定的研究方案：
   - **研究路径1：动态子网络选择策略**
     - **研究方法**：基于梯度重要性评分动态调整子网络。在每个训练迭代中，计算各层参数的梯度方差或梯度幅值，选择变化最显著的参数作为子网络的一部分。
     - **研究步骤**：
       1. 初始化模型参数，随机选择初始子网络。
       2. 在每个训练迭代中，计算梯度变化并更新子网络。
       3. 比较动态子网络选择与固定子网络选择的性能差异。
     - **期望成果**：验证动态子网络选择策略在不同任务上的性能提升，尤其是在数据稀缺场景中的优势。
   - **研究路径2：跨任务共享子网络的优化**
     - **研究方法**：设计一个多任务共享框架，利用共享子网络处理不同任务，同时通过任务特定模块捕获任务独有特性。
     - **研究步骤**：
       1. 构建一个多任务学习框架，定义共享子网络和任务特定子网络。
       2. 使用CHILD-TUNING的任务驱动方法选择共享子网络。
       3. 在GLUE基准测试和跨任务迁移任务上验证框架性能。
     - **期望成果**：证明共享子网络能够提升多任务学习的效率和性能，同时减少模型参数量。
   - **研究路径3：基于CHILD-TUNING的多模态模型微调**
     - **研究方法**：将CHILD-TUNING方法扩展到多模态模型（如CLIP、BLIP等），研究如何在视觉和语言特征之间选择子网络。
     - **研究步骤**：
       1. 对多模态模型的视觉和语言模块分别应用CHILD-TUNING，探索任务驱动的子网络选择。
       2. 比较多模态模型在单模态任务和多模态任务上的性能。
       3. 研究子网络选择对多模态对齐和表示学习的影响。
     - **期望成果**：验证CHILD-TUNING在多模态场景中的有效性，提出一种高效的多模态微调方法。

:::

# Others

[EMNLP-2021] Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuning

> | Resource | Info |
> | :---        |:--- |
> | Paper       | https://arxiv.org/abs/2406.18181 |
> | Code & Data | / |
> | Public      | ASE |
> | Date        | 2024.12.13 |


# Summary Overview

在这篇文章中，作者根据17个 Java 项目，使用了5个开源不同结构和参数规模的 LLM，对于单元测试 (UT) 的能力的探究，进行了全面的评估。是一篇 empirical study。

# Main Content

单元测试的重点是验证每个单独的程序组件（例如每种方法）的功能，以确保其按预期工作。

**Challenges:**
1. Generating syntactically correct tests. 单元测试生成可以作为代码生成的子问题。因此也会有生成的代码需要符合语言语法规则的问题。
2. Generating effective tests. 高质量的单元测试应该彻底探讨目标程序组件的行为。该方法需要了解代码的意图和结构的方法，从而生成具有高测试覆盖范围的缺陷检测能力的测试。
3. Generating maintainable tests. 在软件领域，人们更加喜欢易于阅读和更改的单元测试，这需要在之后的测试中充分体现。

在这项工作中，我们进行了首次广泛的研究，以实现上述目标，该目标基于缺陷4J 2.0基准的17个Java项目。

**Findings:**
1. Prompt 设计（描述样式和所选代码功能）对于LLM在单元测试生成中的有效性至关重要。建议考虑LLMS的代码理解能力和生成单元测试的剩余空间，建议将描述样式与培训数据保持一致，并选择代码功能。值得注意的是，包括目标类别中定义的其他方法（焦点方法除外）会对基于LLM的单位测试生成的总体有效性产生负面影响。
2. 从开源LLM中得出的结论在其他任务中不一定会推广到单位测试生成，包括研究的LLMS之间的优势关系。但是，所有研究的LLM，包括最先进的GPT-4，都在测试覆盖范围内表现不佳。这主要是由于LLMS幻觉的结果是LLMS产生的大量句法无效的单元测试。因此，需要有效的解决方案，例如，设计后处理规则来解决常见的句法问题。
3. 尽管它们在其他任务中有效，但直接适应单位测试生成的思想链（COT）和检索增强产生（RAG）方法并不能提高效力，甚至可能在某些情况下可以降低其效率。 COT主要受LLMS的代码理解能力的限制，而RAG受到检索到的单位测试之间的显着差距和LLMS在生成时出色的差距。需要在单位测试生成中使用ICL方法的特殊设计。
4. LLM生成的单位测试的缺陷检测能力受到限制，这主要是由于其有效性低。尽管有效的单位测试是由LLMS生成许多缺陷的，但仍未发现大量缺陷，主要是因为测试未能产生触发这些缺陷所需的特定输入。因此，为生成的单位测试中的输入设计有效的突变策略可以进一步提高缺陷检测效率。

**Contributions:**
1. 我们进行了第一项经验研究，以评估单位测试生成中的开源LLM。我们从众所周知的缺陷4J基准中研究了17个Java项目，研究了五个强大的代码LLM，范围从7B到34B参数。我们的实验需要大约3,000个NVIDIA A100 GPU小时，这突显了我们研究的规模和强度，以推动基于LLMB的单元测试生成。
2. 我们从我们的研究问题定义的四个方面评估了基于LLM的单位测试生成，包括及时设计的影响，ICL方法的影响，开源LLMS之间的比较，商业GPT4和传统的EvoSuite以及在各种指标的条款（包括句法有效性，测试覆盖范围和缺陷检测）。
3. 我们从我们的广泛研究中总结了9个主要发现，并为指导基于LLM的单位测试生成的未来研究和实践的使用带来了一系列含义。

**Research Questions:**
1. How does prompt design affect the effectiveness of LLMs in unit test generation?
2. How do open-source LLMs perform in unit test generation compared to GPT-4 and Evosuite?
3. How do in-context learning methods affect the effectiveness of LLM-based unit test generation?
4. How effective are the unit tests generated by LLMs in terms of defect detection?

**Prompt Design:**
- Description Style:
    - natural language description
    - code language description
- Code Features:
    - *Focal Method* is the target method to be tested. The code features extracted from it include the method body ($FM_b$) and its parameters ($FM_p$).
    - *Focal Class* is the class containing the focal method. The code features extracted from it include its constructor ($FC_c$), the fields defined within the class ($FC_f$), and other methods defined within the class ($FC_m$).
    - *Related Classes* are the classes including the constructors of the parameters in the focal method (except the focal class). The code feature extracted from it is these constructors ($RC_c$). Among them, $FM_b$ is the basic content in the prompt for LLM-based unit test generation.

**Implications:**
1. Tuning prompt design (including both description styles and code features) for a given LLM is important.
2. Removing useless information from code features helps balance prompt content and space for generating unit tests.
3. Emperically selecting a proper LLM is necessary, instead of relying on the best LLM according to experience in other tasks.
4. Refining the use of ICL methods specific to unit test generation is essential.
5. Designing post-processing strategies to fix invalid unit test can help mitigate the influence of LLM's halolucination.
6. Desiging mutation strategies for the inputs specified in LLM-generated unit tests is helpful in improving defect detection ability.
7. Supervised fine-tuning (SFT) of open-source LLMs by incorporating unit test generation data may fundamentally improve effectiveness.

## Related Work

**Empirical Study on evaluating LLMs in Code-Related Tasks:**

随着LLM的快速发展，有多项研究研究了LLM在与代码相关的任务中的有效性[12，17，22，26，45，46，46，50，52-55，60，60，70，76，85]。特别是，有一些关于基于LLM的单位测试生成的经验研究[14、65、68、88、92]。例如，Yuan等人。 [92]根据单位测试生成中生成的单元测试的正确性，足够性，可读性和可用性，研究了CHATGPT的有效性。 Siddiq等。 [68]研究了GPT-3.5和Codex在单位测试生成中的有效性。 Schafer等。 [65]调查了GPT-3.5在生成JavaScript语言的单元测试中的有效性。但是，这些现有研究主要依赖于基于封闭源LLM的固定提示策略，忽略了先进的开源LLMS的功能和各种提示因素（例如及时设计和ICL方法）。在我们的研究中，我们专注于开源LLM，并广泛考虑了这些因素，并进行了首次研究以证明其有效性。我们的发现为基于LLM的单位测试生成提供了一系列可行的建议。此外，还有一些关于评估LLM的经验研究，以解决其他与代码相关的任务。例如，杜等人。 [17]进行了实验，以研究LLM在类级代码生成中的有效性。 Nam等。 [54]比较了使用LLMS时的代码理解能力来帮助开发人员了解该项目。 Sun等。 [70]进行了实验，以研究CHATGPT在代码摘要中的有效性。 Facundo等。 [53]总结了三种类型的甲壳和威胁，这是由使用LLMS自动生成甲壳引起的。研究人员[31，85]对LLM进行了实验，以研究其在自动化程序维修中的有效性。 Niu等。 [55]对六个与代码相关的任务中预培训模型的有效性进行了全面调查。与它们不同，我们的工作针对单元测试生成的任务。

# Experiments

## Metrics

- Compilation Success Rate (CSR)
- Line Coverage ($Cov_L$)
- Branch Coverage ($Cov_B$)
- the Number of Detected Defects (NDD)

## Results

### Influence of Prompt Design

**Influence of Description Style.**

为了评估NL和CL之间的差异，我们进行了Wilcoxon秩和测试，其显着性水平为0.05，并计算了等级 - 生物相关性得分以显示效果大小。效应大小高于0.3通常表明比较组之间有意义的差异。

![image.png](/static/img/b89940e90112087034607d6d776e7ffa.image.png)

**Influence of Code Features**

具体来说，我们为提示设计创建了五个变体，每个变体分别从提示中删除了所有研究的代码功能的代码功能。表2显示了CSR的结果，表3显示了COVL和COVB方面的结果。

![image.png](/static/img/57bae7450d9c873f20233a78e094cd94.image.png)

![image.png](/static/img/5fa1c40a6fe2fa1967e452649e6a8427.image.png)

从表3中，我们发现了一个令人惊讶的现象，即，即使具有统计学意义，几乎所有“ W/O FCM”的情况都会增加。这表明在及时设计中删除FCM有助于提高代码覆盖范围，即使这对生成的单元测试的句法有效性有害。
### Effectiveness Comparsion

![image.png](/static/img/e959e9a6caa9d4bc714f2303901565f6.image.png)

一个可能的原因是，LLMS生成的大量单位测试（范围从34.44％到61.78％）在语法上无效。

### Effectiveness of In-Context Learning

![image.png](/static/img/3d4e83423194d6725ca70fbd2d3ea791.image.png)

### Defect Detection Ability

我们进一步研究了每种基于LLM的单位测试生成技术在缺陷检测方面的有效性，通过在相应的故障版本上执行其生成的单位测试。

![image.png](/static/img/842e0ee50867c82e4543368dccd46a06.image.png)

![image.png](/static/img/7189d945cc4a75b5f84c1e9daf35fdb4.image.png)

# 🤖ChatGPT

:::info{title="ChapGPT"}
1. 论文的创新之处与独特性：
   - **首次系统评估开放源码LLM在单元测试生成中的效果**：论文填补了现有研究的空白，集中研究了五种开放源码大型语言模型（LLMs）在单元测试生成任务中的表现，并与传统工具（如Evosuite）和商业化模型（如GPT-4）进行了对比。这种全面的实验设计（基于Defects4J基准和多种评估指标）具有很高的学术价值。
   - **深入探讨提示设计对LLM性能的影响**：论文通过实验分析了描述风格（自然语言 vs. 代码语言）和代码特性（如目标方法、类构造器等）对单元测试生成效果的影响，提出了优化提示设计的重要性。
   - **首次探讨上下文学习方法（ICL）在单元测试生成中的适配性**：论文评估了链式思维（CoT）和检索增强生成（RAG）方法在此任务中的作用，发现现有方法需要针对单元测试生成任务进行特殊设计。
   - **揭示LLM生成单元测试的局限性**：论文指出了LLM生成的单元测试在语法正确性、测试覆盖率和缺陷检测能力上的不足，分析了主要原因（如幻觉问题和缺乏特定输入）。

2. 论文中存在的问题及改进建议：
   - **对提示优化的探索不够全面**：论文仅进行了逐一去除代码特性的消融实验，而未考虑代码特性组合的可能性，可能遗漏了更优的提示设计方法。
     - 改进建议：设计全面的实验，探索代码特性组合的影响，或利用自动化搜索算法（如遗传算法）优化提示设计。
   - **上下文学习方法的适配性研究较为局限**：论文仅适配了CoT和RAG两种方法，未探索其他可能的上下文学习方法（如自修复、自一致性等）。
     - 改进建议：引入更多上下文学习方法，尤其是结合单元测试生成任务特点开发新的适配策略。
   - **未充分考虑生成后处理策略的潜力**：论文虽提到生成后处理可以缓解语法错误，但未深入研究具体的后处理方法。
     - 改进建议：开发自动化的后处理工具（如基于静态分析的错误修复），并评估其有效性。
   - **实验范围有限**：论文仅使用了Defects4J基准数据集，可能导致结论的普适性受到限制。
     - 改进建议：扩展实验范围，使用其他基准数据集（如GitBug-Java）验证结论。

3. 基于论文的内容和研究结果，提出的创新点或研究路径：
   - **创新点1：基于动态提示优化的单元测试生成**  
     动态调整提示内容，根据生成结果的质量实时优化提示，提升LLM生成单元测试的有效性。
   - **创新点2：结合生成后处理与上下文学习的增强策略**  
     将上下文学习方法与自动化生成后处理结合，开发端到端的单元测试生成框架。
   - **创新点3：专用单元测试生成语料库的构建与微调**  
     构建专门针对单元测试生成的高质量数据集，结合监督微调（SFT）优化开放源码LLM的性能。

4. 为新的研究路径制定的研究方案：
   - **研究路径1：基于动态提示优化的单元测试生成**
     - **研究方法**：
       1. 设计动态提示优化框架，实时分析LLM生成的单元测试质量（如语法正确性、覆盖率等）。
       2. 根据分析结果动态调整提示内容，例如添加缺失的代码特性或修改描述风格。
       3. 评估动态提示优化对生成效果的提升。
     - **步骤**：
       1. 选取开放源码LLM（如CodeLlama）和Defects4J数据集作为实验基础。
       2. 开发提示优化模块，结合静态分析工具自动检测生成结果中的缺陷。
       3. 对比动态优化前后生成的单元测试在语法正确性、覆盖率和缺陷检测能力上的差异。
     - **期望成果**：
       - 提供一种动态提示优化方法，显著提升LLM生成单元测试的质量。
       - 发布开源工具和实验数据，为后续研究提供参考。

   - **研究路径2：结合生成后处理与上下文学习的增强策略**
     - **研究方法**：
       1. 开发自动化生成后处理模块，修复常见语法错误（如未解析符号、参数不匹配等）。
       2. 探索其他上下文学习方法（如自修复、自一致性），并与生成后处理模块结合。
       3. 评估增强策略对生成质量和缺陷检测能力的影响。
     - **步骤**：
       1. 基于现有研究开发后处理规则，结合静态分析工具实现自动修复。
       2. 适配新型上下文学习方法，优化提示内容或生成过程。
       3. 在多个数据集上评估增强策略的效果，验证其通用性。
     - **期望成果**：
       - 提供一个结合上下文学习与生成后处理的综合框架。
       - 显著提高LLM生成单元测试的有效性和可靠性。

   - **研究路径3：专用单元测试生成语料库的构建与微调**
     - **研究方法**：
       1. 收集和标注高质量单元测试数据，构建专用语料库。
       2. 对开放源码LLM进行监督微调，优化其在单元测试生成任务上的性能。
       3. 将微调后的模型与现有模型进行对比，验证其有效性。
     - **步骤**：
       1. 从开源项目中提取单元测试代码，结合专家标注构建语料库。
       2. 使用微调技术（如LoRA或全量微调）训练开放源码LLM。
       3. 在Defects4J和其他基准数据集上评估微调模型的生成效果。
     - **期望成果**：
       - 构建一个公开可用的单元测试生成语料库。
       - 提供经过微调的开放源码LLM，显著提升单元测试生成的质量。

:::

# Others

[ASE-2024] On the Evaluation of Large Language Models in Unit Test Generation​

> | Resource | Info |
> | :---        |:--- |
> | Paper       | https://arxiv.org/abs/2409.14781 |
> | Code & Data | https://github.com/zhang-wei-chao/DC-PDD |
> | Public      | EMNLP |
> | Date        | 2024.12.11 |


# Summary Overview

最近，已经探讨了通过Black-box访问来推断给定文本是否是LLM培训数据的一部分的数据检测方法。达到最先进结果的Min-K％ProP方法假设一个非训练示例倾向于包含一些具有低令牌概率的异常单词。但是，有效性可能受到限制，因为它倾向于错误分类的非训练文本，这些文本包含许多常见单词，其概率很高。

文章聚焦于检测 LLM 预训练数据检测 (Pretraining Data Detection)，提出了 DC-PDD 方法进行检测。

*How can we detect if a black-box LLM was pretrained on a given text, considering that its training data is undisclosed?*

![image.png](/static/img/47c459b3cb4add49ba9e7d0523c05c97.image.png)

# Main Content

使用 divergence-from-randomness 的基本思想是，the higher divergence of the within-document term-frequency of a word in a document from its frequency within the collection, the more information the word carries.

使用的 benchmark

![image.png](/static/img/14a71e29698b034b00e1ae24da0f424a.image.png)

**Task Description:**

Formally, given a piece of text $x$ and an LLM $\mathcal{M}$ with no knowledge of its pretraining corpus $\mathcal{D}$, the *pretraining data detection task* aims to design a method to determine if $x$ was included in $\mathcal{D}$. Thus, given $x$ and $\mathcal{M}$ as input, a method $\mathcal{A}$ for the pretraining data detection task returns 1 if it predicts that $x$ is included in $\mathcal{D}$ and 0 if it is not:

$$
\mathcal{A}(x,\mathcal{M})\rightarrow\{0,1\}
$$

**PatentMIA 构建:**

1. Data crawling: 随机爬取了5000个在2024.03.01之后的中国专利页面和5000个在2023.01.01之前的中国专利页面。
2. Data preprocessing: 预处理 + 数据清洗
3. Snippet extraction: 对于每个页面，从原始内容中随机提取512个单词的片段，创建了10,000个实例的数据集。

**Four steps of DC-PDD:**
1. *Token probability distribution computation*, by querying $\mathcal{M}$ with $x$.
2. *Token frequency distribution computation*, by using a large-scale publicly available corpus $\mathcal{D^\prime}$ as reference corpus to obtain an estimation of the token frequency distribution since $\mathcal{M}$'s pretraining corpus is not assumed to be accessible.
3. *Score calculation via comparision*, by comparing the above two distributions to calibrate the token probability for each token $x_i$ in $x$, and dervie a score for pretraining data detection based on the calibrated token probabilities.
4. *binary decision*, by applying a predefined threshold to the score, we predict whether $x$ was included in $\mathcal{M}$'s pretraining corpus or not.


![image.png](/static/img/db131818bd82997a384e45c788142ba5.image.png)

**Token Probability Distribution Computation**

$$
x^\prime=x_0x_1x_2\cdots x_n
$$

$$
\{p(x_i|x_{<i}:\mathcal{M}):0<i\leq n\}
$$

simplify $p(x_i|x_{<i};\mathcal{M})$ to $p(x_i;\mathcal{M})$

**Token Frequency Distribution Computation**

$$
p(x_i;\mathcal{D^\prime})=\frac{\text{count}(x_i)}{N^\prime}
$$

加一平滑:

$$
p(x_i;\mathcal{D^\prime})=\frac{\text{count}(x_i)+1}{N^\prime+|V|}
$$

**Score Calculation through Comparison**

$$
\alpha_i=-p(x_i;\mathcal{M})\cdot\log p(x_i;\mathcal{D^\prime})
$$

$$
\alpha_i =
\begin{cases} 
\alpha_i, & \text{if } \alpha_i < a \\ 
a, & \text{if } \alpha_i \geq a.
\end{cases}
$$

$$
\beta=\frac{1}{|\text{FOS}(x)|}\sum_{x_j\in\text{FOS}(x)}\alpha_j
$$

**Binary Decision**

$$
\text{Decision}(x, \mathcal{M}) =
\begin{cases} 
0 \, (x \notin \mathcal{D}), & \text{if } \beta < \tau \\ 
1 \, (x \in \mathcal{D}), & \text{if } \beta \geq \tau.
\end{cases}
$$

## Related Work

大多数研究都集中在黑框设置上，假设文本的令牌概率分布可以提供有关文本是否包含在培训集中的重要信息。 Carlini等。 （2021）将文本的模型视为检测GPT-2预处理数据的指标（Radford等，2019）。他们进一步介绍了三种方法，即Zlib，小写和较小的Ref，这些方法考虑了目标文本的内在复杂性。最近，Shi等人。 （2024年）提出了一种称为Min-k％概率的直接但表现良好的方法。 Min-k％概率倾向于将由常用单词组成的非训练文本分类为培训数据。一项并发研究Min-k％++概率（Zhang等，2024）通过标准化令牌概率来改善Min-k％概率，但需要访问LLM的整个词汇量之间的下一个预测概率分布，这在不可避免的情况下是不可接受的关闭llms等gpt-3（Brown等，2020）。我们考虑黑盒子设置和Calibra

# Experiments

## Metrics

**Area Under the ROC Curve (AUC).** AUC分数量化了分类方法的整体性能。为了计算AUC分数，需要在所有分类阈值下计算正式的正确率 (TPR) 和假阳性率 (FPR)，并绘制 TPR 与 FPR 曲线 (ROC) 曲线。然后将 AUC 定义为 ROC 曲线下的面积，提供了所有可能分类阈值的效果的总测量。因此 AUC 提供了一个全面的，阈值独立的分数，可以反映该方法有效区分正案例和负面案例的能力。

**TPR (true positive rate) at a low FPR (false positive rate).** 通过调整阈值，选择5%作为目标FPR值，并报告相应的TPR值，以较低的FPR报告TPR。

## Baselines

- PPL
- Lowercase
- Zlib
- Small Ref
- Min-k% Prob
- Min-k%++Prob

## Results

**Main Results**

![image.png](/static/img/fd6267de9854eba740d950c62b43e2ac.image.png)


![image.png](/static/img/9d614d6b8ed63e1f5d073fd564b70c8c.image.png)

**Ablation Studies**

![image.png](/static/img/d4e7c627b9f4954f2d68b505305668b9.image.png)

**Impact of Different Factors**

- model size and text length
![image.png](/static/img/3e39025912f10c87f56b5bb7fdc6206f.image.png)
- different reference corpus settings
![image.png](/static/img/76cdca8ff59082cd734ba3199d28ff49.image.png)
- different $\alpha$ settings
![image.png](/static/img/d6719a20c9a2fecb7d54b38df326a07f.image.png)

# 🤖ChatGPT

:::info{title="ChapGPT"}
1. 论文的创新之处与独特性：  
- **创新点**：  
  - 提出了一种基于“随机性偏离”理论的校准方法（DC-PDD），通过计算目标语言模型的预测概率分布与参考语料的词频分布之间的交叉熵（即散度），校准每个词的概率，用于检测给定文本是否属于语言模型的预训练数据。这种方法克服了现有方法对高频词的误分类问题，显著提升了检测性能。  
  - 构建了首个中文预训练数据检测基准数据集PatentMIA，通过谷歌专利数据区分训练与非训练样本，填补了现有研究在非英语语料上的空白。  
  - 提出的DC-PDD方法无需额外的参考模型或对目标语言模型的额外访问权限，降低了实际应用的复杂性，具备更高的通用性和可扩展性。  

- **关键点**：  
  - 通过散度校准改进了基于概率的检测方法，显著提高了检测的准确性和鲁棒性。  
  - 提供了跨语言基准（包括英语和中文）的全面实验，验证了方法的适用性和优越性。  
  - 系统性地分析了文本长度、模型规模、参考语料等因素对检测性能的影响，为后续研究提供了重要的启示。  

2. 论文中存在的问题及改进建议：  
- **问题**：  
  1. **参考语料的局限性**：虽然论文使用公开语料库（如C4和ChineseWebText）估算目标模型的词频分布，但这些语料与目标模型的实际预训练语料可能存在较大差异，可能影响校准结果的准确性。  
  2. **超参数敏感性**：DC-PDD方法对校准概率的上限参数（a）较为敏感，不同数据集和模型需要不同的参数设置，缺乏通用的优化策略。  
  3. **模型规模的限制**：实验主要集中在20亿参数以下的模型，未能验证方法在更大规模模型（如GPT-4级别）上的有效性。  
  4. **方法的适用范围**：DC-PDD方法主要针对文本数据，缺乏对多模态数据（如图像、音频等）的适用性讨论。  

- **改进建议**：  
  1. **增强参考语料的多样性**：引入更大规模、更丰富领域的参考语料库，或设计基于生成模型的动态语料扩充方法，以提高词频分布估计的准确性。  
  2. **自动化超参数优化**：探索基于贝叶斯优化或强化学习的自动化方法，为不同任务动态调整参数a，减轻人工调参的负担。  
  3. **扩展到更大模型**：在硬件条件允许的情况下，验证方法在更大规模模型（如GPT-4、Claude等）上的有效性，进一步提升方法的通用性。  
  4. **探索多模态扩展**：将“散度校准”思想扩展到图像、音频等多模态数据，通过结合跨模态特征分布，设计统一的预训练数据检测框架。  

3. 基于论文的内容和研究结果，提出的创新点或研究路径：  
- **创新点1**：开发基于生成对抗网络（GAN）的动态参考语料生成方法，用于改进DC-PDD的词频分布估计。  
- **创新点2**：设计一种跨模态预训练数据检测框架，结合文本、图像和音频的特征分布，扩展DC-PDD的适用范围。  
- **创新点3**：研究基于模型输出的全黑盒检测方法，避免对目标模型的概率分布访问需求，提升检测方法的适用性。  

4. 为新的研究路径制定的研究方案：  

### 研究路径1：基于GAN的动态参考语料生成  
- **研究方法**：  
  1. 使用生成对抗网络（GAN）生成与目标模型预训练语料相似的参考语料。  
  2. 通过对生成语料的词频分布与真实语料的分布进行对比，选择最接近的生成样本构建参考语料库。  
  3. 将动态生成的参考语料用于DC-PDD方法，重新计算词频分布并校准检测分数。  

- **研究步骤**：  
  1. 收集目标模型可能使用的预训练语料样本，作为GAN的初始训练数据。  
  2. 设计GAN生成器和判别器，优化生成语料的质量和多样性。  
  3. 对比生成语料与目标模型的预测分布，选择最优参考语料。  
  4. 在现有基准数据集（如PatentMIA）上验证改进方法的性能。  

- **期望成果**：  
  - 提高DC-PDD方法对参考语料选择的鲁棒性，显著提升检测准确性。  
  - 发布一套可扩展的动态参考语料生成工具，为其他预训练数据检测任务提供支持。  

### 研究路径2：跨模态预训练数据检测框架  
- **研究方法**：  
  1. 将文本、图像和音频等模态数据的特征分布映射到统一的潜在空间。  
  2. 基于散度校准思想，计算每种模态的特征分布与参考语料分布之间的散度。  
  3. 设计融合策略，将多模态散度综合为统一的检测分数。  

- **研究步骤**：  
  1. 构建跨模态数据集，包括文本、图像和音频的配对样本。  
  2. 设计跨模态特征提取网络，确保不同模态的特征可以在潜在空间中对齐。  
  3. 在每种模态上分别计算散度校准分数，并设计融合策略（如加权平均或注意力机制）。  
  4. 在多模态基准数据集上验证方法性能，并与单模态方法对比。  

- **期望成果**：  
  - 提出首个支持多模态预训练数据检测的通用框架。  
  - 实现跨模态检测性能的显著提升，为多模态模型的透明性研究提供新工具。  

### 研究路径3：基于模型输出的全黑盒检测方法  
- **研究方法**：  
  1. 通过分析目标模型的生成输出特征（如生成文本的重复率、语义一致性等），提取与预训练数据相关的信号。  
  2. 设计无监督聚类方法，将生成输出分为可能的训练数据和非训练数据两类。  
  3. 利用对抗样本生成技术，进一步验证模型对训练数据的敏感性。  

- **研究步骤**：  
  1. 收集目标模型在不同输入条件下的生成输出，构建输出特征库。  
  2. 提取生成输出的统计特征，并分析特征与训练数据的相关性。  
  3. 设计基于特征的聚类算法，预测生成输出是否来自训练数据。  
  4. 在现有基准数据集上验证方法性能，并分析与概率分布方法的差异。  

- **期望成果**：  
  - 提出一种完全黑盒的预训练数据检测方法，适用于封闭源模型。  
  - 为语言模型的隐私保护和数据透明性研究提供新思路。  

:::

# Others

[EMNLP-2024] Pretraining Data Detection for Large Language Models: A Divergence-based Calibration Method

# Jinja Template in `tokenizer`



## Base

在`transformers`中的`tokenizer`往往会有，以`Qwen2.5-7B-Instruct`为例，其`tokenizer_config.json`中有存在`chat_template`的键，其值如下：

```plaintext
"chat_template": "{%- if tools %}\n    {{- '<|im_start|>system\\n' }}\n    {%- if messages[0]['role'] == 'system' %}\n        {{- messages[0]['content'] }}\n    {%- else %}\n        {{- 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.' }}\n    {%- endif %}\n    {{- \"\\n\\n# Tools\\n\\nYou may call one or more functions to assist with the user query.\\n\\nYou are provided with function signatures within <tools></tools> XML tags:\\n<tools>\" }}\n    {%- for tool in tools %}\n        {{- \"\\n\" }}\n        {{- tool | tojson }}\n    {%- endfor %}\n    {{- \"\\n</tools>\\n\\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\\n<tool_call>\\n{\\\"name\\\": <function-name>, \\\"arguments\\\": <args-json-object>}\\n</tool_call><|im_end|>\\n\" }}\n{%- else %}\n    {%- if messages[0]['role'] == 'system' %}\n        {{- '<|im_start|>system\\n' + messages[0]['content'] + '<|im_end|>\\n' }}\n    {%- else %}\n        {{- '<|im_start|>system\\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\\n' }}\n    {%- endif %}\n{%- endif %}\n{%- for message in messages %}\n    {%- if (message.role == \"user\") or (message.role == \"system\" and not loop.first) or (message.role == \"assistant\" and not message.tool_calls) %}\n        {{- '<|im_start|>' + message.role + '\\n' + message.content + '<|im_end|>' + '\\n' }}\n    {%- elif message.role == \"assistant\" %}\n        {{- '<|im_start|>' + message.role }}\n        {%- if message.content %}\n            {{- '\\n' + message.content }}\n        {%- endif %}\n        {%- for tool_call in message.tool_calls %}\n            {%- if tool_call.function is defined %}\n                {%- set tool_call = tool_call.function %}\n            {%- endif %}\n            {{- '\\n<tool_call>\\n{\"name\": \"' }}\n            {{- tool_call.name }}\n            {{- '\", \"arguments\": ' }}\n            {{- tool_call.arguments | tojson }}\n            {{- '}\\n</tool_call>' }}\n        {%- endfor %}\n        {{- '<|im_end|>\\n' }}\n    {%- elif message.role == \"tool\" %}\n        {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != \"tool\") %}\n            {{- '<|im_start|>user' }}\n        {%- endif %}\n        {{- '\\n<tool_response>\\n' }}\n        {{- message.content }}\n        {{- '\\n</tool_response>' }}\n        {%- if loop.last or (messages[loop.index0 + 1].role != \"tool\") %}\n            {{- '<|im_end|>\\n' }}\n        {%- endif %}\n    {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n    {{- '<|im_start|>assistant\\n' }}\n{%- endif %}\n",
```

该模版是用于`tokenizer.apply_chat_template`函数中，通过调用该模版，将多轮对话的`messages`变成字符串，如下所示。

```python
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("/data1/HF-Models/Qwen/Qwen2.5-7B-Instruct")

chat = [
  {"role": "user", "content": "Hello, how are you?"},
  {"role": "assistant", "content": "I'm doing great. How can I help you today?"},
  {"role": "user", "content": "I'd like to show off how chat templating works!"},
]

tokenizer.apply_chat_template(chat, tokenize=False)
```
输出如下：
```plaintext
<|im_start|>system
You are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>
<|im_start|>user
Hello, how are you?<|im_end|>
<|im_start|>assistant
I'm doing great. How can I help you today?<|im_end|>
<|im_start|>user
I'd like to show off how chat templating works!<|im_end|>
```
如果设置，`add_generation_prompt=True`，则打印结果如下：

```plaintext
<|im_start|>system
You are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>
<|im_start|>user
Hello, how are you?<|im_end|>
<|im_start|>assistant
I'm doing great. How can I help you today?<|im_end|>
<|im_start|>user
I'd like to show off how chat templating works!<|im_end|>
<|im_start|>assistant
```

## Advanced: Adding and editing chat templates

很简单，只需编写一个 jinja 模板并设置 tokenizer.chat_template。您可能会发现，从另一个模型的现有模板开始，然后根据您的需要简单地对其进行编辑会更容易！例如，我们可以采用上面的 LLaMA 模板，并将 “[ASST]” 和 “[/ASST]” 添加到助手消息中：

```python
template = tokenizer.chat_template
template = template.replace("SYS", "SYSTEM")  # Change the system token
tokenizer.chat_template = template  # Set the new template
tokenizer.push_to_hub("model_name")  # Upload your new template to the Hub!
```

# References

- HF-Chat Templates: https://huggingface.co/docs/transformers/chat_templating#advanced-extra-inputs-to-chat-templates
- Jinja: https://jinja.palletsprojects.com/en/stable/templates/

Jinja Template in tokenizer

# Writing Perfect Papers


李葆春教授的分享，受益良多，感谢。本文主要是摘抄李教授提供的 Slides 作为后续查找复习资料。

Video: https://www.bilibili.com/video/BV18v411n7mr/

Slides: https://iqua.ece.toronto.edu/papers/writing-perfect-papers-2021.pdf



Our goal today: improve both the skills and the process.

# 1 Writing perfect papers

A top-down approach: from the big picture to tiny details.

## 1.1 The story

How do I make a great movie? A great movie starts from a powerful and convincing story.

**Three essential elements of a great story:**
Your paper is not really the best place to show your theoretical prowess. It's about advancing the state-of-the-art.
- **The problem**
    - Don't have to be something "**trendy**".
    - But it has to be something you really **enjoy** working on.
    - The "**trend**" may not last forever.
    - Following the trend may lead to more "**incremental**" results —— less exciting and less important.
    - Be a **contrarian** and working against the trend may be good idea.
- **What's new?**
    - In the context of what's already done in the related work, **what's new** in your work?
    - First, briefly introduce the **state-of-the-art** in the literature, solving the same problem you wish to solve.
    - Then write about what is **new** in you work that is different from previous work.
    - If the problem is **different** from previous work —— different assumptions, for example —— provide justifications **why** this is **important** and **necessary**.
- **How is it better?**
    - Now that your ideas are new, they should also lead to **bettter** results than before.
    - Show **how** your solution is **better** than previous work: such as stronger theoretical properties or better experimental results.

## 1.2 The work

Two things that you must do: **Read** and **Write**.

Read:
- Goal: become an expert on the problem.
- But how?
    - Start from one paper.
    - Perhaps a highly cited **seminal** paper.
    - Then do an **expanded-ring** search.
        - Papers that cited this paper.
        - Papers that it cited.
        - Papers authored by the same researchers.
    - Try not to depend only on a search engine.
    - Read **quickly** first.
    - Read more carefully if necessary **later**.

Write:
- Write about what you understood.
- Write every day.
- **idea -> write paper -> do research**
- If I don't have an idea what do I write about?
    - You write **anyway**.
    - Write about related work.
    - Write about what's challenging.
    - Write about why the problem is important.
    - Write about what needs to be fixed.
    - Write a survey paper.
    - Submit your survey paper for publication.
    - Write about **what's new** in the context of related work.
- Creating ideas interacts with **writing** closely.
- **Writing** is the best way to force yourself to **think clearly** and be **focused** and to **crystallize** what you don't quite understand yet.
- **Writing** also opens a dialogue.
- For others to read what you wrote.
- To stimulate discussions with others.
- That's why **writing** is a **slow** and **painful** process.
- Write in a **crystal clear** way.
- Write about **one** problem and **one** (potential) solution.
- Write with a **flow of ideas** that's easy to follow.
- Keep your readers engaged **throughout** the paper.
- Write your paper so that it's as easy to understand as **absolutely possible**.
- Your readers don't have to work hard.
- Write your paper **slowly**, so that your readers can read **quickly**.

## 1.3 The art

Pay attention to the title, abstract, introduction, and flow of ideas.

Tital: **2** lines, **10,000** readers.
Abstract: **10** lines, **1,000** readers.
Introduction: **100** lines, **100** readers.
The rest of the paper: **1,000** lines, **10** readers.

### The title

What's good title?

It attracts a reader to read the abstract.

It reflects the essence of the new idea.

It is as simple as possible
- does not have to be a precise summary of the paper.
- does not need to include all the keywords.

The best title is one that is the easiest to understand **at a glance**.

:::note{title="Example"}

Let's say we have a new idea that uses a new coding technique to reduce the latency when delivering data over the Internet. Which title is the best?

- LLRC: A Low-Latency Rate-Controlled System for Fast Data Delivery over the Internet
- Coded Information Distribution: New Content-Delivery Protocols for the Internet
- **Coding Reducces Latency over the Internet**

:::

### The abstract

The abstraction is a very important tool to attract readers to read the introduction.

Again, it conveys essential information about the paper.

It should be concise, and does not need to be long.

Remember the three essential elements: The problem, What's new? and How is it better?

A typical structure

- One sentence to state the background.
- One sentence to state **what the proble is**.
- Two to four sentences to state the original contribution in the paper —— **what's new?**
- One sentence to state **how the solution is better**, validated using analyses, simulations or experiments.

:::note{title="Abstract Example"}

(**Background**) With cloud storage services, it is commonplace for data to be not only stored in the cloud, but also shared across multiple users. (**Problem**) However, public auditing for such shared data -- while preserving both data and identity privacy -- remains to be an open challenge. (**Contribution: Overview**) In this paper, we propose the first privacy-preserving mechanism that allows public auditing on shared data stored in the cloud. (**Contribution: Highlights**) In particular, we exploit ring signatures to compute the metadata needed to verify the integrity of shared data. With our mechanism, the identity of the signer of each data block is kept private from a third party auditor (TPA), who is still able to publicly verify the integrity of shared data without retrieving the entire file. (**Contribution: Additional features**) In addition, we utilize homomorphic MACs and homomorphic hash function to mitigate the overhead of storing our signatures. (**Validation by experiments**) Our experimental results demonstrate the effectiveness and efficiency of our proposed mechanism when auditing shared data.

:::

### The introduction

If your title and abstract can get a reader to read the introduction, you are half way there.

The remaining job: impress the reader with an exciting and clear introduction.

The introduction is so important that I rewrote the introduction in many of papers.

It is important because we wish our reader to understand our ideas, even if they only read the introduction and noting else.

**Write a good introduction**
- It should be self-contained, so that a reader short on time doesn't need to read the rest of the paper.
- It should be clear **what the problem is**.
- It should be easy to understand **why our solution is new**, and **how it's better** than previous work.
- Don't make it too long.
- Don't spend too much space on the background and related work.
    - There can be a separate "Related Work" section.
- Don't make your work sound more original than it really is.

**Introduciont: a typical structure**
- Frist (opening) paragraph:
    - A **general overview** of the research field -- basic facts needed to "warm up" the reader and to prepare for the problem statement.
    - Not too long -- 2-3 sentences are good enough.
- The second paragraph: **problem statement**
    - A clear description of the challenges to be addressed and the **problems** to be solved.
    - Appeal to the intuition of the reader.
    - Describe at a high level.
    - It is fine (and even a good idea) to include an **intuitive solution** to the problem.

Unnecessary details inhibit the **natural** and **brisk** flow of the main ideas.

Returning to the typical structure of the introduction:
- General overview to warm-up the reader (the opening paragraph)
- State the problem (challenge) and existing solutions (1-2 paragraphs)
- Limitations of existing solutions that motivate this paper (2-3 paragraphs)
- Proposed solution: main idea (1-2 paragraphs)

**Write about what's new**
- Why is the proposed solution different from and better than existing solutions?
- State 1-2 most impressive highlights, not all of them.
- Make the originality of the paper crystal clear and stand out.
- You may use the sentence: "Highlights of our original ocntributions in this paper are as follows. First, ... Second, ... Finally, ..."

The list of original contributions drives the **entire paper** -- the rest of the paper substantiates the claims you have made.

Your readers may think: "Hey, if they can really deliver this, that'll be very exciting! I'd better read the rest of the paper."

It is a good idea to include a **table** to compare important properties of the proposed solution with its "direct competitors" in the existing literature, highlighting your **advantages**.

Use an example:
- It's an even better idea to show an intuitive **example**.
- Your **example** shows how your **main idea** works in a **special case**.
- Continue with **more** examples throughout the paper.
- After each theorem is proved -- or each algorithm shown -- explain their **implications** or **intuition** with an example.
- Your examples need to be really **simple**.
- They helps readers to understand your solution well.
- It'll be a greate idea to include a well-designed **figure** to illustrate your example.
- Includes as many figures as you can throughout the paper.

The **flow of ideas:**
- Just like the storyboard when making a movie, the **flow of ideas** needs to be carefully **designed**.
- Make sure that each paragraph contains one **complete** idea - **split** a paragraph that contains more than one idea, and merge short paragraphs that cover the same idea.
- Where is the section on **related work**?
    - Some prefer to place it after the introduction
        - Rationale: the section can be used to "warm up" the reader.
    - Some others prefer to place it before the conclusion
        - Rationale: After the introduction, the readers don't understand your main ideas yet, there's no point in talking about their differences from related work.
    - It depends on the design of your flow of ideas.
- Where do you write about **experimental results**?
    - Most papers collect all the experimental results and put them **at the end**.
    - But you don't have to.
    - Some **negative** or **preliminary** experimental results may be important for motivating the main idea in the paper.
    - Your flow of ideas can then be: "**initial results - idea to improve - more results to show better performance**"
    - You can even interleave experimental results with descriptions of your idea, if this provides the best flow.
- How do you allocate space across sections?
    - The short answer is: no one knows it better than you, because it depends on the flow of ideas.

**General rules of thumb:**
- Don't write a very long abstract (200 words), introduction (one page), related work (half a page), or concluding remarks (1-2 paragraphs)
- Throughout the paper, make it self-contained, but don't use a lot of space for unnecessary background.
- Keep the motivation short and concise.
## 1.4 The details

**Use a git repository to manage your workflow.**

Get repositories help you to track all histories and collaborate with others.

Place everything - source code, reference - into your git repository.

Use any of the public cloud services such as GitHub.

**Eliminate typographical and grammatical mistakes.**

First, eliminate all spelling mistakes by running your paper through a spell checker.

Use a UNIX command-line tool such as spell to check spelling - not Microsoft Word.

Then fix grammar and usage problems by proofreading.

**Proofreading** also helps you to fix the remaining spelling mistakes that a spell checker cannot catch.

You cannot rely on someone else - certainly not the reviewers - to proofread for you, it's your paper!

**Use transitional words, phrases, and sentences**

Use **transitions** across the boundary of sentences, paragraphs and sections.

**Important** for readers to follow your **flow of ideas**.

If you don't add transitons, readers will need to add them subocnsciously in their mind, anyway. You are asking readers to do the hard work!


:::note{title="Examples of transitions"}
Connecting two halves of a sentence:
- as, since ,or else

Connecting sentences:
- However, In addition, Further, Nevertheless, Fortunately, Unfortunately, Suprisingly,
- To make matters worse, to further exacerbate the problem, The bad new is,
- The implications are two-fold, It turns out that, As as example, To take ... a step further,

Connecting paragraphs and sections:
- It only remains to see ...
- The simple answer to this question is,
- The only challenge that remains now is,
- To address this challenge,
- We first present...
- Next, we evaluate...
- We are now ready to...

:::

**Use correct English**

- Keep punctuations marks inside the closing quotation mark.
- Don't use long sentences
    - Don't use long sentences with more than one comma in the middle of the sentence - and **abuse** "where", "in which", "whose", 'so that", "such that".
- Don't be too colloquial
    - Don't use words that are too informal and colloquial.
    - "a lot of" is more colloquial than "a large number of".
    - Instead of "big" use "substantial" or "large".
- Don't be too formal either
    - Overly formal words and phrases sometimes feel awkward.
    - "we endeavour to ascertain that..." - "We show that..."
    - "It can be ascribed to..." - "It is due to..."
    - "The overwhelming quantity of..." - "The exceedingly large number of..."
- Don't use words that are too emotional
- Countable vs. uncountable nouns
    - If a word is countable, when using its plural form, remember to use "a few", "a number of", "fewer". Instead of "less bits," use "fewer bits".
    - If not, do not invent its plural form (such as "performances," "advices," "equipments," "informations"),
    - and use "less" or "lower" rather than "fewer"
- Agreement of the verb with the subject.
- Watch out on common words and phrases.
    - It almost takes a lifetime to master the usage of common words, such as "with", "of", "for", "at", "against", "in", "on"
    - Or to master the large number of phrases in English "in other words" or "in another word"?
- Articles as determiners
    - The articles "a" / "an" (the indefinite article) and "the" (the definite article) are frequently used in correctly by new students
    - Plural nouns are typically used without an article:
        - "The source node receives acknowledgements" (not "the acknowledgments")
    - The indefinite is weaker than the definite article:
        - "a large portion of" (not "the large portion of")
    - But there's no need for "the" in section titles - instead of "The System Model," just say "System Model."
    - Articles can be tricky to use, but there are too many of them in paper - pay attention!

Good news: most English problems are not hard to fix - just proofread every sentence with plenty of time!

**Typest your paper correctly and beautifully**

1. Use LaTex, no matter what.
2. Draw figures using a vector-based application: Image-based drawing applications produce images that become fuzzy when zoomed in on a reader's iPad.
3. Proofread your bibliography and make it consistent: Use BibTex to typeset the bibliography in your paper. After downloading BibTex entries from the web, proofread to get them consistent. Keep a consistent style of abbreviating journal and conference titles throughout the bibliography. 

Try your best to find out the venue where the paper was published in, rather than citing arxiv.org.

# 2 Writing good papers fast!

Writing good papers is hard. That's why you are advised to write slowly. In the good old days, academics can take years to write one research paper. We also have the "publish or perish" culture, where the number of papers is an indication of quality. I need to graduate - tell me how to write papers quickly.

http://faculty.hitsz.edu.cn/hubaotian

![截屏2024-10-19 10.02.46.png](/static/img/083df19c1637a02c1271. 5cfe4c8f96e32.%C3%A6%C2%88%C2%AA%C3%A5%C2%B1%C2%8F2024-10-19%2010.02.46.png)

**Know your deadline**

At all times, always work towards a paer deadline.

Deadlines are all equal: It doesn't matter what kind of deadline it is - 
- a conference deadline
- a deadline from a journal special issue
- a deadline you set up yourself for submitting a journal paper

1. You will have to make it no matter what happens.

Set your deadline realistically. Admit that you are not able to write a good paper in a month or two.

But a deadline six months later may not carry a lot of weight either - unless you are extremely disciplined.

2. Make it public knowledge

Tell as many peoples as you can when your deadline will be.

So that it feels like a public disgrace if you fail to meet the deadline. As a researcher, your sure wish to be seen as responsible and dependable.

You work around the clock to make sure that others who depend on you will have the peace of mind that the work will be done and done by the deadline.

3. Allocate your time well.

Design a rough timeline at the very beginning. Assign deadlines to each milestone in your timeline.

If necessary, ervise your timeline along the way. But think seriously why you missed your milestone deadlines. Don't procrastinate and delay all the work to the days before the deadline. It may be feasible, but it will be stressful and it degrades the quality of your work

4. Don't follow, lead!

You are the first author of the new paper, right? Then you are the leader, the boss, and the person who cares most about this work.

Start your team at the beginning of your project. Only invite trustworthy collaborators when building your team. Your collaborators include your advisors - get them to work for you. Get them to commit their time. Warn them of the tough work ahead along the way. Promise them an exciting outcome. Assign your collaborators to milestones - they are your "resources". Remind them about their milestone deadlines often. Persist if they forget about your requests. Cancel their "membership" in your team if they consistently fail. Have backup plans if they fail to deliver by the deadline.

**Choose a problem with the right size**

It must be feasible to be completed before the deadline.

Goal: make solid improvements over on paper, and one paper only.

Choose a paper that you enjoyed reading or even better, a paper with source code. If its source code is not on GitHub, ask the authors. If you don't have source code, estimate the amount of time to reproduce the work. If it may take a long time, give up and choose another paper.

Read the paper along with its source code. The source code contains secret ingredients in its "recipe". It helps you to understand the paper completely and deeply. And only with a complete and deep understanding can you create something new. And create something new quickly. To do it, you will have to a deadline in your timeline for creating something new.

A deadline forces you to think long and hard. Revise the source code and do a feasibility test for your idea. If its feasibility is promising, think about its theoretical properties - anything provably correct? Your little problem must fit into the scope of the conference you'll submit to.

Goal: get the paper accepted.

Many conferences have their favoured research topics.