v1.1.0

.gitignore

@@ -0,0 +1,4 @@
+**/__pycache__
+run.bat
+openai_api_test.py
+assets/

README.md

@@ -10,53 +10,8 @@ pinned: false
 license: apache-2.0
 ---
 
-# EasyTranslator v1.0.6
+# EasyTranslator v1.1.0
 基于gradio的汉化辅助工具
-## v1.0.6更新内容
-1. 更新文件合并功能，方便多人协作。在文件合并页中可将依照指示将两个json文件合并，同步人工翻译进度。并支持导出小规模json文件方便传输。
-
-## v1.0.5更新内容
-1. 支持键盘快捷键<br>
-    shift+w: ↑<br>
-    shift+x: ↓<br>
-    shift+s: save json<br>
-    shift+r: replace<br>
-    shift+g: gpt translate<br>
-    shift+b: baidu translate<br>
-
-## v1.0.4更新内容
-1. 追加摸鱼模式, 将必要组件集中在半个屏幕内。在`config.json`中`moyu_mode`设为1开启, 设为0关闭
-2. 加入对GPT翻译的超时检测, 时间上限在`config.json`的`openai_api_settings`中的`time_limit`处设置, 单位为秒。若请求超时, 会打印超时提示, 但不会报错）
-3. GPT翻译现在将不返回重复结果
-
-## v1.0.3更新内容
-1. 支持预览页直接修改译文, 建议保存JSON后再使用此功能
-2. 可选是否即时更新上次编辑id
-
-    `config.json`中设置`"if_save_id_immediately"`参数, 若为1则逻辑与之前一样, 在切换id时立刻保存进`config.json`；若为0则会显示保存编辑id按钮`SAVE last edited position`, 在点击后存入`config.json`。
-
-## v1.0.2更新内容
-1. 支持批量机翻
-
-## v1.0.1更新内容
-1. 优化文件读取逻辑
-2. 增加错误提示、警告等。保存JSON成功时会提示更新的译文条数
-3. 允许自定义传输到gpt的prompt、自定义百度翻译的原文及目标语言
-4. 追加上下文预览功能, 并允许自定义预览条数和编号。指定id将会以双星号标记, 修改过的译文将会在前面加星号标记
-5. 优化按钮手感
-
-## 特性
-1. 一键机翻接口, 提供复制到剪贴板按钮
-2. 便捷的上下句切换, 直接跳转功能
-3. 记忆上次编辑位置功能
-4. 人名翻译记忆功能, 一次修改将会同步到全体。人名词典在程序启动时读取并在保存JSON文件时保存。开启程序时可以直接改`name_cn`, 关闭程序后可以修改人名词典。下次开启程序时人名词典中的内容将会覆盖JSON文件中的`name_cn`。
-5. 文本翻译记忆功能, 机翻/修改后只要不关闭程序, 切换上下句, 刷新 网页都不会影响
-6. 译文缓存。相对地原文不会缓存, 所以手滑改或删掉只要切换或者刷新即可恢复。因此想查看原文具体某个词的翻译也可以直接编辑原文再机翻, 不会影响原文本。
-7. 一键替换功能, 用于专有名词错译的情况。会将机翻及手翻文本中的对象全部替换。替换词典可以在运行中直接更改, 不用重开程序。
-8. 便利的api key管理及prompt修改等
-9. 提供JSON文件与CSV文件互转
-10. 上下文预览功能
-<br><br>
 
 ## 使用
 至少需要安装python3(作者使用的版本是3.10, 其它版本尚未测试)
@@ -102,10 +57,60 @@ json文件格式要求为：
 ```
 python EasyTranslator.py
 ```
-(hf版为app.py)
 然后在网页中打开程序给出的网址（eg: http://127.0.0.1:7860 ）
 <br><br>
 
+## v1.1.0更新内容
+1. 现支持通常翻译和批量翻译中gemini, claude, qwen, deepseek...等模型的自由选择和调用。请在`config.json`中或API页填写api key (注：当设置了OpenRouter的api时，会优先使用OpenRouter的接口)。使用gemini, claude, doubao等的官方接口请安装相关依赖包。可以自行修改`utils.py`中的MODEL_LIST以调整可选模型，避免冗余。
+2. 修改了键盘快捷键设置<br>
+    alt+w: ↑<br>
+    alt+x: ↓<br>
+    alt+s: save json<br>
+    alt+r: replace<br>
+    alt+q: model1 translate<br>
+    alt+e: model2 translate<br>
+
+## v1.0.6更新内容
+1. 更新文件合并功能，方便多人协作。在文件合并页中可将依照指示将两个json文件合并，同步人工翻译进度。并支持导出小规模json文件方便传输。
+
+## v1.0.5更新内容
+1. 支持键盘快捷键
+
+## v1.0.4更新内容
+1. 追加摸鱼模式, 将必要组件集中在半个屏幕内。在`config.json`中`moyu_mode`设为1开启, 设为0关闭
+2. 加入对GPT翻译的超时检测, 时间上限在`config.json`的`openai_api_settings`中的`time_limit`处设置, 单位为秒。若请求超时, 会打印超时提示, 但不会报错）
+3. GPT翻译现在将不返回重复结果
+
+## v1.0.3更新内容
+1. 支持预览页直接修改译文, 建议保存JSON后再使用此功能
+2. 可选是否即时更新上次编辑id
+
+    `config.json`中设置`"if_save_id_immediately"`参数, 若为1则逻辑与之前一样, 在切换id时立刻保存进`config.json`；若为0则会显示保存编辑id按钮`SAVE last edited position`, 在点击后存入`config.json`。
+
+## v1.0.2更新内容
+1. 支持批量机翻
+
+## v1.0.1更新内容
+1. 优化文件读取逻辑
+2. 增加错误提示、警告等。保存JSON成功时会提示更新的译文条数
+3. 允许自定义传输到gpt的prompt、自定义百度翻译的���文及目标语言
+4. 追加上下文预览功能, 并允许自定义预览条数和编号。指定id将会以双星号标记, 修改过的译文将会在前面加星号标记
+5. 优化按钮手感
+
+## 特性
+1. 一键机翻接口, 提供复制到剪贴板按钮
+2. 便捷的上下句切换, 直接跳转功能
+3. 记忆上次编辑位置功能
+4. 人名翻译记忆功能, 一次修改将会同步到全体。人名词典在程序启动时读取并在保存JSON文件时保存。开启程序时可以直接改`name_cn`, 关闭程序后可以修改人名词典。下次开启程序时人名词典中的内容将会覆盖JSON文件中的`name_cn`。
+5. 文本翻译记忆功能, 机翻/修改后只要不关闭程序, 切换上下句, 刷新 网页都不会影响
+6. 译文缓存。相对地原文不会缓存, 所以手滑改或删掉只要切换或者刷新即可恢复。因此想查看原文具体某个词的翻译也可以直接编辑原文再机翻, 不会影响原文本。
+7. 一键替换功能, 用于专有名词错译的情况。会将机翻及手翻文本中的对象全部替换。替换词典可以在运行中直接更改, 不用重开程序。
+8. 便利的api key管理及prompt修改等
+9. 提供JSON文件与CSV文件互转
+10. 上下文预览功能
+<br><br>
+
+
 ## 演示
 摸鱼模式 \
 ![image](https://github.com/alienet1109/EasyTranslator/blob/master/assets/moyu_mode.png) \

modules/llm/BaseLLM.py

@@ -0,0 +1,32 @@
+from abc import ABC, abstractmethod
+
+class BaseLLM(ABC):
+
+    def __init__(self):
+        pass
+    
+    @abstractmethod
+    def initialize_message(self):
+        pass
+
+    @abstractmethod    
+    def ai_message(self, payload):
+        pass
+
+    @abstractmethod
+    def system_message(self, payload):
+        pass
+
+    @abstractmethod
+    def user_message(self, payload):
+        pass
+
+    @abstractmethod
+    def get_response(self):
+        pass
+
+    @abstractmethod
+    def print_prompt(self):
+        pass
+
+

modules/llm/Claude.py

@@ -0,0 +1,55 @@
+import anthropic
+import os
+from .BaseLLM import BaseLLM
+
+class Claude(BaseLLM):
+
+    def __init__(self, model="claude-3-5-sonnet-20240620"):
+        super(Claude, self).__init__()
+        self.model_name = model
+        self.client = anthropic.Anthropic(
+            api_key = os.environ.get("ANTHROPIC_API_KEY")
+        )
+        # add api_base        
+        self.messages = []
+
+    def initialize_message(self):
+        self.messages = []
+
+    def ai_message(self, payload):
+        self.messages.append({"role": "ai", "content": payload})
+
+    def system_message(self, payload):
+        self.messages.append({"role": "system", "content": payload})
+
+    def user_message(self, payload):
+        self.messages.append({"role": "user", "content": payload})
+
+    def get_response(self):
+        message = self.client.messages.create(
+            model=self.model_name,
+            max_tokens=4096,
+            messages=self.messages
+        )
+        return message.content
+
+    def chat(self,text):
+        self.initialize_message()
+        if isinstance(messages, str):
+            self.user_message(text)
+            response = self.get_response()
+        return response
+    
+    def print_prompt(self):
+        for message in self.messages:
+            print(message)
+            
+if __name__ == '__main__':
+    messages = [{"role": "system", "content": "Hello, how are you?"}]
+    model = "claude-3-5-sonnet-20240620"
+    #model = 'gpt-4o'
+    llm = Claude()
+    
+    print(llm.chat("Say it is a test."))
+        
+

modules/llm/DeepSeek.py

@@ -0,0 +1,49 @@
+from .BaseLLM import BaseLLM
+from openai import OpenAI
+import os
+
+class DeepSeek(BaseLLM):
+    
+    def __init__(self, model="deepseek-chat"):
+        super(DeepSeek, self).__init__()
+        self.client = OpenAI(
+        api_key=os.getenv("DEEPSEEK_API_KEY"), 
+        base_url="https://api.deepseek.com",
+    )
+        self.model_name = model
+        self.messages = []
+
+
+    def initialize_message(self):
+        self.messages = []
+
+    def ai_message(self, payload):
+        self.messages.append({"role": "ai", "content": payload})
+
+    def system_message(self, payload):
+        self.messages.append({"role": "system", "content": payload})
+
+    def user_message(self, payload):
+        self.messages.append({"role": "user", "content": payload})
+
+    def get_response(self,temperature = 0.8):
+    
+        response = self.client.chat.completions.create(
+        model="deepseek-chat",
+        messages=[
+            {"role": "system", "content": "You are a helpful assistant"},
+            {"role": "user", "content": "Hello"},
+        ],
+        stream=False
+)
+        return response.choices[0].message.content
+    
+    def chat(self,text):
+        self.initialize_message()
+        self.user_message(text)
+        response = self.get_response()
+        return response
+    
+    def print_prompt(self):
+        for message in self.messages:
+            print(message)

modules/llm/Doubao.py

@@ -0,0 +1,44 @@
+from .BaseLLM import BaseLLM
+from volcenginesdkarkruntime import Ark
+import os
+
+class Doubao(BaseLLM):
+    
+    def __init__(self, model="ep-20241228220355-cqxcs"):
+        super(Doubao, self).__init__()
+        self.client = Ark(api_key=os.environ.get("ARK_API_KEY"))
+        self.model_name = model
+        self.messages = []
+
+    def initialize_message(self):
+        self.messages = []
+
+    def ai_message(self, payload):
+        self.messages.append({"role": "ai", "content": payload})
+
+    def system_message(self, payload):
+        self.messages.append({"role": "system", "content": payload})
+
+    def user_message(self, payload):
+        self.messages.append({"role": "user", "content": payload})
+
+    def get_response(self,temperature = 0.8):
+    
+        completion = self.client.chat.completions.create(
+        model=self.model_name,
+        messages=self.messages,
+        temperature=temperature,
+        top_p=0.8
+        )
+
+        return completion.choices[0].message.content
+    
+    def chat(self,text):
+        self.initialize_message()
+        self.user_message(text)
+        response = self.get_response()
+        return response
+    
+    def print_prompt(self):
+        for message in self.messages:
+            print(message)

modules/llm/Gemini.py

@@ -0,0 +1,45 @@
+from .BaseLLM import BaseLLM
+import google.generativeai as genai
+import os
+import time
+
+class Gemini(BaseLLM):
+    def __init__(self, model="gemini-1.5-flash"):
+        super(Gemini, self).__init__()
+        genai.configure(api_key=os.getenv("GEMINI_API_KEY"))
+        self.model_name = model
+        self.model = genai.GenerativeModel(model)
+        self.messages = []
+
+
+    def initialize_message(self):
+        self.messages = []
+
+    def ai_message(self, payload):
+        self.messages.append({"role": "model", "parts": payload})
+
+    def system_message(self, payload):
+        self.messages.append({"role": "system", "parts": payload})
+
+    def user_message(self, payload):
+        self.messages.append({"role": "user", "parts": payload})
+
+    def get_response(self,temperature = 0.8):
+        time.sleep(3)
+        chat = self.model.start_chat(
+            history = self.messages
+        )
+        response = chat.send_message(generation_config=genai.GenerationConfig(
+        temperature=temperature,
+        ))
+        
+        return response.text
+    
+    def chat(self,text):
+        chat = self.model.start_chat()
+        response = chat.send_message(text)
+        return response.text
+    
+    def print_prompt(self):
+        for message in self.messages:
+            print(message)

modules/llm/LangChainGPT.py

@@ -0,0 +1,46 @@
+from .BaseLLM import BaseLLM
+from openai import OpenAI
+import os
+
+class LangChainGPT(BaseLLM):
+
+    def __init__(self, model="gpt-4o-mini"):
+        super(LangChainGPT, self).__init__()
+        self.client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))
+        self.model_name = model
+        # add api_base        
+        self.messages = []
+
+    def initialize_message(self):
+        self.messages = []
+
+    def ai_message(self, payload):
+        self.messages.append({"role": "ai", "content": payload})
+
+    def system_message(self, payload):
+        self.messages.append({"role": "system", "content": payload})
+
+    def user_message(self, payload):
+        self.messages.append({"role": "user", "content": payload})
+
+    def get_response(self,temperature = 0.8):
+    
+        completion = self.client.chat.completions.create(
+        model=self.model_name,
+        messages=self.messages,
+        temperature=temperature,
+        top_p=0.8
+        )
+        return completion.choices[0].message.content
+    
+    def chat(self,text,temperature = 0.8):
+        self.initialize_message()
+        self.user_message(text)
+        response = self.get_response(temperature = temperature)
+        return response
+    
+    def print_prompt(self):
+        for message in self.messages:
+            print(message)
+
+

modules/llm/LocalModel.py

@@ -0,0 +1,66 @@
+from .BaseLLM import BaseLLM
+from peft import PeftModel
+import os
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+
+class LocalModel(BaseLLM):
+    def __init__(self, model, adapter_path = None):
+        super(LocalModel, self).__init__()
+        model_name = model
+        self.model = AutoModelForCausalLM.from_pretrained(
+            model_name,
+            torch_dtype="auto",
+            device_map="auto",
+            
+        )
+        if isinstance(adapter_path,str):
+            self.model = PeftModel.from_pretrained(self.model, adapter_path)
+        elif isinstance(adapter_path,list):
+            for path in adapter_path:
+                self.model = PeftModel.from_pretrained(self.model, path)
+            
+        self.tokenizer = AutoTokenizer.from_pretrained(model_name)
+        self.model_name = model      
+        self.messages = []
+
+    def initialize_message(self):
+        self.messages = []
+
+    def ai_message(self, payload):
+        self.messages.append({"role": "ai", "content": payload})
+
+    def system_message(self, payload):
+        self.messages.append({"role": "system", "content": payload})
+
+    def user_message(self, payload):
+        self.messages.append({"role": "user", "content": payload})
+
+    def get_response(self,temperature = 0.8):
+    
+        text = self.tokenizer.apply_chat_template(
+                self.messages,
+                tokenize=False,
+                add_generation_prompt=True
+            )
+        model_inputs = self.tokenizer([text], return_tensors="pt").to(self.model.device)
+        generated_ids = self.model.generate(
+            **model_inputs,
+            max_new_tokens=512
+        )
+        generated_ids = [
+            output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
+        ]
+
+        response = self.tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
+        return response
+    
+    def chat(self,text,temperature = 0.8):
+        self.initialize_message()
+        self.user_message(text)
+        response = self.get_response(temperature = temperature)
+        return response
+    
+    def print_prompt(self):
+        for message in self.messages:
+            print(message)

modules/llm/OpenRouter.py

@@ -0,0 +1,42 @@
+from .BaseLLM import BaseLLM
+import os
+from openai import OpenAI
+
+class OpenRouter(BaseLLM):
+    def __init__(self, model="deepseek/deepseek-r1:free"):
+        super(OpenRouter, self).__init__()
+        self.client = OpenAI(
+        api_key=os.getenv("OPENROUTER_API_KEY"), 
+        base_url="https://openrouter.ai/api/v1",
+        )
+        self.model_name = model
+        self.messages = []
+
+    def initialize_message(self):
+        self.messages = []
+
+    def ai_message(self, payload):
+        self.messages.append({"role": "ai", "content": payload})
+
+    def system_message(self, payload):
+        self.messages.append({"role": "system", "content": payload})
+
+    def user_message(self, payload):
+        self.messages.append({"role": "user", "content": payload})
+
+    def get_response(self,temperature = 0.8):
+        completion = self.client.chat.completions.create(
+            model=self.model_name,
+            messages=self.messages
+            )
+        return completion.choices[0].message.content
+    
+    def chat(self,text,temperature = 0.8):
+        self.initialize_message()
+        self.user_message(text)
+        response = self.get_response(temperature = temperature)
+        return response
+    
+    def print_prompt(self):
+        for message in self.messages:
+            print(message)

modules/llm/Qwen.py

@@ -0,0 +1,48 @@
+from .BaseLLM import BaseLLM
+from openai import OpenAI
+import os
+
+class Qwen(BaseLLM):
+
+    def __init__(self, model="qwen-max"):
+        # qwen-max, qwen-plus, qwen-turbo
+        super(Qwen, self).__init__()
+        self.client = OpenAI(
+        api_key=os.getenv("DASHSCOPE_API_KEY"), 
+        base_url="https://dashscope.aliyuncs.com/compatible-mode/v1",
+    )
+        self.model_name = model
+        # add api_base        
+        self.messages = []
+
+    def initialize_message(self):
+        self.messages = []
+
+    def ai_message(self, payload):
+        self.messages.append({"role": "ai", "content": payload})
+
+    def system_message(self, payload):
+        self.messages.append({"role": "system", "content": payload})
+
+    def user_message(self, payload):
+        self.messages.append({"role": "user", "content": payload})
+
+    def get_response(self,temperature = 0.8):
+    
+        completion = self.client.chat.completions.create(
+        model=self.model_name,
+        messages=self.messages,
+        temperature=temperature,
+        top_p=0.8
+        )
+        return completion.choices[0].message.content
+    
+    def chat(self,text,temperature = 0.8):
+        self.initialize_message()
+        self.user_message(text)
+        response = self.get_response(temperature = temperature)
+        return response
+    
+    def print_prompt(self):
+        for message in self.messages:
+            print(message)

utils.py

@@ -4,9 +4,71 @@ import random
 import json
 from hashlib import md5
 from os import path as osp
+import os
 import csv
 import threading
 
+MODEL_NAME_DICT = {
+    "gpt-4":"openai/gpt-4",
+    "gpt-4o":"openai/gpt-4o",
+    "gpt-4o-mini":"openai/gpt-4o-mini",
+    "gpt-3.5-turbo":"openai/gpt-3.5-turbo",
+    "deepseek-r1":"deepseek/deepseek-r1",
+    "deepseek-v3":"deepseek/deepseek-chat",
+    "gemini-2":"google/gemini-2.0-flash-001",
+    "gemini-1.5":"google/gemini-flash-1.5",
+    "llama3-70b": "meta-llama/llama-3.3-70b-instruct",
+    "qwen-turbo":"qwen/qwen-turbo",
+    "qwen-plus":"qwen/qwen-plus",
+    "qwen-max":"qwen/qwen-max",
+    "qwen-2.5-72b":"qwen/qwen-2.5-72b-instruct",
+    "claude-3.5-sonnet":"anthropic/claude-3.5-sonnet",
+    "phi-4":"microsoft/phi-4",
+}
+
+def get_models(model_name):
+    # return the combination of llm, embedding and tokenizer
+    if os.getenv("OPENROUTER_API_KEY", default="") and "YOUR" not in os.getenv("OPENROUTER_API_KEY", default="") and model_name in MODEL_NAME_DICT:
+        from modules.llm.OpenRouter import OpenRouter
+        return OpenRouter(model=MODEL_NAME_DICT[model_name])
+    elif model_name == 'openai':
+        from modules.llm.LangChainGPT import LangChainGPT
+        return LangChainGPT()
+    elif model_name.startswith('gpt-3.5'):
+        from modules.llm.LangChainGPT import LangChainGPT
+        return LangChainGPT(model="gpt-3.5-turbo")
+    elif model_name == 'gpt-4':
+        from modules.llm.LangChainGPT import LangChainGPT
+        return LangChainGPT(model="gpt-4")
+    elif model_name == 'gpt-4o':
+        from modules.llm.LangChainGPT import LangChainGPT
+        return LangChainGPT(model="gpt-4o")
+    elif model_name == "gpt-4o-mini":
+        from modules.llm.LangChainGPT import LangChainGPT
+        return LangChainGPT(model="gpt-4o-mini")
+    elif model_name.startswith("claude-3-5"):
+        from modules.llm.Claude import Claude
+        return Claude(model="claude-3-5-sonnet-20241022")
+    elif model_name in ["qwen-turbo","qwen-plus","qwen-max"]:
+        from modules.llm.Qwen import Qwen
+        return Qwen(model = model_name)
+    elif model_name.startswith('doubao'):
+        from modules.llm.Doubao import Doubao
+        return Doubao()
+    elif model_name.startswith('gemini-2'):
+        from modules.llm.Gemini import Gemini
+        return Gemini("gemini-2.0-flash")
+    elif model_name.startswith('gemini-1.5'):
+        from modules.llm.Gemini import Gemini
+        return Gemini("gemini-1.5-flash")
+    elif model_name.startswith("deepseek"):
+        from modules.llm.DeepSeek import DeepSeek
+        return DeepSeek()
+    else:
+        print(f'Warning! undefined model {model_name}, use gpt-4o-mini instead.')
+        from modules.llm.LangChainGPT import LangChainGPT
+        return LangChainGPT()
+
 def load_config(filepath):
     with open(filepath, "r", encoding="utf-8") as file:
         args = json.load(file)
@@ -46,6 +108,7 @@ def get_baidu_completion(text,api_id,api_key,from_lang,to_lang):
 openai_api_key = args["openai_api_settings"]["openai_api_key"]
 time_limit = float(args["openai_api_settings"]["time_limit"])
 client = openai.OpenAI(api_key = openai_api_key)
+
 class GPTThread(threading.Thread):
     def __init__(self, model, messages, temperature):
         super().__init__()
@@ -63,19 +126,44 @@ class GPTThread(threading.Thread):
     )
         self.result = response.choices[0].message.content
     
-def get_gpt_completion(prompt, model="gpt-3.5-turbo",api_key = openai_api_key):
+def get_gpt_completion(prompt, time_limit = 10, model="gpt-40-mini"):
     messages = [{"role": "user", "content": prompt}]
     temperature = random.uniform(0,1)
     thread = GPTThread(model, messages,temperature)
     thread.start()
-    thread.join(10)
+    thread.join(time_limit)
     if thread.is_alive():
         thread.terminate()
         print("请求超时")
         return "TimeoutError", False
     else:
         return thread.result, True
-
+    
+class LLMThread(threading.Thread):
+    def __init__(self, llm, prompt, temperature):
+        super().__init__()
+        self.llm = llm
+        self.prompt = prompt
+        self.temperature = temperature
+        self.result = ""
+    def terminate(self):
+        self._running = False 
+    def run(self):
+        self.result = self.llm.chat(self.prompt, temperature = self.temperature)
+    
+def get_llm_completion(prompt, time_limit = 10, model_name="gpt-4o-mini"):
+    llm = get_models(model_name)
+    temperature = 0.7
+    thread = LLMThread(llm, prompt,temperature)
+    thread.start()
+    thread.join(time_limit)
+    if thread.is_alive():
+        thread.terminate()
+        print("请求超时")
+        return "TimeoutError", False
+    else:
+        return thread.result, True
+    
 def left_pad_zero(number, digit):
     number_str = str(number)
     padding_count = digit - len(number_str)
@@ -101,7 +189,7 @@ def convert_to_json(files, text_col, name_col, id_col):
         with open(path,"r",encoding="utf-8") as f:
             reader = csv.DictReader(f)
             line_num = sum(1 for _ in open(path,"r",encoding="utf-8"))
-            fieldnames = reader.fieldnames
+            fieldnames = reader.fieldnames if reader.fieldnames else []
             if id_col not in fieldnames:
                 ids = generate_ids(line_num)
                 i = 0