This commit is contained in:
liaibo
2025-12-06 16:47:17 +08:00
parent 15921bfeb7
commit 9ef8e06d67
75 changed files with 559237 additions and 93 deletions
+33
View File
@@ -0,0 +1,33 @@
{
"_name_or_path": "/fsx-llm/lblecher/checkpoints/nougat/small/20230426_125023",
"align_long_axis": false,
"architectures": [
"NougatModel"
],
"decoder_layer": 4,
"embed_dim": 128,
"encoder_layer": [
2,
2,
14,
2
],
"hidden_dimension": 1024,
"input_size": [
896,
672
],
"max_length": 3584,
"max_position_embeddings": 3584,
"model_type": "nougat",
"num_heads": [
4,
8,
16,
32
],
"patch_size": 4,
"torch_dtype": "float32",
"transformers_version": "4.25.1",
"window_size": 7
}
+6
View File
@@ -0,0 +1,6 @@
{
"bos_token": "<s>",
"eos_token": "</s>",
"pad_token": "<pad>",
"unk_token": "<unk>"
}
File diff suppressed because it is too large Load Diff
+4
View File
@@ -0,0 +1,4 @@
{
"model_max_length": 1000000000000000019884624838656,
"tokenizer_class": "PreTrainedTokenizerFast"
}
BIN
View File
Binary file not shown.

After

Width:  |  Height:  |  Size: 12 KiB

+76
View File
@@ -0,0 +1,76 @@
针对化学书籍(包含大量分子式、化学反应方程式、苯环结构图、上下标)的识别,传统的 OCR(如 RapidOCR、PaddleOCR)基本无法满足需求,因为它们只能识别“文字”,无法理解“化学结构的拓扑关系”或将其转换为 LaTeX 公式。
鉴于你是为了集成到笔记软件中,且已经验证 Ollama 可以成功调用你的 6800XT,最佳的“高精度 + 本地化”方案是利用 多模态大模型(VLM) 进行“视觉转录”,而不是传统的 OCR。
以下是针对 Windows + 6800XT 的具体高精度方案:
方案一:Ollama + MiniCPM-V 2.6 (最推荐,支持 6800XT)
这是目前开源界公认的小参数量下 OCR 能力最强(尤其是针对密集文本和科学公式)的模型之一。它能直接“看懂”图片里的化学式,并输出 Markdown/LaTeX。
1. 为什么选这个?
懂化学: 它不是像传统 OCR 那样把苯环识别成圆圈,而是理解这是化学结构,并能输出 LaTeX 代码。
硬件兼容: 既然你的 Ollama 能跑显卡,直接 pull 这个模型就能利用 6800XT 加速。
API 友好: 笔记软件插件可以通过 HTTP 请求直接调用本地的 localhost:11434。
2. 操作步骤
在终端中运行(确保 Ollama 已更新到最新版):
Bash
ollama run minicpm-v
# 或者尝试更轻量的
ollama run llama3.2-vision
3. 笔记软件集成逻辑 (Python 示例)
VLM 最大的优势是可以接受 Prompt(提示词)。你可以强制要求它只输出 LaTeX 格式。
Python
import base64
import requests
import json
def recognize_chemistry_page(image_path):
# 1. 图片转 Base64
with open(image_path, "rb") as f:
base64_image = base64.b64encode(f.read()).decode("utf-8")
# 2. 构造 Prompt,专门针对化学公式优化
prompt = """
Identify the content in this image.
Transcribe all text into Markdown format.
For chemical formulas and mathematical equations, you MUST use LaTeX format (e.g., $H_2O$, $C_6H_6$).
Do not explain the image, just output the content.
"""
# 3. 调用 Ollama API
url = "http://localhost:11434/api/generate"
data = {
"model": "minicpm-v", # 或 llama3.2-vision
"prompt": prompt,
"images": [base64_image],
"stream": False
}
response = requests.post(url, json=data)
result = response.json()
return result['response']
# 测试
print(recognize_chemistry_page("chem_book_page.jpg"))
方案二:Nougat (Meta 专门开发的学术 OCR)
如果你发现通用的大模型有时候会“胡编乱造”(幻觉),那么 Nougat 是专门为科学文献(PDF转Markdown)设计的“小”模型。
优点: 极高精度还原数学和化学公式(LaTeX)。
Windows 兼容性: 它基于 PyTorch。虽然原生 Windows + AMD 比较麻烦,但你可以强制使用 CPU 运行。
注意: 因为模型参数不大(Base版约 350M),用 CPU 跑虽然比 GPU 慢,但处理单页截图作为笔记插件完全可以接受(通常几秒钟)。
安装与使用:
Bash
pip install nougat-ocr
代码调用(CPU模式): 你需要通过命令行或 Python 封装调用,强制指定设备为 CPU。虽然慢一点,但它是目前开源界处理科学公式的“标准答案”。
+9
View File
@@ -0,0 +1,9 @@
# install4j Wizard
![](images/516ed8cc717d5949e9b831ecb5c409711ba657b4fe8a39a36cd02c79b56d0b28.jpg)
The install4j wizard could not find a Java(TM) Runtime Environment on your system. Please locate a suitable 64-bit JRE. (minimum version: 17, maximum version: 23)
Locate
Cancel
@@ -0,0 +1,71 @@
[
{
"type": "text",
"text": "install4j Wizard",
"text_level": 1,
"bbox": [
19,
10,
327,
128
],
"page_idx": 0
},
{
"type": "image",
"img_path": "images/516ed8cc717d5949e9b831ecb5c409711ba657b4fe8a39a36cd02c79b56d0b28.jpg",
"image_caption": [],
"image_footnote": [],
"bbox": [
52,
288,
156,
481
],
"page_idx": 0
},
{
"type": "text",
"text": "The install4j wizard could not find a Java(TM) Runtime Environment on your system. Please locate a suitable 64-bit JRE. (minimum version: 17, maximum version: 23)",
"bbox": [
191,
229,
967,
572
],
"page_idx": 0
},
{
"type": "text",
"text": "Locate",
"bbox": [
568,
834,
667,
903
],
"page_idx": 0
},
{
"type": "text",
"text": "Cancel",
"bbox": [
813,
834,
913,
903
],
"page_idx": 0
},
{
"type": "header",
"text": "X",
"bbox": [
937,
21,
982,
101
],
"page_idx": 0
}
]
+227
View File
@@ -0,0 +1,227 @@
{
"pdf_info": [
{
"para_blocks": [
{
"bbox": [
8,
2,
132,
24
],
"type": "title",
"angle": 0,
"lines": [
{
"bbox": [
8,
2,
132,
24
],
"spans": [
{
"bbox": [
8,
2,
132,
24
],
"type": "text",
"content": "install4j Wizard"
}
]
}
],
"index": 0
},
{
"type": "image",
"bbox": [
21,
54,
63,
90
],
"blocks": [
{
"bbox": [
21,
54,
63,
90
],
"lines": [
{
"bbox": [
21,
54,
63,
90
],
"spans": [
{
"bbox": [
21,
54,
63,
90
],
"type": "image",
"image_path": "516ed8cc717d5949e9b831ecb5c409711ba657b4fe8a39a36cd02c79b56d0b28.jpg"
}
]
}
],
"index": 2,
"angle": 0,
"type": "image_body"
}
],
"index": 2
},
{
"bbox": [
77,
43,
390,
107
],
"type": "text",
"angle": 0,
"lines": [
{
"bbox": [
77,
43,
390,
107
],
"spans": [
{
"bbox": [
77,
43,
390,
107
],
"type": "text",
"content": "The install4j wizard could not find a Java(TM) Runtime Environment on your system. Please locate a suitable 64-bit JRE. (minimum version: 17, maximum version: 23)"
}
]
}
],
"index": 3
},
{
"bbox": [
229,
156,
269,
169
],
"type": "text",
"angle": 0,
"lines": [
{
"bbox": [
229,
156,
269,
169
],
"spans": [
{
"bbox": [
229,
156,
269,
169
],
"type": "text",
"content": "Locate"
}
]
}
],
"index": 4
},
{
"bbox": [
328,
156,
368,
169
],
"type": "text",
"angle": 0,
"lines": [
{
"bbox": [
328,
156,
368,
169
],
"spans": [
{
"bbox": [
328,
156,
368,
169
],
"type": "text",
"content": "Cancel"
}
]
}
],
"index": 5
}
],
"discarded_blocks": [
{
"bbox": [
378,
4,
396,
19
],
"type": "header",
"angle": 0,
"lines": [
{
"bbox": [
378,
4,
396,
19
],
"spans": [
{
"bbox": [
378,
4,
396,
19
],
"type": "text",
"content": "X"
}
]
}
],
"index": 1
}
],
"page_size": [
403,
187
],
"page_idx": 0
}
],
"_backend": "vlm",
"_version_name": "2.6.3"
}
+70
View File
@@ -0,0 +1,70 @@
[
[
{
"type": "title",
"bbox": [
0.02,
0.014,
0.328,
0.131
],
"angle": 0,
"content": "install4j Wizard"
},
{
"type": "header",
"bbox": [
0.939,
0.025,
0.985,
0.103
],
"angle": 0,
"content": "X"
},
{
"type": "image",
"bbox": [
0.053,
0.29,
0.157,
0.484
],
"angle": 0,
"content": null
},
{
"type": "text",
"bbox": [
0.193,
0.234,
0.969,
0.577
],
"angle": 0,
"content": "The install4j wizard could not find a Java(TM) Runtime Environment on your system. Please locate a suitable 64-bit JRE. (minimum version: 17, maximum version: 23)"
},
{
"type": "text",
"bbox": [
0.569,
0.839,
0.668,
0.906
],
"angle": 0,
"content": "Locate"
},
{
"type": "text",
"bbox": [
0.814,
0.837,
0.915,
0.906
],
"angle": 0,
"content": "Cancel"
}
]
]
Binary file not shown.

After

Width:  |  Height:  |  Size: 2.3 KiB

@@ -0,0 +1,7 @@
首先是羰基氧原子接受质子, 而后生成烯醇。决定反应速率的步骤是生成烯醇的一步, 生成烯醇后, 卤素作为亲电试剂与烯醇的双键发生亲电加成, 生成 $\alpha$ -卤代羰基化合物和卤化氢, 所以酸催化常常是自催化。例如治疗支气管炎、喘息性支气管炎药物氯丙那林(Clorprenaline)中间体 $\alpha$ -溴代邻氯苯乙酮(1)的制备 (张宝丰. 中国医药工业杂志, 1989, 20:33):
![](images/13d3f822a19e58284df7e8f6375844713ec35907bdd2e920ebf752ecfc2c1b95.jpg)
反应时经常有一个诱导期,因为反应开始时烯醇化速率较慢。随着反应的进行,卤化氢浓度增大,烯醇化速率加快,反应也相应加快。反应初期,可加入少量氢卤酸以缩短诱导期,光照也常起到明显的催化效果。Lewis酸对某些反应有催化作用,例如苯乙酮的溴化,在催化量的三氯化铝存在下生成 $\alpha$ -溴代苯乙酮,而三氯化铝过量时生成间-溴苯乙酮。
![](images/b1459176fc74696f7ac62ec4c74a5593cc9087862c96fbcc27be5e98a3933f39.jpg)
@@ -0,0 +1,50 @@
[
{
"type": "text",
"text": "首先是羰基氧原子接受质子, 而后生成烯醇。决定反应速率的步骤是生成烯醇的一步, 生成烯醇后, 卤素作为亲电试剂与烯醇的双键发生亲电加成, 生成 $\\alpha$ -卤代羰基化合物和卤化氢, 所以酸催化常常是自催化。例如治疗支气管炎、喘息性支气管炎药物氯丙那林(Clorprenaline)中间体 $\\alpha$ -溴代邻氯苯乙酮(1)的制备 (张宝丰. 中国医药工业杂志, 1989, 20:33):",
"bbox": [
10,
0,
998,
200
],
"page_idx": 0
},
{
"type": "image",
"img_path": "images/13d3f822a19e58284df7e8f6375844713ec35907bdd2e920ebf752ecfc2c1b95.jpg",
"image_caption": [],
"image_footnote": [],
"bbox": [
45,
223,
531,
431
],
"page_idx": 0
},
{
"type": "text",
"text": "反应时经常有一个诱导期,因为反应开始时烯醇化速率较慢。随着反应的进行,卤化氢浓度增大,烯醇化速率加快,反应也相应加快。反应初期,可加入少量氢卤酸以缩短诱导期,光照也常起到明显的催化效果。Lewis酸对某些反应有催化作用,例如苯乙酮的溴化,在催化量的三氯化铝存在下生成 $\\alpha$ -溴代苯乙酮,而三氯化铝过量时生成间-溴苯乙酮。",
"bbox": [
14,
516,
992,
714
],
"page_idx": 0
},
{
"type": "image",
"img_path": "images/b1459176fc74696f7ac62ec4c74a5593cc9087862c96fbcc27be5e98a3933f39.jpg",
"image_caption": [],
"image_footnote": [],
"bbox": [
45,
741,
730,
996
],
"page_idx": 0
}
]
@@ -0,0 +1,232 @@
{
"pdf_info": [
{
"para_blocks": [
{
"bbox": [
14,
0,
1298,
216
],
"type": "text",
"angle": 0,
"lines": [
{
"bbox": [
14,
0,
1298,
216
],
"spans": [
{
"bbox": [
14,
0,
1298,
216
],
"type": "text",
"content": "首先是羰基氧原子接受质子, 而后生成烯醇。决定反应速率的步骤是生成烯醇的一步, 生成烯醇后, 卤素作为亲电试剂与烯醇的双键发生亲电加成, 生成 "
},
{
"bbox": [
14,
0,
1298,
216
],
"type": "inline_equation",
"content": "\\alpha"
},
{
"bbox": [
14,
0,
1298,
216
],
"type": "text",
"content": "-卤代羰基化合物和卤化氢, 所以酸催化常常是自催化。例如治疗支气管炎、喘息性支气管炎药物氯丙那林(Clorprenaline)中间体 "
},
{
"bbox": [
14,
0,
1298,
216
],
"type": "inline_equation",
"content": "\\alpha"
},
{
"bbox": [
14,
0,
1298,
216
],
"type": "text",
"content": "-溴代邻氯苯乙酮(1)的制备 (张宝丰. 中国医药工业杂志, 1989, 20:33):"
}
]
}
],
"index": 0
},
{
"type": "image",
"bbox": [
59,
241,
691,
464
],
"blocks": [
{
"bbox": [
59,
241,
691,
464
],
"lines": [
{
"bbox": [
59,
241,
691,
464
],
"spans": [
{
"bbox": [
59,
241,
691,
464
],
"type": "image",
"image_path": "13d3f822a19e58284df7e8f6375844713ec35907bdd2e920ebf752ecfc2c1b95.jpg"
}
]
}
],
"index": 1,
"angle": 0,
"type": "image_body"
}
],
"index": 1
},
{
"bbox": [
19,
556,
1290,
769
],
"type": "text",
"angle": 0,
"lines": [
{
"bbox": [
19,
556,
1290,
769
],
"spans": [
{
"bbox": [
19,
556,
1290,
769
],
"type": "text",
"content": "反应时经常有一个诱导期,因为反应开始时烯醇化速率较慢。随着反应的进行,卤化氢浓度增大,烯醇化速率加快,反应也相应加快。反应初期,可加入少量氢卤酸以缩短诱导期,光照也常起到明显的催化效果。Lewis酸对某些反应有催化作用,例如苯乙酮的溴化,在催化量的三氯化铝存在下生成 "
},
{
"bbox": [
19,
556,
1290,
769
],
"type": "inline_equation",
"content": "\\alpha"
},
{
"bbox": [
19,
556,
1290,
769
],
"type": "text",
"content": "-溴代苯乙酮,而三氯化铝过量时生成间-溴苯乙酮。"
}
]
}
],
"index": 2
},
{
"type": "image",
"bbox": [
59,
798,
950,
1072
],
"blocks": [
{
"bbox": [
59,
798,
950,
1072
],
"lines": [
{
"bbox": [
59,
798,
950,
1072
],
"spans": [
{
"bbox": [
59,
798,
950,
1072
],
"type": "image",
"image_path": "b1459176fc74696f7ac62ec4c74a5593cc9087862c96fbcc27be5e98a3933f39.jpg"
}
]
}
],
"index": 3,
"angle": 0,
"type": "image_body"
}
],
"index": 3
}
],
"discarded_blocks": [],
"page_size": [
1300,
1076
],
"page_idx": 0
}
],
"_backend": "vlm",
"_version_name": "2.6.3"
}
@@ -0,0 +1,48 @@
[
[
{
"type": "text",
"bbox": [
0.011,
0.0,
0.999,
0.201
],
"angle": 0,
"content": "首先是羰基氧原子接受质子, 而后生成烯醇。决定反应速率的步骤是生成烯醇的一步, 生成烯醇后, 卤素作为亲电试剂与烯醇的双键发生亲电加成, 生成 \\(\\alpha\\)-卤代羰基化合物和卤化氢, 所以酸催化常常是自催化。例如治疗支气管炎、喘息性支气管炎药物氯丙那林(Clorprenaline)中间体 \\(\\alpha\\)-溴代邻氯苯乙酮(1)的制备 (张宝丰. 中国医药工业杂志, 1989, 20:33):"
},
{
"type": "image",
"bbox": [
0.046,
0.224,
0.532,
0.432
],
"angle": 0,
"content": null
},
{
"type": "text",
"bbox": [
0.015,
0.517,
0.993,
0.715
],
"angle": 0,
"content": "反应时经常有一个诱导期,因为反应开始时烯醇化速率较慢。随着反应的进行,卤化氢浓度增大,烯醇化速率加快,反应也相应加快。反应初期,可加入少量氢卤酸以缩短诱导期,光照也常起到明显的催化效果。Lewis酸对某些反应有催化作用,例如苯乙酮的溴化,在催化量的三氯化铝存在下生成 \\(\\alpha\\)-溴代苯乙酮,而三氯化铝过量时生成间-溴苯乙酮。"
},
{
"type": "image",
"bbox": [
0.046,
0.742,
0.731,
0.997
],
"angle": 0,
"content": null
}
]
]
Binary file not shown.

After

Width:  |  Height:  |  Size: 345 KiB