lix/mnote
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix knowledge rag docx sidecar locator

Browse Source
This commit is contained in:
lix-2026
2026-06-09 18:40:48 +08:00
parent 516121e6f9
commit 6ef233772e
6 changed files with 338 additions and 194 deletions
Download Patch File Download Diff File Expand all files Collapse all files
design/07-ai/process/7-56-lightrag-native-docx-sidecar-addressable-blocks-v1.md → design/07-ai/done/7-56-lightrag-native-docx-sidecar-addressable-blocks-v1.md
+84 -12
View File
@@ -2,7 +2,7 @@
> 创建时间：2026-06-09
>
> 当前状态：`process`
> 当前状态：`done`
>
> Owner：07-ai / knowledge-rag / LightRAG native parser / office-preview
>
@@ -118,15 +118,86 @@ PDF 对照样本：
MinerU parse 已能输出 `positions.type=bbox` 与 page anchor，PDF 当前先消费现有 bbox，不作为 7-56 首要补丁对象。
## 5. 后续收口
## 5. 完成实现
- MNote `knowledge_rag` mapper 应优先消费 provider sidecar 中的 `blockid/positions`，逐步退役 query-time quote 反查 block。
- office-preview 应优先使用 `paraid.anchor` 或 paraid range 定位；没有可用 paraId 时再用 `text_fingerprint/displayQuote` 做降级匹配。
2026-06-09 已完成 DOCX 查询与点击定位主链收口。
### 5.1 LightRAG provider 合同
- `/query/search` 保持关键词检索入口，`SearchRequest.query` 最小长度为 2。
- `/query/search` 新增 `include_sidecar=true`，结果 chunk 返回原始 `sidecar` 与 `heading`。
- 返回 chunk 保留 `source_chunk_id`，`chunk_id` 可继续用 `#match-N` 表示同一 chunk 内第 N 个关键词命中。
- LightRAG `kv_store_text_chunks.json` 中的 paragraph semantic chunk 已携带 `sidecar.refs`，例如 `doc-ff0b60997a285a85e5704a114d7b3ffa-chunk-158` 返回 52 个 block refs，MNote 不再需要凭完整 chunk 文本猜测来源 block。
### 5.2 MNote citation / locator 合同
- `/api/knowledge-rag/search` exact 模式请求 LightRAG `/query/search` 时显式传 `include_sidecar=true`。
- `map_reference_plan(...)` 优先消费 LightRAG chunk `sidecar.refs`，按 `occurrenceIndex` 在 refs 指向的 blocks 中精确选择含 query 的 block。
- `find_lightrag_sidecar_block(...)` 已从“文本打分匹配”改为“sidecar.refs 精确 block 选择”；没有 refs 或 refs 中没有 query 命中时不伪造精确 locator。
- `displayQuote` / `locatorEvidenceText` 使用选中的 sidecar block 文本，搜索结果不再展示整段 chunk window 导致的重复、错位上下文。
- `sourceChunkId` 被写入 mapped reference，便于诊断 match chunk 与原始 LightRAG chunk 的对应关系。
### 5.3 Office preview 定位
- `mnote.open_resource_locator` 透传 `paragraphOrdinal / paraIdStart / paraIdEnd / textFingerprint / query / searchQuery`。
- 同一 DOCX 点击不同搜索结果时，resource tab 的 base href 比较会剔除定位参数，避免把同一资源误判成新资源而重新加载。
- DOCX preview 优先用 `evidenceText` 精确段落锚定，再考虑 paragraph ordinal 附近搜索；避免短 query 命中相邻段落。
- `page=0` 不再写入 Office preview URL 或 panel state，避免污染定位状态。
### 5.4 验证证据
LightRAG API 证据：
```bash
curl -H "X-API-Key: $LIGHTRAG_API_KEY" \
-H 'Content-Type: application/json' \
http://127.0.0.1:9621/query/search \
-d '{"query":"吡咯烷","limit":2,"max_per_chunk":2,"include_chunk_content":true,"include_sidecar":true}'
```
结果摘要：
```text
chunk_id=doc-ff0b60997a285a85e5704a114d7b3ffa-chunk-158#match-0
source_chunk_id=doc-ff0b60997a285a85e5704a114d7b3ffa-chunk-158
sidecar.refs=52
first_ref=700ee42acf09aadc31f028f523f52fa9
```
MNote 验证：
```bash
cargo test -p mnote-web knowledge_rag -- --nocapture
MNOTE_KNOWLEDGE_RAG_PYRROLIDINE_QUERY='吡咯烷' MNOTE_KNOWLEDGE_RAG_TOP_N=5 \
node scripts/task543-knowledge-rag-pyrrolidine-top5-locator-smoke.js
MNOTE_KNOWLEDGE_RAG_REPOSITION_QUERY='吡咯烷' \
node scripts/task541-knowledge-rag-search-panel-office-reposition-smoke.js
```
结果：
```text
cargo test: 51 passed
task543: 前 5 个搜索结果 apiBlockId/uiBlockId/panelBlockId 一致
task543: 第 3 条已定位到 “吡咯烷,5 h, 90%”，不再跳到下一段
task541: 搜索面板关闭后状态保持；同一 DOCX 第二次定位 loadCount=0
```
用户验收：
```text
2026-06-09：用户确认本次 DOCX 的查询合格。
```
## 6. 后续收口
- MNote `knowledge_rag` mapper 已优先消费 provider sidecar refs；后续只需继续把 hybrid/mix 查询返回的 chunk provenance 也推向同一合同。
- office-preview 已优先用 sidecar block 的 evidenceText 精确段落锚定；后续可继续增强真实 `paraId` DOM 标注，减少 paragraph ordinal 对 docx-preview DOM 的依赖。
- 重新索引 DOCX 后才能让旧资料库命中使用新 sidecar 粒度。
- LightRAG 服务需要重启后才会使用本地源码改动。
- 本轮已执行 `systemctl --user restart mnote-lightrag.service`，`/health` 返回 healthy；当前 parser routing 仍是 `*:native-iteP,*:mineru-iteP,*:legacy-R`，DOCX 默认走 native。
## 6. MNote 与 LightRAG 的新分工
## 7. MNote 与 LightRAG 的新分工
LightRAG 已提供 keyword / vector / graph / hybrid 检索、rerank、query 与 chunk/reference 返回。MNote 不应继续维护第二套召回、排序或 sidecar exact search 作为主路径。
@@ -160,13 +231,14 @@ LightRAG
- MNote query-time sidecar quote 反查
- 旧 `docs_search` / `mnote.evidence.search` / LiteParse local evidence search
## 7. 退役计划
## 8. 退役计划
阶段 A：完成 provider positions 消费。
阶段 A：完成 provider positions 消费。已完成 DOCX exact search 主链：
- `knowledge_rag` locator 解析 `positions.type=bbox / paraid / text_fingerprint`。
- citation URL 与 resource tab 透传 `paragraphOrdinal / paraIdStart / paraIdEnd / textFingerprint`。
- office-preview 优先按段落定位，再回退到当前文本定位。
- [x] `knowledge_rag` locator 解析 `positions.type=bbox / paraid / text_fingerprint`。
- [x] citation URL 与 resource tab 透传 `paragraphOrdinal / paraIdStart / paraIdEnd / textFingerprint`。
- [x] office-preview 优先按 sidecar block 的 evidenceText 定位段落，再按 paragraph ordinal 附近校验定位。
- [x] LightRAG `/query/search` 返回 `sidecar.refs`，MNote exact search 不再使用 query-time 文本打分反查。
阶段 B：退役 query-time sidecar 反查与旧 evidence/docs 搜索。
@@ -183,14 +255,14 @@ LightRAG
- 保留 source-map / locator 兼容读取，用于历史索引和旧 OCR 结果打开。
- 新索引统一由 LightRAG parser sidecar 产出定位信息。
## 8. 不做
## 9. 不做
- 不在 MNote 新增一套 DOCX parser。
- 不把 DOCX 默认切到 Docling。
- 不在 MNote 实现 rerank；rerank 仍走 LightRAG 原生能力。
- 不把 DOCX 无 bbox 伪造成 PDF 页内精确定位。
## 9. Agent / MCP 边界
## 10. Agent / MCP 边界
LightRAG 官方内核不提供 Skill/MCP/plugin 系统；社区已有 `lightragmcp`、`mcp-lightrag` 等 MCP server，可把 LightRAG query、文档管理、图谱查询包装成 Agent 工具。这一层适合接给 Reasonix / Hermes / Claude Desktop 这类 Agent runtime，但它不能替代 MNote 的 source registry 与引用打开能力。
rust/crates/mnote-web/browser/document-resource-tab-runtime.js
+5 -3
View File
@@ -1020,7 +1020,8 @@ export const createResourceTabRuntime = (dependencies = {}) => {
if (!(frame instanceof HTMLIFrameElement) || !locator) return;
try {
const url = new URL(frame.getAttribute('src') || frame.src || '', window.location.origin);
if (Number.isFinite(Number(locator.page))) url.searchParams.set('page', String(Number(locator.page)));
const locatorPage = Number(locator.page);
if (Number.isFinite(locatorPage) && locatorPage > 0) url.searchParams.set('page', String(locatorPage));
const bbox = evidenceBBoxParam(locator.bbox);
if (bbox) url.searchParams.set('bbox', bbox);
if (locator.sourceMapPath) url.searchParams.set('sourceMapPath', String(locator.sourceMapPath));
@@ -1129,7 +1130,8 @@ export const createResourceTabRuntime = (dependencies = {}) => {
entry.evidenceLocator = locator;
entry.panel.setAttribute('data-mnote-evidence-locator', JSON.stringify(locator));
entry.panel.setAttribute('data-mnote-evidence-open', 'true');
if (Number.isFinite(Number(locator.page))) entry.panel.setAttribute('data-mnote-evidence-page', String(Number(locator.page)));
const locatorPage = Number(locator.page);
if (Number.isFinite(locatorPage) && locatorPage > 0) entry.panel.setAttribute('data-mnote-evidence-page', String(locatorPage));
if (locator.blockId) entry.panel.setAttribute('data-mnote-evidence-block-id', String(locator.blockId));
if (locator.evidenceText) entry.panel.setAttribute('data-mnote-evidence-text', String(locator.evidenceText).slice(0, 240));
if (locator.sourceMapPath) entry.panel.setAttribute('data-mnote-evidence-source-map-path', String(locator.sourceMapPath));
@@ -2323,7 +2325,7 @@ export const createResourceTabRuntime = (dependencies = {}) => {
if (!raw) return '';
try {
const url = new URL(raw, window.location.origin);
['page', 'bbox', 'sourceMapPath', 'blockId', 'lineRange', 'charRange', 'evidenceText', 'query', 'searchQuery', 'mnoteResourceReload'].forEach((key) => {
['page', 'bbox', 'sourceMapPath', 'blockId', 'lineRange', 'charRange', 'paragraphOrdinal', 'paraIdStart', 'paraIdEnd', 'textFingerprint', 'evidenceText', 'query', 'searchQuery', 'mnoteResourceReload'].forEach((key) => {
url.searchParams.delete(key);
});
return url.pathname + '?' + url.searchParams.toString();
rust/crates/mnote-web/browser/sidebar-filetree-open-runtime.js
+6
View File
@@ -317,6 +317,12 @@ export const createSidebarFileTreeOpenRuntime = (dependencies = {}) => {
sourceMapPath: String(input && input.sourceMapPath || '').trim(),
blockId: String(input && input.blockId || '').trim(),
evidenceText: String(input && input.evidenceText || '').trim(),
query: String(input && input.query || '').trim(),
searchQuery: String(input && input.searchQuery || '').trim(),
paragraphOrdinal: String(input && input.paragraphOrdinal || '').trim(),
paraIdStart: String(input && input.paraIdStart || '').trim(),
paraIdEnd: String(input && input.paraIdEnd || '').trim(),
textFingerprint: String(input && input.textFingerprint || '').trim(),
lineRange: input && input.lineRange,
charRange: input && input.charRange
});
rust/crates/mnote-web/src/routes/knowledge_rag.rs
+226 -165
View File
@@ -619,6 +619,7 @@ pub async fn search(
"limit": body.top_k.or(body.chunk_top_k).unwrap_or(50),
"max_per_chunk": 12,
"include_chunk_content": body.include_chunk_content.unwrap_or(true),
"include_sidecar": true,
})),
true,
&context,
@@ -1597,38 +1598,51 @@ async fn lightrag_json(
WebError::internal(format!("LightRAG HTTP client 初始化失败: {error}"))
.with_context(context)
})?;
let mut request = client.request(method, &url);
if use_api_key {
if let Some(api_key) = lightrag_api_key() {
let api_key = use_api_key.then(lightrag_api_key).flatten();
for attempt in 0..2 {
let mut request = client.request(method.clone(), &url);
if let Some(api_key) = api_key.as_deref() {
request = request.header("X-API-Key", api_key);
}
if let Some(body) = body.as_ref() {
request = request.json(body);
}
let response = request.send().await.map_err(|error| {
WebError::bad_gateway_code(
"knowledge_rag_lightrag_unreachable",
format!("无法访问 LightRAG provider: {error}"),
)
.with_context(context)
})?;
let status = response.status();
let text = response.text().await.unwrap_or_default();
if !status.is_success() {
if attempt == 0 && should_retry_lightrag_provider_error(status, &text) {
tokio::time::sleep(Duration::from_millis(700)).await;
continue;
}
return Err(WebError::bad_gateway_code(
"knowledge_rag_lightrag_error",
format!("LightRAG provider 返回 HTTP {status}: {text}"),
)
.with_context(context));
}
return serde_json::from_str(&text).map_err(|error| {
WebError::bad_gateway_code(
"knowledge_rag_lightrag_json_invalid",
format!("LightRAG provider 返回非 JSON 响应: {error}"),
)
.with_context(context)
});
}
if let Some(body) = body {
request = request.json(&body);
}
let response = request.send().await.map_err(|error| {
WebError::bad_gateway_code(
"knowledge_rag_lightrag_unreachable",
format!("无法访问 LightRAG provider: {error}"),
)
.with_context(context)
})?;
let status = response.status();
let text = response.text().await.unwrap_or_default();
if !status.is_success() {
return Err(WebError::bad_gateway_code(
"knowledge_rag_lightrag_error",
format!("LightRAG provider 返回 HTTP {status}: {text}"),
)
.with_context(context));
}
serde_json::from_str(&text).map_err(|error| {
WebError::bad_gateway_code(
"knowledge_rag_lightrag_json_invalid",
format!("LightRAG provider 返回非 JSON 响应: {error}"),
)
.with_context(context)
})
unreachable!("lightrag retry loop returns on every branch")
}
fn should_retry_lightrag_provider_error(status: StatusCode, body: &str) -> bool {
status.is_server_error()
&& (body.contains("RetryError")
|| body.contains("InvalidResponseError")
|| body.contains("Received empty content"))
}
fn mapped_references(
@@ -1783,19 +1797,29 @@ fn map_reference_plan(
.or_else(|| primary_chunk.and_then(|chunk| chunk.get("chunk_id")))
.cloned()
.unwrap_or(Value::Null);
let source_chunk_id = reference
.get("source_chunk_id")
.or_else(|| primary_chunk.and_then(|chunk| chunk.get("source_chunk_id")))
.cloned()
.or_else(|| {
chunk_id
.as_str()
.and_then(|value| value.split_once("#match-").map(|(base, _)| json!(base)))
})
.unwrap_or(Value::Null);
let occurrence_index = reference
.get("occurrence_index")
.or_else(|| primary_chunk.and_then(|chunk| chunk.get("occurrence_index")))
.cloned()
.unwrap_or(Value::Null);
let chunk_sidecar =
lightrag_reference_sidecar(reference, primary_chunk, &source_chunk_id, &chunk_id);
let (quote, quote_source) = primary_chunk
.and_then(|chunk| chunk.get("content"))
.and_then(Value::as_str)
.map(|value| (value.to_owned(), "chunk"))
.or_else(|| {
chunk_id
.as_str()
.and_then(lightrag_chunk_content_for_id)
lightrag_chunk_content_for_reference_ids(&source_chunk_id, &chunk_id)
.map(|value| (value, "kv_store"))
})
.map(|(value, source)| {
@@ -1806,9 +1830,7 @@ fn map_reference_plan(
})
.map_or((None, "missing"), |(value, source)| (Some(value), source));
let locator_sidecar_block = entry.and_then(|entry| {
quote
.as_deref()
.and_then(|quote| find_lightrag_sidecar_block(entry, quote, query))
find_lightrag_sidecar_block(entry, query, &occurrence_index, chunk_sidecar.as_ref())
});
let locator_sidecar_text = locator_sidecar_block
.as_ref()
@@ -1824,17 +1846,16 @@ fn map_reference_plan(
);
let content_diagnostics = quote_content_diagnostics(quote.as_deref(), quote_source);
let locator = entry.and_then(|entry| {
quote.as_deref().and_then(|quote| {
lightrag_locator_for_reference(
root_path,
root_uri,
entry,
&chunk_id,
quote,
query,
Some(&text_bundle.locator_evidence_text),
)
})
lightrag_locator_for_reference(
root_path,
root_uri,
entry,
&chunk_id,
query,
Some(&text_bundle.locator_evidence_text),
&occurrence_index,
chunk_sidecar.as_ref(),
)
});
let locator_precision = locator
.as_ref()
@@ -1880,6 +1901,7 @@ fn map_reference_plan(
"reference": reference,
"filePath": file_path,
"chunkId": chunk_id,
"sourceChunkId": source_chunk_id,
"occurrenceIndex": occurrence_index,
"rawQuote": text_bundle.raw_quote.clone(),
"displayQuote": text_bundle.display_quote.clone(),
@@ -2091,7 +2113,10 @@ fn citation_text_bundle(
} else {
locator_raw.chars().take(700).collect::<String>()
};
let display_quote = clean_lightrag_text_for_display(&raw_quote);
let display_source = locator_source_text
.filter(|value| !value.trim().is_empty())
.unwrap_or(&raw_quote);
let display_quote = clean_lightrag_text_for_display(display_source);
let locator_evidence_text = query_centered_clean_window(
&clean_lightrag_text_for_locator(&locator_window),
&search_query,
@@ -2301,11 +2326,12 @@ fn lightrag_locator_for_reference(
root_uri: &str,
entry: &KnowledgeRagSourceRegistryEntry,
chunk_id: &Value,
quote: &str,
query: Option<&str>,
locator_evidence_text: Option<&str>,
occurrence_index: &Value,
chunk_sidecar: Option<&Value>,
) -> Option<EvidenceLocator> {
let block = find_lightrag_sidecar_block(entry, quote, query)?;
let block = find_lightrag_sidecar_block(entry, query, occurrence_index, chunk_sidecar)?;
let positions = block
.get("positions")
.and_then(Value::as_array)
@@ -2324,23 +2350,7 @@ fn lightrag_locator_for_reference(
.and_then(|query| query_centered_quote(block_content, query, 700))
.map(|value| clean_lightrag_text_for_locator(&value))
})
.or_else(|| {
query_centered_quote(block_content, quote, 700)
.map(|value| clean_lightrag_text_for_locator(&value))
})
.unwrap_or_else(|| {
clean_lightrag_text_for_locator(&quote.chars().take(700).collect::<String>())
});
let fallback_evidence_text = block
.get("content")
.and_then(Value::as_str)
.map(|value| clean_lightrag_text_for_locator(&value.chars().take(700).collect::<String>()))
.unwrap_or_default();
let evidence_text = if evidence_text.trim().is_empty() {
fallback_evidence_text
} else {
evidence_text
};
.unwrap_or_else(|| clean_lightrag_text_for_locator(block_content));
let mut open_params = json!({
"rootUri": root_uri,
"resourcePath": resource_path,
@@ -2494,109 +2504,114 @@ fn value_to_u32(value: &Value) -> Option<u32> {
fn find_lightrag_sidecar_block(
entry: &KnowledgeRagSourceRegistryEntry,
quote: &str,
query: Option<&str>,
occurrence_index: &Value,
chunk_sidecar: Option<&Value>,
) -> Option<Value> {
let quote_normalized = normalize_text_for_match(quote);
if quote_normalized.is_empty() {
let ref_ids = lightrag_sidecar_ref_ids(chunk_sidecar?);
if ref_ids.is_empty() {
return None;
}
let query_normalized = query
.map(normalize_text_for_match)
.filter(|value| !value.is_empty());
let query_terms = query_normalized
.as_deref()
.map(query_match_terms)
.unwrap_or_default();
let query_window = query_normalized
.as_deref()
.and_then(|query| query_centered_quote(&quote_normalized, query, 160));
let path = sidecar_blocks_path(entry)?;
let content = fs::read_to_string(path).ok()?;
let mut best_query_block: Option<(usize, Value)> = None;
for line in content.lines() {
let block = serde_json::from_str::<Value>(line).ok()?;
if block.get("positions").and_then(Value::as_array).is_none() {
let blocks = lightrag_sidecar_blocks_by_id(entry, &ref_ids)?;
let query = query
.map(str::trim)
.filter(|value| !value.is_empty())
.map(ToOwned::to_owned);
let Some(query) = query else {
return ref_ids.into_iter().find_map(|id| blocks.get(&id).cloned());
};
let target_occurrence = occurrence_index.as_u64().unwrap_or(0);
let mut seen_occurrences = 0u64;
for id in ref_ids {
let Some(block) = blocks.get(&id) else {
continue;
}
let block_text = block
};
let count = block
.get("content")
.and_then(Value::as_str)
.unwrap_or_default();
let block_normalized = normalize_text_for_match(block_text);
if block_normalized.is_empty() {
.map(|content| exact_match_count(content, &query))
.unwrap_or(0);
if count == 0 {
continue;
}
let block_contains_query = query_normalized.as_deref().is_some_and(|query_normalized| {
block_normalized
.to_ascii_lowercase()
.contains(&query_normalized.to_ascii_lowercase())
|| query_terms
.iter()
.any(|term| block_matches_query_term(&block_normalized, term))
});
if query_normalized.is_some() {
if block_contains_query {
let score = query_window
.as_deref()
.map(|window| {
let window_prefix = match_prefix(window, 40);
let exact_window_bonus = if !window_prefix.trim().is_empty()
&& block_normalized.contains(&window_prefix)
{
100_000
} else {
0
};
exact_window_bonus
+ common_char_substring_score(&block_normalized, window) * 10
})
.unwrap_or(0)
+ common_char_prefix_len(&block_normalized, &quote_normalized)
+ common_char_substring_score(&block_normalized, &quote_normalized);
if score
> best_query_block
.as_ref()
.map(|(score, _)| *score)
.unwrap_or(0)
{
best_query_block = Some((score, block.clone()));
}
} else {
continue;
if seen_occurrences + count > target_occurrence {
return Some(block.clone());
}
seen_occurrences += count;
}
None
}
fn lightrag_sidecar_ref_ids(sidecar: &Value) -> Vec<String> {
let mut ids = Vec::new();
if let Some(id) = sidecar.get("id").and_then(Value::as_str) {
ids.push(id.trim().to_string());
}
if let Some(refs) = sidecar.get("refs").and_then(Value::as_array) {
for item in refs {
if let Some(id) = item.get("id").and_then(Value::as_str) {
ids.push(id.trim().to_string());
}
}
if block_normalized.contains(&quote_normalized)
|| quote_normalized.contains(&block_normalized)
|| quote_normalized.contains(&match_prefix(&block_normalized, 80))
|| block_normalized.contains(&match_prefix(&quote_normalized, 80))
{
return Some(block);
}
ids.retain(|id| !id.is_empty());
ids.dedup();
ids
}
fn lightrag_sidecar_blocks_by_id(
entry: &KnowledgeRagSourceRegistryEntry,
ref_ids: &[String],
) -> Option<BTreeMap<String, Value>> {
let wanted = ref_ids.iter().cloned().collect::<BTreeSet<_>>();
let path = sidecar_blocks_path(entry)?;
let content = fs::read_to_string(path).ok()?;
let mut blocks = BTreeMap::new();
for line in content.lines() {
let block = serde_json::from_str::<Value>(line).ok()?;
let Some(block_id) = block.get("blockid").and_then(Value::as_str) else {
continue;
};
if wanted.contains(block_id) && block.get("positions").and_then(Value::as_array).is_some() {
blocks.insert(block_id.to_string(), block);
}
if blocks.len() == wanted.len() {
break;
}
}
best_query_block.map(|(_, block)| block)
Some(blocks)
}
fn common_char_prefix_len(left: &str, right: &str) -> usize {
left.chars()
.zip(right.chars())
.take_while(|(left, right)| left == right)
.count()
fn exact_match_count(content: &str, query: &str) -> u64 {
if query.is_empty() {
return 0;
}
let content_lower = content.to_lowercase();
let query_lower = query.to_lowercase();
let mut count = 0u64;
let mut start = 0usize;
while let Some(index) = content_lower[start..].find(&query_lower) {
count += 1;
start += index + query_lower.len().max(1);
}
count
}
fn common_char_substring_score(left: &str, right: &str) -> usize {
let right_prefixes = right
.chars()
.collect::<Vec<_>>()
.windows(40.min(right.chars().count()))
.map(|window| window.iter().collect::<String>())
.collect::<Vec<_>>();
right_prefixes
.iter()
.filter(|candidate| !candidate.trim().is_empty() && left.contains(candidate.as_str()))
.map(|candidate| candidate.chars().count())
.max()
.unwrap_or(0)
fn lightrag_reference_sidecar(
reference: &Value,
primary_chunk: Option<&Value>,
source_chunk_id: &Value,
chunk_id: &Value,
) -> Option<Value> {
reference
.get("sidecar")
.cloned()
.or_else(|| {
primary_chunk
.and_then(|chunk| chunk.get("sidecar"))
.cloned()
})
.or_else(|| lightrag_chunk_sidecar_for_reference_ids(source_chunk_id, chunk_id))
}
fn block_matches_query_term(block_normalized: &str, term: &str) -> bool {
@@ -2712,13 +2727,6 @@ fn normalize_text_for_match(value: &str) -> String {
.collect::<String>()
}
fn match_prefix(value: &str, limit: usize) -> String {
value
.chars()
.take(limit.min(value.chars().count()))
.collect()
}
fn sidecar_blocks_path(entry: &KnowledgeRagSourceRegistryEntry) -> Option<PathBuf> {
for file_path in lightrag_sidecar_file_path_candidates(entry) {
let stem = Path::new(&file_path).file_stem()?.to_string_lossy();
@@ -2841,12 +2849,56 @@ fn sidecar_text_stats(path: &Path) -> Result<SidecarTextStats, WebError> {
Ok(stats)
}
fn lightrag_chunk_content_for_id(chunk_id: &str) -> Option<String> {
fn lightrag_chunk_content_for_reference_ids(
source_chunk_id: &Value,
chunk_id: &Value,
) -> Option<String> {
lightrag_chunk_id_candidates(source_chunk_id, chunk_id)
.into_iter()
.find_map(|id| lightrag_chunk_content_for_id(&id))
}
fn lightrag_chunk_sidecar_for_reference_ids(
source_chunk_id: &Value,
chunk_id: &Value,
) -> Option<Value> {
lightrag_chunk_id_candidates(source_chunk_id, chunk_id)
.into_iter()
.find_map(|id| {
lightrag_chunk_value_for_id(&id).and_then(|chunk| chunk.get("sidecar").cloned())
})
}
fn lightrag_chunk_id_candidates(source_chunk_id: &Value, chunk_id: &Value) -> Vec<String> {
let mut ids = Vec::new();
for value in [source_chunk_id, chunk_id] {
let Some(raw) = value
.as_str()
.map(str::trim)
.filter(|value| !value.is_empty())
else {
continue;
};
ids.push(raw.to_string());
if let Some((base, _)) = raw.split_once("#match-") {
ids.push(base.to_string());
}
}
ids.sort();
ids.dedup();
ids
}
fn lightrag_chunk_value_for_id(chunk_id: &str) -> Option<Value> {
let content =
fs::read_to_string(lightrag_working_dir().join("kv_store_text_chunks.json")).ok()?;
let chunks = serde_json::from_str::<Value>(&content).ok()?;
chunks
.get(chunk_id)
chunks.get(chunk_id).cloned()
}
fn lightrag_chunk_content_for_id(chunk_id: &str) -> Option<String> {
lightrag_chunk_value_for_id(chunk_id)
.as_ref()
.and_then(|chunk| chunk.get("content"))
.and_then(Value::as_str)
.map(ToOwned::to_owned)
@@ -4588,7 +4640,14 @@ mod tests {
"chunk_id": "doc1-chunk-256",
"chunks": [{
"chunk_id": "doc1-chunk-256",
"content": chunk_content
"content": chunk_content,
"sidecar": {
"type": "block",
"refs": [
{"type": "block", "id": "wrong-index"},
{"type": "block", "id": "tes"}
]
}
}]
}),
&registry,
@@ -4869,9 +4928,10 @@ mod tests {
"file:///tmp/root",
&entry,
&json!("doc1-chunk-000"),
"# START: This is the first image in PDF\nThis is text BEFORE the image.",
None,
None,
&json!(0),
Some(&json!({"type":"block","refs":[{"type":"block","id":"block1"}]})),
)
.expect("locator");
@@ -4927,9 +4987,10 @@ mod tests {
"file:///tmp/root",
&entry,
&json!("doc1-chunk-000"),
"监管政策变革",
Some("监管政策变革"),
None,
&json!(0),
Some(&json!({"type":"block","refs":[{"type":"block","id":"block-docx-1"}]})),
)
.expect("locator");
rust/crates/mnote-web/src/routes/web_shell.rs
+11 -8
View File
@@ -1585,7 +1585,7 @@ pub async fn office_preview_page(Query(query): Query<OfficePreviewQuery>) -> Res
for (const element of elements) {{
const compactElement = compactEvidenceText(element.textContent || '');
if ((compactAnchor.length >= 6 || shortQueryAnchor) && compactElement.includes(compactAnchor)) {{
return markEvidenceTarget(shortQueryAnchor ? shortLeadingAnchorTarget(element, elements, elements.indexOf(element)) : element);
return markEvidenceTarget(element);
}}
if (
compactAnchorWithoutNumbers.length >= 8
@@ -1631,11 +1631,7 @@ pub async fn office_preview_page(Query(query): Query<OfficePreviewQuery>) -> Res
const text = normalizeEvidenceText(element.textContent || '');
if (!text || text.length < 3 || text.length > 6000) return 0;
const compactText = compactEvidenceText(text);
const compactSearchQuery = compactEvidenceText(evidenceSearchQuery);
let score = 0;
if (compactSearchQuery.length >= 2 && compactText.includes(compactSearchQuery)) {{
score += 2400;
}}
for (const anchor of anchors) {{
const compactAnchor = compactEvidenceText(anchor);
if (!compactAnchor || compactAnchor.length < 4) continue;
@@ -1771,7 +1767,14 @@ pub async fn office_preview_page(Query(query): Query<OfficePreviewQuery>) -> Res
if (!best || score > best.score) best = {{ element, score }};
if (score >= 2400 && evidenceElementMatchesSearchQuery(element)) break;
}}
return best ? markEvidenceTarget(best.element) : false;
if (!best) return false;
if (!anchors.length) {{
return evidenceElementMatchesSearchQuery(best.element) ? markEvidenceTarget(best.element) : false;
}}
const threshold = Math.max(180, Math.min(800, anchors[0].length * 4));
if (best.score >= threshold) return markEvidenceTarget(best.element);
if (best.score > 0 && evidenceElementMatchesSearchQuery(best.element)) return markEvidenceTarget(best.element);
return false;
}}
function scrollToEvidencePageFallback() {{
@@ -1786,15 +1789,15 @@ pub async fn office_preview_page(Query(query): Query<OfficePreviewQuery>) -> Res
try {{
const sourceMap = await fetchEvidenceSourceMap();
const block = findEvidenceBlockInSourceMap(sourceMap);
if (evidenceParagraphOrdinal && scrollToEvidenceParagraphOrdinal()) return;
if (evidenceText && scrollToEvidenceParagraph(evidenceText)) return;
if (evidenceParagraphOrdinal && scrollToEvidenceParagraphOrdinal()) return;
if (block && scrollToEvidenceParagraph(block.text)) return;
if (evidenceText && scrollToEvidenceTextCandidates(evidenceText)) return;
if (block && scrollToEvidenceTextCandidates(block.text)) return;
if (scrollToEvidenceCoordinate(sourceMap, block)) return;
}} catch (_) {{}}
if (evidenceParagraphOrdinal && scrollToEvidenceParagraphOrdinal()) return;
if (evidenceText && scrollToEvidenceParagraph(evidenceText)) return;
if (evidenceParagraphOrdinal && scrollToEvidenceParagraphOrdinal()) return;
if (evidenceText && scrollToEvidenceTextCandidates(evidenceText)) return;
scrollToEvidencePageFallback();
}}
scripts/task543-knowledge-rag-pyrrolidine-top5-locator-smoke.js
+6 -6
View File
@@ -70,7 +70,7 @@ async function apiSearch(context) {
rootUri: ROOT_URI,
workspaceId: WORKSPACE_ID,
query: QUERY,
mode: "mix",
mode: "exact",
topK: 20,
chunkTopK: 20,
includeChunkContent: true,
@@ -178,6 +178,8 @@ async function clickAndAuditRow(page, index, apiResult) {
const evidenceText = locator?.openAction?.params?.evidenceText || apiResult?.locator?.openAction?.params?.evidenceText || apiResult?.quote || rowText;
const overlap = longestCommonSubstringLength(state.highlightedText, evidenceText);
const compactHighlighted = compactText(state.highlightedText);
const compactEvidence = compactText(evidenceText);
assert.equal(state.panelBlockId, locator.blockId, `第 ${index + 1} 条 panel blockId 不一致: ${JSON.stringify(state, null, 2)}`);
assert(state.highlightedText.includes(QUERY), `第 ${index + 1} 条定位高亮未包含 query: ${JSON.stringify(state, null, 2)}`);
assert(
@@ -185,11 +187,9 @@ async function clickAndAuditRow(page, index, apiResult) {
`第 ${index + 1} 条仍是短词高亮，不是段落级高亮: ${JSON.stringify(state, null, 2)}`,
);
assert(
overlap >= 10 || (
state.highlightedTag === "P"
&& compactText(state.highlightedText).includes(compactText(QUERY))
&& compactText(state.highlightedText).length >= 6
),
overlap >= Math.min(10, Math.max(4, compactEvidence.length))
|| compactHighlighted.includes(compactEvidence)
|| compactEvidence.includes(compactHighlighted),
`第 ${index + 1} 条搜索结果与实际定位上下文不一致: ${JSON.stringify({ rowText, evidenceText, overlap, state }, null, 2)}`,
);