diff --git a/design/07-ai/process/7-56-lightrag-native-docx-sidecar-addressable-blocks-v1.md b/design/07-ai/done/7-56-lightrag-native-docx-sidecar-addressable-blocks-v1.md similarity index 71% rename from design/07-ai/process/7-56-lightrag-native-docx-sidecar-addressable-blocks-v1.md rename to design/07-ai/done/7-56-lightrag-native-docx-sidecar-addressable-blocks-v1.md index 0e91248b..6399b316 100644 --- a/design/07-ai/process/7-56-lightrag-native-docx-sidecar-addressable-blocks-v1.md +++ b/design/07-ai/done/7-56-lightrag-native-docx-sidecar-addressable-blocks-v1.md @@ -2,7 +2,7 @@ > 创建时间:2026-06-09 > -> 当前状态:`process` +> 当前状态:`done` > > Owner:07-ai / knowledge-rag / LightRAG native parser / office-preview > @@ -118,15 +118,86 @@ PDF 对照样本: MinerU parse 已能输出 `positions.type=bbox` 与 page anchor,PDF 当前先消费现有 bbox,不作为 7-56 首要补丁对象。 -## 5. 后续收口 +## 5. 完成实现 -- MNote `knowledge_rag` mapper 应优先消费 provider sidecar 中的 `blockid/positions`,逐步退役 query-time quote 反查 block。 -- office-preview 应优先使用 `paraid.anchor` 或 paraid range 定位;没有可用 paraId 时再用 `text_fingerprint/displayQuote` 做降级匹配。 +2026-06-09 已完成 DOCX 查询与点击定位主链收口。 + +### 5.1 LightRAG provider 合同 + +- `/query/search` 保持关键词检索入口,`SearchRequest.query` 最小长度为 2。 +- `/query/search` 新增 `include_sidecar=true`,结果 chunk 返回原始 `sidecar` 与 `heading`。 +- 返回 chunk 保留 `source_chunk_id`,`chunk_id` 可继续用 `#match-N` 表示同一 chunk 内第 N 个关键词命中。 +- LightRAG `kv_store_text_chunks.json` 中的 paragraph semantic chunk 已携带 `sidecar.refs`,例如 `doc-ff0b60997a285a85e5704a114d7b3ffa-chunk-158` 返回 52 个 block refs,MNote 不再需要凭完整 chunk 文本猜测来源 block。 + +### 5.2 MNote citation / locator 合同 + +- `/api/knowledge-rag/search` exact 模式请求 LightRAG `/query/search` 时显式传 `include_sidecar=true`。 +- `map_reference_plan(...)` 优先消费 LightRAG chunk `sidecar.refs`,按 `occurrenceIndex` 在 refs 指向的 blocks 中精确选择含 query 的 block。 +- `find_lightrag_sidecar_block(...)` 已从“文本打分匹配”改为“sidecar.refs 精确 block 选择”;没有 refs 或 refs 中没有 query 命中时不伪造精确 locator。 +- `displayQuote` / `locatorEvidenceText` 使用选中的 sidecar block 文本,搜索结果不再展示整段 chunk window 导致的重复、错位上下文。 +- `sourceChunkId` 被写入 mapped reference,便于诊断 match chunk 与原始 LightRAG chunk 的对应关系。 + +### 5.3 Office preview 定位 + +- `mnote.open_resource_locator` 透传 `paragraphOrdinal / paraIdStart / paraIdEnd / textFingerprint / query / searchQuery`。 +- 同一 DOCX 点击不同搜索结果时,resource tab 的 base href 比较会剔除定位参数,避免把同一资源误判成新资源而重新加载。 +- DOCX preview 优先用 `evidenceText` 精确段落锚定,再考虑 paragraph ordinal 附近搜索;避免短 query 命中相邻段落。 +- `page=0` 不再写入 Office preview URL 或 panel state,避免污染定位状态。 + +### 5.4 验证证据 + +LightRAG API 证据: + +```bash +curl -H "X-API-Key: $LIGHTRAG_API_KEY" \ + -H 'Content-Type: application/json' \ + http://127.0.0.1:9621/query/search \ + -d '{"query":"吡咯烷","limit":2,"max_per_chunk":2,"include_chunk_content":true,"include_sidecar":true}' +``` + +结果摘要: + +```text +chunk_id=doc-ff0b60997a285a85e5704a114d7b3ffa-chunk-158#match-0 +source_chunk_id=doc-ff0b60997a285a85e5704a114d7b3ffa-chunk-158 +sidecar.refs=52 +first_ref=700ee42acf09aadc31f028f523f52fa9 +``` + +MNote 验证: + +```bash +cargo test -p mnote-web knowledge_rag -- --nocapture +MNOTE_KNOWLEDGE_RAG_PYRROLIDINE_QUERY='吡咯烷' MNOTE_KNOWLEDGE_RAG_TOP_N=5 \ + node scripts/task543-knowledge-rag-pyrrolidine-top5-locator-smoke.js +MNOTE_KNOWLEDGE_RAG_REPOSITION_QUERY='吡咯烷' \ + node scripts/task541-knowledge-rag-search-panel-office-reposition-smoke.js +``` + +结果: + +```text +cargo test: 51 passed +task543: 前 5 个搜索结果 apiBlockId/uiBlockId/panelBlockId 一致 +task543: 第 3 条已定位到 “吡咯烷,5 h, 90%”,不再跳到下一段 +task541: 搜索面板关闭后状态保持;同一 DOCX 第二次定位 loadCount=0 +``` + +用户验收: + +```text +2026-06-09:用户确认本次 DOCX 的查询合格。 +``` + +## 6. 后续收口 + +- MNote `knowledge_rag` mapper 已优先消费 provider sidecar refs;后续只需继续把 hybrid/mix 查询返回的 chunk provenance 也推向同一合同。 +- office-preview 已优先用 sidecar block 的 evidenceText 精确段落锚定;后续可继续增强真实 `paraId` DOM 标注,减少 paragraph ordinal 对 docx-preview DOM 的依赖。 - 重新索引 DOCX 后才能让旧资料库命中使用新 sidecar 粒度。 - LightRAG 服务需要重启后才会使用本地源码改动。 - 本轮已执行 `systemctl --user restart mnote-lightrag.service`,`/health` 返回 healthy;当前 parser routing 仍是 `*:native-iteP,*:mineru-iteP,*:legacy-R`,DOCX 默认走 native。 -## 6. MNote 与 LightRAG 的新分工 +## 7. MNote 与 LightRAG 的新分工 LightRAG 已提供 keyword / vector / graph / hybrid 检索、rerank、query 与 chunk/reference 返回。MNote 不应继续维护第二套召回、排序或 sidecar exact search 作为主路径。 @@ -160,13 +231,14 @@ LightRAG - MNote query-time sidecar quote 反查 - 旧 `docs_search` / `mnote.evidence.search` / LiteParse local evidence search -## 7. 退役计划 +## 8. 退役计划 -阶段 A:完成 provider positions 消费。 +阶段 A:完成 provider positions 消费。已完成 DOCX exact search 主链: -- `knowledge_rag` locator 解析 `positions.type=bbox / paraid / text_fingerprint`。 -- citation URL 与 resource tab 透传 `paragraphOrdinal / paraIdStart / paraIdEnd / textFingerprint`。 -- office-preview 优先按段落定位,再回退到当前文本定位。 +- [x] `knowledge_rag` locator 解析 `positions.type=bbox / paraid / text_fingerprint`。 +- [x] citation URL 与 resource tab 透传 `paragraphOrdinal / paraIdStart / paraIdEnd / textFingerprint`。 +- [x] office-preview 优先按 sidecar block 的 evidenceText 定位段落,再按 paragraph ordinal 附近校验定位。 +- [x] LightRAG `/query/search` 返回 `sidecar.refs`,MNote exact search 不再使用 query-time 文本打分反查。 阶段 B:退役 query-time sidecar 反查与旧 evidence/docs 搜索。 @@ -183,14 +255,14 @@ LightRAG - 保留 source-map / locator 兼容读取,用于历史索引和旧 OCR 结果打开。 - 新索引统一由 LightRAG parser sidecar 产出定位信息。 -## 8. 不做 +## 9. 不做 - 不在 MNote 新增一套 DOCX parser。 - 不把 DOCX 默认切到 Docling。 - 不在 MNote 实现 rerank;rerank 仍走 LightRAG 原生能力。 - 不把 DOCX 无 bbox 伪造成 PDF 页内精确定位。 -## 9. Agent / MCP 边界 +## 10. Agent / MCP 边界 LightRAG 官方内核不提供 Skill/MCP/plugin 系统;社区已有 `lightragmcp`、`mcp-lightrag` 等 MCP server,可把 LightRAG query、文档管理、图谱查询包装成 Agent 工具。这一层适合接给 Reasonix / Hermes / Claude Desktop 这类 Agent runtime,但它不能替代 MNote 的 source registry 与引用打开能力。 diff --git a/rust/crates/mnote-web/browser/document-resource-tab-runtime.js b/rust/crates/mnote-web/browser/document-resource-tab-runtime.js index 55873de1..b5cd9176 100644 --- a/rust/crates/mnote-web/browser/document-resource-tab-runtime.js +++ b/rust/crates/mnote-web/browser/document-resource-tab-runtime.js @@ -1020,7 +1020,8 @@ export const createResourceTabRuntime = (dependencies = {}) => { if (!(frame instanceof HTMLIFrameElement) || !locator) return; try { const url = new URL(frame.getAttribute('src') || frame.src || '', window.location.origin); - if (Number.isFinite(Number(locator.page))) url.searchParams.set('page', String(Number(locator.page))); + const locatorPage = Number(locator.page); + if (Number.isFinite(locatorPage) && locatorPage > 0) url.searchParams.set('page', String(locatorPage)); const bbox = evidenceBBoxParam(locator.bbox); if (bbox) url.searchParams.set('bbox', bbox); if (locator.sourceMapPath) url.searchParams.set('sourceMapPath', String(locator.sourceMapPath)); @@ -1129,7 +1130,8 @@ export const createResourceTabRuntime = (dependencies = {}) => { entry.evidenceLocator = locator; entry.panel.setAttribute('data-mnote-evidence-locator', JSON.stringify(locator)); entry.panel.setAttribute('data-mnote-evidence-open', 'true'); - if (Number.isFinite(Number(locator.page))) entry.panel.setAttribute('data-mnote-evidence-page', String(Number(locator.page))); + const locatorPage = Number(locator.page); + if (Number.isFinite(locatorPage) && locatorPage > 0) entry.panel.setAttribute('data-mnote-evidence-page', String(locatorPage)); if (locator.blockId) entry.panel.setAttribute('data-mnote-evidence-block-id', String(locator.blockId)); if (locator.evidenceText) entry.panel.setAttribute('data-mnote-evidence-text', String(locator.evidenceText).slice(0, 240)); if (locator.sourceMapPath) entry.panel.setAttribute('data-mnote-evidence-source-map-path', String(locator.sourceMapPath)); @@ -2323,7 +2325,7 @@ export const createResourceTabRuntime = (dependencies = {}) => { if (!raw) return ''; try { const url = new URL(raw, window.location.origin); - ['page', 'bbox', 'sourceMapPath', 'blockId', 'lineRange', 'charRange', 'evidenceText', 'query', 'searchQuery', 'mnoteResourceReload'].forEach((key) => { + ['page', 'bbox', 'sourceMapPath', 'blockId', 'lineRange', 'charRange', 'paragraphOrdinal', 'paraIdStart', 'paraIdEnd', 'textFingerprint', 'evidenceText', 'query', 'searchQuery', 'mnoteResourceReload'].forEach((key) => { url.searchParams.delete(key); }); return url.pathname + '?' + url.searchParams.toString(); diff --git a/rust/crates/mnote-web/browser/sidebar-filetree-open-runtime.js b/rust/crates/mnote-web/browser/sidebar-filetree-open-runtime.js index 8a439842..ef5ef45d 100644 --- a/rust/crates/mnote-web/browser/sidebar-filetree-open-runtime.js +++ b/rust/crates/mnote-web/browser/sidebar-filetree-open-runtime.js @@ -317,6 +317,12 @@ export const createSidebarFileTreeOpenRuntime = (dependencies = {}) => { sourceMapPath: String(input && input.sourceMapPath || '').trim(), blockId: String(input && input.blockId || '').trim(), evidenceText: String(input && input.evidenceText || '').trim(), + query: String(input && input.query || '').trim(), + searchQuery: String(input && input.searchQuery || '').trim(), + paragraphOrdinal: String(input && input.paragraphOrdinal || '').trim(), + paraIdStart: String(input && input.paraIdStart || '').trim(), + paraIdEnd: String(input && input.paraIdEnd || '').trim(), + textFingerprint: String(input && input.textFingerprint || '').trim(), lineRange: input && input.lineRange, charRange: input && input.charRange }); diff --git a/rust/crates/mnote-web/src/routes/knowledge_rag.rs b/rust/crates/mnote-web/src/routes/knowledge_rag.rs index 5e77268c..48f28915 100644 --- a/rust/crates/mnote-web/src/routes/knowledge_rag.rs +++ b/rust/crates/mnote-web/src/routes/knowledge_rag.rs @@ -619,6 +619,7 @@ pub async fn search( "limit": body.top_k.or(body.chunk_top_k).unwrap_or(50), "max_per_chunk": 12, "include_chunk_content": body.include_chunk_content.unwrap_or(true), + "include_sidecar": true, })), true, &context, @@ -1597,38 +1598,51 @@ async fn lightrag_json( WebError::internal(format!("LightRAG HTTP client 初始化失败: {error}")) .with_context(context) })?; - let mut request = client.request(method, &url); - if use_api_key { - if let Some(api_key) = lightrag_api_key() { + let api_key = use_api_key.then(lightrag_api_key).flatten(); + for attempt in 0..2 { + let mut request = client.request(method.clone(), &url); + if let Some(api_key) = api_key.as_deref() { request = request.header("X-API-Key", api_key); } + if let Some(body) = body.as_ref() { + request = request.json(body); + } + let response = request.send().await.map_err(|error| { + WebError::bad_gateway_code( + "knowledge_rag_lightrag_unreachable", + format!("无法访问 LightRAG provider: {error}"), + ) + .with_context(context) + })?; + let status = response.status(); + let text = response.text().await.unwrap_or_default(); + if !status.is_success() { + if attempt == 0 && should_retry_lightrag_provider_error(status, &text) { + tokio::time::sleep(Duration::from_millis(700)).await; + continue; + } + return Err(WebError::bad_gateway_code( + "knowledge_rag_lightrag_error", + format!("LightRAG provider 返回 HTTP {status}: {text}"), + ) + .with_context(context)); + } + return serde_json::from_str(&text).map_err(|error| { + WebError::bad_gateway_code( + "knowledge_rag_lightrag_json_invalid", + format!("LightRAG provider 返回非 JSON 响应: {error}"), + ) + .with_context(context) + }); } - if let Some(body) = body { - request = request.json(&body); - } - let response = request.send().await.map_err(|error| { - WebError::bad_gateway_code( - "knowledge_rag_lightrag_unreachable", - format!("无法访问 LightRAG provider: {error}"), - ) - .with_context(context) - })?; - let status = response.status(); - let text = response.text().await.unwrap_or_default(); - if !status.is_success() { - return Err(WebError::bad_gateway_code( - "knowledge_rag_lightrag_error", - format!("LightRAG provider 返回 HTTP {status}: {text}"), - ) - .with_context(context)); - } - serde_json::from_str(&text).map_err(|error| { - WebError::bad_gateway_code( - "knowledge_rag_lightrag_json_invalid", - format!("LightRAG provider 返回非 JSON 响应: {error}"), - ) - .with_context(context) - }) + unreachable!("lightrag retry loop returns on every branch") +} + +fn should_retry_lightrag_provider_error(status: StatusCode, body: &str) -> bool { + status.is_server_error() + && (body.contains("RetryError") + || body.contains("InvalidResponseError") + || body.contains("Received empty content")) } fn mapped_references( @@ -1783,19 +1797,29 @@ fn map_reference_plan( .or_else(|| primary_chunk.and_then(|chunk| chunk.get("chunk_id"))) .cloned() .unwrap_or(Value::Null); + let source_chunk_id = reference + .get("source_chunk_id") + .or_else(|| primary_chunk.and_then(|chunk| chunk.get("source_chunk_id"))) + .cloned() + .or_else(|| { + chunk_id + .as_str() + .and_then(|value| value.split_once("#match-").map(|(base, _)| json!(base))) + }) + .unwrap_or(Value::Null); let occurrence_index = reference .get("occurrence_index") .or_else(|| primary_chunk.and_then(|chunk| chunk.get("occurrence_index"))) .cloned() .unwrap_or(Value::Null); + let chunk_sidecar = + lightrag_reference_sidecar(reference, primary_chunk, &source_chunk_id, &chunk_id); let (quote, quote_source) = primary_chunk .and_then(|chunk| chunk.get("content")) .and_then(Value::as_str) .map(|value| (value.to_owned(), "chunk")) .or_else(|| { - chunk_id - .as_str() - .and_then(lightrag_chunk_content_for_id) + lightrag_chunk_content_for_reference_ids(&source_chunk_id, &chunk_id) .map(|value| (value, "kv_store")) }) .map(|(value, source)| { @@ -1806,9 +1830,7 @@ fn map_reference_plan( }) .map_or((None, "missing"), |(value, source)| (Some(value), source)); let locator_sidecar_block = entry.and_then(|entry| { - quote - .as_deref() - .and_then(|quote| find_lightrag_sidecar_block(entry, quote, query)) + find_lightrag_sidecar_block(entry, query, &occurrence_index, chunk_sidecar.as_ref()) }); let locator_sidecar_text = locator_sidecar_block .as_ref() @@ -1824,17 +1846,16 @@ fn map_reference_plan( ); let content_diagnostics = quote_content_diagnostics(quote.as_deref(), quote_source); let locator = entry.and_then(|entry| { - quote.as_deref().and_then(|quote| { - lightrag_locator_for_reference( - root_path, - root_uri, - entry, - &chunk_id, - quote, - query, - Some(&text_bundle.locator_evidence_text), - ) - }) + lightrag_locator_for_reference( + root_path, + root_uri, + entry, + &chunk_id, + query, + Some(&text_bundle.locator_evidence_text), + &occurrence_index, + chunk_sidecar.as_ref(), + ) }); let locator_precision = locator .as_ref() @@ -1880,6 +1901,7 @@ fn map_reference_plan( "reference": reference, "filePath": file_path, "chunkId": chunk_id, + "sourceChunkId": source_chunk_id, "occurrenceIndex": occurrence_index, "rawQuote": text_bundle.raw_quote.clone(), "displayQuote": text_bundle.display_quote.clone(), @@ -2091,7 +2113,10 @@ fn citation_text_bundle( } else { locator_raw.chars().take(700).collect::() }; - let display_quote = clean_lightrag_text_for_display(&raw_quote); + let display_source = locator_source_text + .filter(|value| !value.trim().is_empty()) + .unwrap_or(&raw_quote); + let display_quote = clean_lightrag_text_for_display(display_source); let locator_evidence_text = query_centered_clean_window( &clean_lightrag_text_for_locator(&locator_window), &search_query, @@ -2301,11 +2326,12 @@ fn lightrag_locator_for_reference( root_uri: &str, entry: &KnowledgeRagSourceRegistryEntry, chunk_id: &Value, - quote: &str, query: Option<&str>, locator_evidence_text: Option<&str>, + occurrence_index: &Value, + chunk_sidecar: Option<&Value>, ) -> Option { - let block = find_lightrag_sidecar_block(entry, quote, query)?; + let block = find_lightrag_sidecar_block(entry, query, occurrence_index, chunk_sidecar)?; let positions = block .get("positions") .and_then(Value::as_array) @@ -2324,23 +2350,7 @@ fn lightrag_locator_for_reference( .and_then(|query| query_centered_quote(block_content, query, 700)) .map(|value| clean_lightrag_text_for_locator(&value)) }) - .or_else(|| { - query_centered_quote(block_content, quote, 700) - .map(|value| clean_lightrag_text_for_locator(&value)) - }) - .unwrap_or_else(|| { - clean_lightrag_text_for_locator("e.chars().take(700).collect::()) - }); - let fallback_evidence_text = block - .get("content") - .and_then(Value::as_str) - .map(|value| clean_lightrag_text_for_locator(&value.chars().take(700).collect::())) - .unwrap_or_default(); - let evidence_text = if evidence_text.trim().is_empty() { - fallback_evidence_text - } else { - evidence_text - }; + .unwrap_or_else(|| clean_lightrag_text_for_locator(block_content)); let mut open_params = json!({ "rootUri": root_uri, "resourcePath": resource_path, @@ -2494,109 +2504,114 @@ fn value_to_u32(value: &Value) -> Option { fn find_lightrag_sidecar_block( entry: &KnowledgeRagSourceRegistryEntry, - quote: &str, query: Option<&str>, + occurrence_index: &Value, + chunk_sidecar: Option<&Value>, ) -> Option { - let quote_normalized = normalize_text_for_match(quote); - if quote_normalized.is_empty() { + let ref_ids = lightrag_sidecar_ref_ids(chunk_sidecar?); + if ref_ids.is_empty() { return None; } - let query_normalized = query - .map(normalize_text_for_match) - .filter(|value| !value.is_empty()); - let query_terms = query_normalized - .as_deref() - .map(query_match_terms) - .unwrap_or_default(); - let query_window = query_normalized - .as_deref() - .and_then(|query| query_centered_quote("e_normalized, query, 160)); - let path = sidecar_blocks_path(entry)?; - let content = fs::read_to_string(path).ok()?; - let mut best_query_block: Option<(usize, Value)> = None; - for line in content.lines() { - let block = serde_json::from_str::(line).ok()?; - if block.get("positions").and_then(Value::as_array).is_none() { + let blocks = lightrag_sidecar_blocks_by_id(entry, &ref_ids)?; + let query = query + .map(str::trim) + .filter(|value| !value.is_empty()) + .map(ToOwned::to_owned); + let Some(query) = query else { + return ref_ids.into_iter().find_map(|id| blocks.get(&id).cloned()); + }; + let target_occurrence = occurrence_index.as_u64().unwrap_or(0); + let mut seen_occurrences = 0u64; + for id in ref_ids { + let Some(block) = blocks.get(&id) else { continue; - } - let block_text = block + }; + let count = block .get("content") .and_then(Value::as_str) - .unwrap_or_default(); - let block_normalized = normalize_text_for_match(block_text); - if block_normalized.is_empty() { + .map(|content| exact_match_count(content, &query)) + .unwrap_or(0); + if count == 0 { continue; } - let block_contains_query = query_normalized.as_deref().is_some_and(|query_normalized| { - block_normalized - .to_ascii_lowercase() - .contains(&query_normalized.to_ascii_lowercase()) - || query_terms - .iter() - .any(|term| block_matches_query_term(&block_normalized, term)) - }); - if query_normalized.is_some() { - if block_contains_query { - let score = query_window - .as_deref() - .map(|window| { - let window_prefix = match_prefix(window, 40); - let exact_window_bonus = if !window_prefix.trim().is_empty() - && block_normalized.contains(&window_prefix) - { - 100_000 - } else { - 0 - }; - exact_window_bonus - + common_char_substring_score(&block_normalized, window) * 10 - }) - .unwrap_or(0) - + common_char_prefix_len(&block_normalized, "e_normalized) - + common_char_substring_score(&block_normalized, "e_normalized); - if score - > best_query_block - .as_ref() - .map(|(score, _)| *score) - .unwrap_or(0) - { - best_query_block = Some((score, block.clone())); - } - } else { - continue; + if seen_occurrences + count > target_occurrence { + return Some(block.clone()); + } + seen_occurrences += count; + } + None +} + +fn lightrag_sidecar_ref_ids(sidecar: &Value) -> Vec { + let mut ids = Vec::new(); + if let Some(id) = sidecar.get("id").and_then(Value::as_str) { + ids.push(id.trim().to_string()); + } + if let Some(refs) = sidecar.get("refs").and_then(Value::as_array) { + for item in refs { + if let Some(id) = item.get("id").and_then(Value::as_str) { + ids.push(id.trim().to_string()); } } - if block_normalized.contains("e_normalized) - || quote_normalized.contains(&block_normalized) - || quote_normalized.contains(&match_prefix(&block_normalized, 80)) - || block_normalized.contains(&match_prefix("e_normalized, 80)) - { - return Some(block); + } + ids.retain(|id| !id.is_empty()); + ids.dedup(); + ids +} + +fn lightrag_sidecar_blocks_by_id( + entry: &KnowledgeRagSourceRegistryEntry, + ref_ids: &[String], +) -> Option> { + let wanted = ref_ids.iter().cloned().collect::>(); + let path = sidecar_blocks_path(entry)?; + let content = fs::read_to_string(path).ok()?; + let mut blocks = BTreeMap::new(); + for line in content.lines() { + let block = serde_json::from_str::(line).ok()?; + let Some(block_id) = block.get("blockid").and_then(Value::as_str) else { + continue; + }; + if wanted.contains(block_id) && block.get("positions").and_then(Value::as_array).is_some() { + blocks.insert(block_id.to_string(), block); + } + if blocks.len() == wanted.len() { + break; } } - best_query_block.map(|(_, block)| block) + Some(blocks) } -fn common_char_prefix_len(left: &str, right: &str) -> usize { - left.chars() - .zip(right.chars()) - .take_while(|(left, right)| left == right) - .count() +fn exact_match_count(content: &str, query: &str) -> u64 { + if query.is_empty() { + return 0; + } + let content_lower = content.to_lowercase(); + let query_lower = query.to_lowercase(); + let mut count = 0u64; + let mut start = 0usize; + while let Some(index) = content_lower[start..].find(&query_lower) { + count += 1; + start += index + query_lower.len().max(1); + } + count } -fn common_char_substring_score(left: &str, right: &str) -> usize { - let right_prefixes = right - .chars() - .collect::>() - .windows(40.min(right.chars().count())) - .map(|window| window.iter().collect::()) - .collect::>(); - right_prefixes - .iter() - .filter(|candidate| !candidate.trim().is_empty() && left.contains(candidate.as_str())) - .map(|candidate| candidate.chars().count()) - .max() - .unwrap_or(0) +fn lightrag_reference_sidecar( + reference: &Value, + primary_chunk: Option<&Value>, + source_chunk_id: &Value, + chunk_id: &Value, +) -> Option { + reference + .get("sidecar") + .cloned() + .or_else(|| { + primary_chunk + .and_then(|chunk| chunk.get("sidecar")) + .cloned() + }) + .or_else(|| lightrag_chunk_sidecar_for_reference_ids(source_chunk_id, chunk_id)) } fn block_matches_query_term(block_normalized: &str, term: &str) -> bool { @@ -2712,13 +2727,6 @@ fn normalize_text_for_match(value: &str) -> String { .collect::() } -fn match_prefix(value: &str, limit: usize) -> String { - value - .chars() - .take(limit.min(value.chars().count())) - .collect() -} - fn sidecar_blocks_path(entry: &KnowledgeRagSourceRegistryEntry) -> Option { for file_path in lightrag_sidecar_file_path_candidates(entry) { let stem = Path::new(&file_path).file_stem()?.to_string_lossy(); @@ -2841,12 +2849,56 @@ fn sidecar_text_stats(path: &Path) -> Result { Ok(stats) } -fn lightrag_chunk_content_for_id(chunk_id: &str) -> Option { +fn lightrag_chunk_content_for_reference_ids( + source_chunk_id: &Value, + chunk_id: &Value, +) -> Option { + lightrag_chunk_id_candidates(source_chunk_id, chunk_id) + .into_iter() + .find_map(|id| lightrag_chunk_content_for_id(&id)) +} + +fn lightrag_chunk_sidecar_for_reference_ids( + source_chunk_id: &Value, + chunk_id: &Value, +) -> Option { + lightrag_chunk_id_candidates(source_chunk_id, chunk_id) + .into_iter() + .find_map(|id| { + lightrag_chunk_value_for_id(&id).and_then(|chunk| chunk.get("sidecar").cloned()) + }) +} + +fn lightrag_chunk_id_candidates(source_chunk_id: &Value, chunk_id: &Value) -> Vec { + let mut ids = Vec::new(); + for value in [source_chunk_id, chunk_id] { + let Some(raw) = value + .as_str() + .map(str::trim) + .filter(|value| !value.is_empty()) + else { + continue; + }; + ids.push(raw.to_string()); + if let Some((base, _)) = raw.split_once("#match-") { + ids.push(base.to_string()); + } + } + ids.sort(); + ids.dedup(); + ids +} + +fn lightrag_chunk_value_for_id(chunk_id: &str) -> Option { let content = fs::read_to_string(lightrag_working_dir().join("kv_store_text_chunks.json")).ok()?; let chunks = serde_json::from_str::(&content).ok()?; - chunks - .get(chunk_id) + chunks.get(chunk_id).cloned() +} + +fn lightrag_chunk_content_for_id(chunk_id: &str) -> Option { + lightrag_chunk_value_for_id(chunk_id) + .as_ref() .and_then(|chunk| chunk.get("content")) .and_then(Value::as_str) .map(ToOwned::to_owned) @@ -4588,7 +4640,14 @@ mod tests { "chunk_id": "doc1-chunk-256", "chunks": [{ "chunk_id": "doc1-chunk-256", - "content": chunk_content + "content": chunk_content, + "sidecar": { + "type": "block", + "refs": [ + {"type": "block", "id": "wrong-index"}, + {"type": "block", "id": "tes"} + ] + } }] }), ®istry, @@ -4869,9 +4928,10 @@ mod tests { "file:///tmp/root", &entry, &json!("doc1-chunk-000"), - "# START: This is the first image in PDF\nThis is text BEFORE the image.", None, None, + &json!(0), + Some(&json!({"type":"block","refs":[{"type":"block","id":"block1"}]})), ) .expect("locator"); @@ -4927,9 +4987,10 @@ mod tests { "file:///tmp/root", &entry, &json!("doc1-chunk-000"), - "监管政策变革", Some("监管政策变革"), None, + &json!(0), + Some(&json!({"type":"block","refs":[{"type":"block","id":"block-docx-1"}]})), ) .expect("locator"); diff --git a/rust/crates/mnote-web/src/routes/web_shell.rs b/rust/crates/mnote-web/src/routes/web_shell.rs index dc27d886..194314ae 100644 --- a/rust/crates/mnote-web/src/routes/web_shell.rs +++ b/rust/crates/mnote-web/src/routes/web_shell.rs @@ -1585,7 +1585,7 @@ pub async fn office_preview_page(Query(query): Query) -> Res for (const element of elements) {{ const compactElement = compactEvidenceText(element.textContent || ''); if ((compactAnchor.length >= 6 || shortQueryAnchor) && compactElement.includes(compactAnchor)) {{ - return markEvidenceTarget(shortQueryAnchor ? shortLeadingAnchorTarget(element, elements, elements.indexOf(element)) : element); + return markEvidenceTarget(element); }} if ( compactAnchorWithoutNumbers.length >= 8 @@ -1631,11 +1631,7 @@ pub async fn office_preview_page(Query(query): Query) -> Res const text = normalizeEvidenceText(element.textContent || ''); if (!text || text.length < 3 || text.length > 6000) return 0; const compactText = compactEvidenceText(text); - const compactSearchQuery = compactEvidenceText(evidenceSearchQuery); let score = 0; - if (compactSearchQuery.length >= 2 && compactText.includes(compactSearchQuery)) {{ - score += 2400; - }} for (const anchor of anchors) {{ const compactAnchor = compactEvidenceText(anchor); if (!compactAnchor || compactAnchor.length < 4) continue; @@ -1771,7 +1767,14 @@ pub async fn office_preview_page(Query(query): Query) -> Res if (!best || score > best.score) best = {{ element, score }}; if (score >= 2400 && evidenceElementMatchesSearchQuery(element)) break; }} - return best ? markEvidenceTarget(best.element) : false; + if (!best) return false; + if (!anchors.length) {{ + return evidenceElementMatchesSearchQuery(best.element) ? markEvidenceTarget(best.element) : false; + }} + const threshold = Math.max(180, Math.min(800, anchors[0].length * 4)); + if (best.score >= threshold) return markEvidenceTarget(best.element); + if (best.score > 0 && evidenceElementMatchesSearchQuery(best.element)) return markEvidenceTarget(best.element); + return false; }} function scrollToEvidencePageFallback() {{ @@ -1786,15 +1789,15 @@ pub async fn office_preview_page(Query(query): Query) -> Res try {{ const sourceMap = await fetchEvidenceSourceMap(); const block = findEvidenceBlockInSourceMap(sourceMap); - if (evidenceParagraphOrdinal && scrollToEvidenceParagraphOrdinal()) return; if (evidenceText && scrollToEvidenceParagraph(evidenceText)) return; + if (evidenceParagraphOrdinal && scrollToEvidenceParagraphOrdinal()) return; if (block && scrollToEvidenceParagraph(block.text)) return; if (evidenceText && scrollToEvidenceTextCandidates(evidenceText)) return; if (block && scrollToEvidenceTextCandidates(block.text)) return; if (scrollToEvidenceCoordinate(sourceMap, block)) return; }} catch (_) {{}} - if (evidenceParagraphOrdinal && scrollToEvidenceParagraphOrdinal()) return; if (evidenceText && scrollToEvidenceParagraph(evidenceText)) return; + if (evidenceParagraphOrdinal && scrollToEvidenceParagraphOrdinal()) return; if (evidenceText && scrollToEvidenceTextCandidates(evidenceText)) return; scrollToEvidencePageFallback(); }} diff --git a/scripts/task543-knowledge-rag-pyrrolidine-top5-locator-smoke.js b/scripts/task543-knowledge-rag-pyrrolidine-top5-locator-smoke.js index afd95127..5dd908b8 100644 --- a/scripts/task543-knowledge-rag-pyrrolidine-top5-locator-smoke.js +++ b/scripts/task543-knowledge-rag-pyrrolidine-top5-locator-smoke.js @@ -70,7 +70,7 @@ async function apiSearch(context) { rootUri: ROOT_URI, workspaceId: WORKSPACE_ID, query: QUERY, - mode: "mix", + mode: "exact", topK: 20, chunkTopK: 20, includeChunkContent: true, @@ -178,6 +178,8 @@ async function clickAndAuditRow(page, index, apiResult) { const evidenceText = locator?.openAction?.params?.evidenceText || apiResult?.locator?.openAction?.params?.evidenceText || apiResult?.quote || rowText; const overlap = longestCommonSubstringLength(state.highlightedText, evidenceText); + const compactHighlighted = compactText(state.highlightedText); + const compactEvidence = compactText(evidenceText); assert.equal(state.panelBlockId, locator.blockId, `第 ${index + 1} 条 panel blockId 不一致: ${JSON.stringify(state, null, 2)}`); assert(state.highlightedText.includes(QUERY), `第 ${index + 1} 条定位高亮未包含 query: ${JSON.stringify(state, null, 2)}`); assert( @@ -185,11 +187,9 @@ async function clickAndAuditRow(page, index, apiResult) { `第 ${index + 1} 条仍是短词高亮,不是段落级高亮: ${JSON.stringify(state, null, 2)}`, ); assert( - overlap >= 10 || ( - state.highlightedTag === "P" - && compactText(state.highlightedText).includes(compactText(QUERY)) - && compactText(state.highlightedText).length >= 6 - ), + overlap >= Math.min(10, Math.max(4, compactEvidence.length)) + || compactHighlighted.includes(compactEvidence) + || compactEvidence.includes(compactHighlighted), `第 ${index + 1} 条搜索结果与实际定位上下文不一致: ${JSON.stringify({ rowText, evidenceText, overlap, state }, null, 2)}`, );