chore: init monorepo snapshot
This commit is contained in:
liaibo
@@ -0,0 +1,661 @@
|
||||
# LLM Query Cache Cleanup Tool - User Guide
|
||||
|
||||
## Overview
|
||||
|
||||
This tool cleans up LightRAG's LLM query cache from KV storage implementations. It specifically targets query caches generated during RAG query operations (modes: `mix`, `hybrid`, `local`, `global`), including both query and keywords caches.
|
||||
|
||||
## Supported Storage Types
|
||||
|
||||
1. **JsonKVStorage** - File-based JSON storage
|
||||
2. **RedisKVStorage** - Redis database storage
|
||||
3. **PGKVStorage** - PostgreSQL database storage
|
||||
4. **MongoKVStorage** - MongoDB database storage
|
||||
|
||||
## Cache Types
|
||||
|
||||
The tool cleans up the following query cache types:
|
||||
|
||||
### Query Cache Modes (4 types)
|
||||
- `mix:*` - Mixed mode query caches
|
||||
- `hybrid:*` - Hybrid mode query caches
|
||||
- `local:*` - Local mode query caches
|
||||
- `global:*` - Global mode query caches
|
||||
|
||||
### Cache Content Types (2 types)
|
||||
- `*:query:*` - Query result caches
|
||||
- `*:keywords:*` - Keywords extraction caches
|
||||
|
||||
### Cache Key Format
|
||||
```
|
||||
<mode>:<cache_type>:<hash>
|
||||
```
|
||||
|
||||
Examples:
|
||||
- `mix:query:5ce04d25e957c290216cee5bfe6344fa`
|
||||
- `mix:keywords:fee77b98244a0b047ce95e21060de60e`
|
||||
- `global:query:abc123def456...`
|
||||
- `local:keywords:789xyz...`
|
||||
|
||||
**Important Note**: This tool does NOT clean extraction caches (`default:extract:*` and `default:summary:*`). Use the migration tool or manual deletion for those caches.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- The tool reads storage configuration from environment variables or `config.ini`
|
||||
- Ensure the target storage is properly configured and accessible
|
||||
- Backup important data before running cleanup operations
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic Usage
|
||||
|
||||
Run from the LightRAG project root directory:
|
||||
|
||||
```bash
|
||||
python -m lightrag.tools.clean_llm_query_cache
|
||||
# or
|
||||
python lightrag/tools/clean_llm_query_cache.py
|
||||
```
|
||||
|
||||
### Interactive Workflow
|
||||
|
||||
The tool guides you through the following steps:
|
||||
|
||||
#### 1. Select Storage Type
|
||||
```
|
||||
============================================================
|
||||
LLM Query Cache Cleanup Tool - LightRAG
|
||||
============================================================
|
||||
|
||||
=== Storage Setup ===
|
||||
|
||||
Supported KV Storage Types:
|
||||
[1] JsonKVStorage
|
||||
[2] RedisKVStorage
|
||||
[3] PGKVStorage
|
||||
[4] MongoKVStorage
|
||||
|
||||
Select storage type (1-4) (Press Enter to exit): 1
|
||||
```
|
||||
|
||||
**Note**: You can press Enter or type `0` at any prompt to exit gracefully.
|
||||
|
||||
#### 2. Storage Validation
|
||||
The tool will:
|
||||
- Check required environment variables
|
||||
- Auto-detect workspace configuration
|
||||
- Initialize and connect to storage
|
||||
- Verify connection status
|
||||
|
||||
```
|
||||
Checking configuration...
|
||||
✓ All required environment variables are set
|
||||
|
||||
Initializing storage...
|
||||
- Storage Type: JsonKVStorage
|
||||
- Workspace: space1
|
||||
- Connection Status: ✓ Success
|
||||
```
|
||||
|
||||
#### 3. View Cache Statistics
|
||||
|
||||
The tool displays a detailed breakdown of query caches by mode and type:
|
||||
|
||||
```
|
||||
Counting query cache records...
|
||||
|
||||
📊 Query Cache Statistics (Before Cleanup):
|
||||
┌────────────┬────────────┬────────────┬────────────┐
|
||||
│ Mode │ Query │ Keywords │ Total │
|
||||
├────────────┼────────────┼────────────┼────────────┤
|
||||
│ mix │ 1,234 │ 567 │ 1,801 │
|
||||
│ hybrid │ 890 │ 423 │ 1,313 │
|
||||
│ local │ 2,345 │ 1,123 │ 3,468 │
|
||||
│ global │ 678 │ 345 │ 1,023 │
|
||||
├────────────┼────────────┼────────────┼────────────┤
|
||||
│ Total │ 5,147 │ 2,458 │ 7,605 │
|
||||
└────────────┴────────────┴────────────┴────────────┘
|
||||
```
|
||||
|
||||
#### 4. Select Cleanup Scope
|
||||
|
||||
Choose what type of caches to delete:
|
||||
|
||||
```
|
||||
=== Cleanup Options ===
|
||||
[1] Delete all query caches (both query and keywords)
|
||||
[2] Delete query caches only (keep keywords)
|
||||
[3] Delete keywords caches only (keep query)
|
||||
[0] Cancel
|
||||
|
||||
Select cleanup option (0-3): 1
|
||||
```
|
||||
|
||||
**Cleanup Types:**
|
||||
- **Option 1 (all)**: Deletes both query and keywords caches across all modes
|
||||
- **Option 2 (query)**: Deletes only query caches, preserves keywords caches
|
||||
- **Option 3 (keywords)**: Deletes only keywords caches, preserves query caches
|
||||
|
||||
#### 5. Confirm Deletion
|
||||
|
||||
Review the cleanup plan and confirm:
|
||||
|
||||
```
|
||||
============================================================
|
||||
Cleanup Confirmation
|
||||
============================================================
|
||||
Storage: JsonKVStorage (workspace: space1)
|
||||
Cleanup Type: all
|
||||
Records to Delete: 7,605 / 7,605
|
||||
|
||||
⚠️ WARNING: This will delete ALL query caches across all modes!
|
||||
|
||||
Continue with deletion? (y/n): y
|
||||
```
|
||||
|
||||
#### 6. Execute Cleanup
|
||||
|
||||
The tool performs batch deletion with real-time progress:
|
||||
|
||||
**JsonKVStorage Example:**
|
||||
```
|
||||
=== Starting Cleanup ===
|
||||
💡 Processing 1,000 records at a time from JsonKVStorage
|
||||
|
||||
Batch 1/8: ████░░░░░░░░░░░░░░░░ 1,000/7,605 (13.1%) ✓
|
||||
Batch 2/8: ████████░░░░░░░░░░░░ 2,000/7,605 (26.3%) ✓
|
||||
...
|
||||
Batch 8/8: ████████████████████ 7,605/7,605 (100.0%) ✓
|
||||
|
||||
Persisting changes to storage...
|
||||
✓ Changes persisted successfully
|
||||
```
|
||||
|
||||
**RedisKVStorage Example:**
|
||||
```
|
||||
=== Starting Cleanup ===
|
||||
💡 Processing Redis keys in batches of 1,000
|
||||
|
||||
Batch 1: Deleted 1,000 keys (Total: 1,000) ✓
|
||||
Batch 2: Deleted 1,000 keys (Total: 2,000) ✓
|
||||
...
|
||||
```
|
||||
|
||||
**PostgreSQL Example:**
|
||||
```
|
||||
=== Starting Cleanup ===
|
||||
💡 Executing PostgreSQL DELETE query
|
||||
|
||||
✓ Deleted 7,605 records in 0.45s
|
||||
```
|
||||
|
||||
**MongoDB Example:**
|
||||
```
|
||||
=== Starting Cleanup ===
|
||||
💡 Executing MongoDB deleteMany operations
|
||||
|
||||
Pattern 1/8: Deleted 1,234 records ✓
|
||||
Pattern 2/8: Deleted 567 records ✓
|
||||
...
|
||||
Total deleted: 7,605 records
|
||||
```
|
||||
|
||||
#### 7. Review Cleanup Report
|
||||
|
||||
The tool provides a comprehensive final report:
|
||||
|
||||
**Successful Cleanup:**
|
||||
```
|
||||
============================================================
|
||||
Cleanup Complete - Final Report
|
||||
============================================================
|
||||
|
||||
📊 Statistics:
|
||||
Total records to delete: 7,605
|
||||
Total batches: 8
|
||||
Successful batches: 8
|
||||
Failed batches: 0
|
||||
Successfully deleted: 7,605
|
||||
Failed to delete: 0
|
||||
Success rate: 100.00%
|
||||
|
||||
📈 Before/After Comparison:
|
||||
Total caches before: 7,605
|
||||
Total caches after: 0
|
||||
Net reduction: 7,605
|
||||
|
||||
============================================================
|
||||
✓ SUCCESS: All records cleaned up successfully!
|
||||
============================================================
|
||||
|
||||
📊 Query Cache Statistics (After Cleanup):
|
||||
┌────────────┬────────────┬────────────┬────────────┐
|
||||
│ Mode │ Query │ Keywords │ Total │
|
||||
├────────────┼────────────┼────────────┼────────────┤
|
||||
│ mix │ 0 │ 0 │ 0 │
|
||||
│ hybrid │ 0 │ 0 │ 0 │
|
||||
│ local │ 0 │ 0 │ 0 │
|
||||
│ global │ 0 │ 0 │ 0 │
|
||||
├────────────┼────────────┼────────────┼────────────┤
|
||||
│ Total │ 0 │ 0 │ 0 │
|
||||
└────────────┴────────────┴────────────┴────────────┘
|
||||
```
|
||||
|
||||
**Cleanup with Errors:**
|
||||
```
|
||||
============================================================
|
||||
Cleanup Complete - Final Report
|
||||
============================================================
|
||||
|
||||
📊 Statistics:
|
||||
Total records to delete: 7,605
|
||||
Total batches: 8
|
||||
Successful batches: 7
|
||||
Failed batches: 1
|
||||
Successfully deleted: 6,605
|
||||
Failed to delete: 1,000
|
||||
Success rate: 86.85%
|
||||
|
||||
📈 Before/After Comparison:
|
||||
Total caches before: 7,605
|
||||
Total caches after: 1,000
|
||||
Net reduction: 6,605
|
||||
|
||||
⚠️ Errors encountered: 1
|
||||
|
||||
Error Details:
|
||||
------------------------------------------------------------
|
||||
|
||||
Error Summary:
|
||||
- ConnectionError: 1 occurrence(s)
|
||||
|
||||
First 5 errors:
|
||||
|
||||
1. Batch 3
|
||||
Type: ConnectionError
|
||||
Message: Connection timeout after 30s
|
||||
Records lost: 1,000
|
||||
|
||||
============================================================
|
||||
⚠️ WARNING: Cleanup completed with errors!
|
||||
Please review the error details above.
|
||||
============================================================
|
||||
```
|
||||
|
||||
## Technical Details
|
||||
|
||||
### Workspace Handling
|
||||
|
||||
The tool retrieves workspace in the following priority order:
|
||||
|
||||
1. **Storage-specific workspace environment variables**
|
||||
- PGKVStorage: `POSTGRES_WORKSPACE`
|
||||
- MongoKVStorage: `MONGODB_WORKSPACE`
|
||||
- RedisKVStorage: `REDIS_WORKSPACE`
|
||||
|
||||
2. **Generic workspace environment variable**
|
||||
- `WORKSPACE`
|
||||
|
||||
3. **Default value**
|
||||
- Empty string (uses storage's default workspace)
|
||||
|
||||
### Batch Deletion
|
||||
|
||||
- Default batch size: 1000 records/batch
|
||||
- Prevents memory overflow and connection timeouts
|
||||
- Each batch is processed independently
|
||||
- Failed batches are logged but don't stop cleanup
|
||||
|
||||
### Storage-Specific Deletion Strategies
|
||||
|
||||
#### JsonKVStorage
|
||||
- Collects all matching keys first (snapshot approach)
|
||||
- Deletes in batches with lock protection
|
||||
- Fast in-memory operations
|
||||
|
||||
#### RedisKVStorage
|
||||
- Uses SCAN with pattern matching
|
||||
- Pipeline DELETE for batch operations
|
||||
- Cursor-based iteration for large datasets
|
||||
|
||||
#### PostgreSQL
|
||||
- Single DELETE query with OR conditions
|
||||
- Efficient server-side bulk deletion
|
||||
- Uses LIKE patterns for mode/type matching
|
||||
|
||||
#### MongoDB
|
||||
- Multiple deleteMany operations (one per pattern)
|
||||
- Regex-based document matching
|
||||
- Returns exact deletion counts
|
||||
|
||||
### Pattern Matching Implementation
|
||||
|
||||
**JsonKVStorage:**
|
||||
```python
|
||||
# Direct key prefix matching
|
||||
if key.startswith("mix:query:") or key.startswith("mix:keywords:")
|
||||
```
|
||||
|
||||
**RedisKVStorage:**
|
||||
```python
|
||||
# SCAN with namespace-prefixed patterns
|
||||
pattern = f"{namespace}:mix:query:*"
|
||||
cursor, keys = await redis.scan(cursor, match=pattern)
|
||||
```
|
||||
|
||||
**PostgreSQL:**
|
||||
```python
|
||||
# SQL LIKE conditions
|
||||
WHERE id LIKE 'mix:query:%' OR id LIKE 'mix:keywords:%'
|
||||
```
|
||||
|
||||
**MongoDB:**
|
||||
```python
|
||||
# Regex queries on _id field
|
||||
{"_id": {"$regex": "^mix:query:"}}
|
||||
```
|
||||
|
||||
## Error Handling & Resilience
|
||||
|
||||
The tool implements comprehensive error tracking:
|
||||
|
||||
### Batch-Level Error Tracking
|
||||
- Each batch is independently error-checked
|
||||
- Failed batches are logged with full details
|
||||
- Successful batches commit even if later batches fail
|
||||
- Real-time progress shows ✓ (success) or ✗ (failed)
|
||||
|
||||
### Error Reporting
|
||||
After cleanup completes, a detailed report includes:
|
||||
- **Statistics**: Total records, success/failure counts, success rate
|
||||
- **Before/After Comparison**: Net reduction in cache count
|
||||
- **Error Summary**: Grouped by error type with occurrence counts
|
||||
- **Error Details**: Batch number, error type, message, and records lost
|
||||
- **Recommendations**: Clear indication of success or need for review
|
||||
|
||||
### Verification
|
||||
- Post-cleanup count verification
|
||||
- Before/after statistics comparison
|
||||
- Identifies partial cleanup scenarios
|
||||
|
||||
## Important Notes
|
||||
|
||||
1. **Irreversible Operation**
|
||||
- Deleted caches cannot be recovered
|
||||
- Always backup important data before cleanup
|
||||
- Test on non-production data first
|
||||
|
||||
2. **Performance Impact**
|
||||
- Query performance may degrade temporarily after cleanup
|
||||
- Caches will rebuild on subsequent queries
|
||||
- Consider cleanup during off-peak hours
|
||||
|
||||
3. **Selective Cleanup**
|
||||
- Choose cleanup scope carefully
|
||||
- Keywords caches may be valuable for future queries
|
||||
- Query caches rebuild faster than keywords caches
|
||||
|
||||
4. **Workspace Isolation**
|
||||
- Cleanup only affects the selected workspace
|
||||
- Other workspaces remain untouched
|
||||
- Verify workspace before confirming cleanup
|
||||
|
||||
5. **Interrupt and Resume**
|
||||
- Cleanup can be interrupted at any time (Ctrl+C)
|
||||
- Already deleted records cannot be recovered
|
||||
- No automatic resume - must run tool again
|
||||
|
||||
## Storage Configuration
|
||||
|
||||
The tool supports multiple configuration methods with the following priority:
|
||||
|
||||
1. **Environment variables** (highest priority)
|
||||
2. **config.ini file** (medium priority)
|
||||
3. **Default values** (lowest priority)
|
||||
|
||||
### Environment Variable Configuration
|
||||
|
||||
Configure storage settings in your `.env` file:
|
||||
|
||||
#### Workspace Configuration (Optional)
|
||||
|
||||
```bash
|
||||
# Generic workspace (shared by all storages)
|
||||
WORKSPACE=space1
|
||||
|
||||
# Or configure independent workspace for specific storage
|
||||
POSTGRES_WORKSPACE=pg_space
|
||||
MONGODB_WORKSPACE=mongo_space
|
||||
REDIS_WORKSPACE=redis_space
|
||||
```
|
||||
|
||||
**Workspace Priority**: Storage-specific > Generic WORKSPACE > Empty string
|
||||
|
||||
#### JsonKVStorage
|
||||
|
||||
```bash
|
||||
WORKING_DIR=./rag_storage
|
||||
```
|
||||
|
||||
#### RedisKVStorage
|
||||
|
||||
```bash
|
||||
REDIS_URI=redis://localhost:6379
|
||||
```
|
||||
|
||||
#### PGKVStorage
|
||||
|
||||
```bash
|
||||
POSTGRES_HOST=localhost
|
||||
POSTGRES_PORT=5432
|
||||
POSTGRES_USER=your_username
|
||||
POSTGRES_PASSWORD=your_password
|
||||
POSTGRES_DATABASE=your_database
|
||||
```
|
||||
|
||||
#### MongoKVStorage
|
||||
|
||||
```bash
|
||||
MONGO_URI=mongodb://root:root@localhost:27017/
|
||||
MONGO_DATABASE=LightRAG
|
||||
```
|
||||
|
||||
### config.ini Configuration
|
||||
|
||||
Alternatively, create a `config.ini` file in the project root:
|
||||
|
||||
```ini
|
||||
[redis]
|
||||
uri = redis://localhost:6379
|
||||
|
||||
[postgres]
|
||||
host = localhost
|
||||
port = 5432
|
||||
user = postgres
|
||||
password = yourpassword
|
||||
database = lightrag
|
||||
|
||||
[mongodb]
|
||||
uri = mongodb://root:root@localhost:27017/
|
||||
database = LightRAG
|
||||
```
|
||||
|
||||
**Note**: Environment variables take precedence over config.ini settings.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Missing Environment Variables
|
||||
```
|
||||
⚠️ Warning: Missing environment variables: POSTGRES_USER, POSTGRES_PASSWORD
|
||||
```
|
||||
**Solution**: Add missing variables to your `.env` file or configure in `config.ini`
|
||||
|
||||
### Connection Failed
|
||||
```
|
||||
✗ Initialization failed: Connection refused
|
||||
```
|
||||
**Solutions**:
|
||||
- Check if database service is running
|
||||
- Verify connection parameters (host, port, credentials)
|
||||
- Check firewall settings
|
||||
- Ensure network connectivity for remote databases
|
||||
|
||||
### No Caches Found
|
||||
```
|
||||
⚠️ No query caches found in storage
|
||||
```
|
||||
**Possible Reasons**:
|
||||
- No queries have been run yet
|
||||
- Caches were already cleaned
|
||||
- Wrong workspace selected
|
||||
- Different storage type was used for queries
|
||||
|
||||
### Partial Cleanup
|
||||
```
|
||||
⚠️ WARNING: Cleanup completed with errors!
|
||||
```
|
||||
**Solutions**:
|
||||
- Check error details in the report
|
||||
- Verify storage connection stability
|
||||
- Re-run tool to clean remaining caches
|
||||
- Check storage capacity and permissions
|
||||
|
||||
## Use Cases
|
||||
|
||||
### Use Case 1: Clean All Query Caches
|
||||
|
||||
**Scenario**: Free up storage space by removing all query caches
|
||||
|
||||
```bash
|
||||
# Run tool
|
||||
python -m lightrag.tools.clean_llm_query_cache
|
||||
|
||||
# Select: Storage type -> Option 1 (all) -> Confirm (y)
|
||||
```
|
||||
|
||||
**Result**: All query and keywords caches deleted, maximum storage freed
|
||||
|
||||
### Use Case 2: Refresh Query Caches Only
|
||||
|
||||
**Scenario**: Force query cache rebuild while keeping keywords
|
||||
|
||||
```bash
|
||||
# Run tool
|
||||
python -m lightrag.tools.clean_llm_query_cache
|
||||
|
||||
# Select: Storage type -> Option 2 (query only) -> Confirm (y)
|
||||
```
|
||||
|
||||
**Result**: Query caches deleted, keywords preserved for faster rebuild
|
||||
|
||||
### Use Case 3: Clean Stale Keywords
|
||||
|
||||
**Scenario**: Remove outdated keywords while keeping recent query results
|
||||
|
||||
```bash
|
||||
# Run tool
|
||||
python -m lightrag.tools.clean_llm_query_cache
|
||||
|
||||
# Select: Storage type -> Option 3 (keywords only) -> Confirm (y)
|
||||
```
|
||||
|
||||
**Result**: Keywords deleted, query caches preserved
|
||||
|
||||
### Use Case 4: Workspace-Specific Cleanup
|
||||
|
||||
**Scenario**: Clean caches for a specific workspace
|
||||
|
||||
```bash
|
||||
# Configure workspace
|
||||
export WORKSPACE=development
|
||||
|
||||
# Run tool
|
||||
python -m lightrag.tools.clean_llm_query_cache
|
||||
|
||||
# Select: Storage type -> Cleanup option -> Confirm (y)
|
||||
```
|
||||
|
||||
**Result**: Only development workspace caches cleaned
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Backup Before Cleanup**
|
||||
- Always backup your storage before major cleanup
|
||||
- Test cleanup on non-production data first
|
||||
- Document cleanup decisions
|
||||
|
||||
2. **Monitor Performance**
|
||||
- Watch storage metrics during cleanup
|
||||
- Monitor query performance after cleanup
|
||||
- Allow time for cache rebuild
|
||||
|
||||
3. **Scheduled Cleanup**
|
||||
- Clean caches periodically (weekly/monthly)
|
||||
- Automate cleanup for development environments
|
||||
- Keep production cleanup manual for safety
|
||||
|
||||
4. **Selective Deletion**
|
||||
- Consider cleanup scope based on needs
|
||||
- Keywords caches are harder to rebuild
|
||||
- Query caches rebuild automatically
|
||||
|
||||
5. **Storage Capacity**
|
||||
- Monitor storage usage trends
|
||||
- Clean caches before reaching capacity limits
|
||||
- Archive old data if needed
|
||||
|
||||
## Comparison with Migration Tool
|
||||
|
||||
| Feature | Cleanup Tool | Migration Tool |
|
||||
|---------|-------------|----------------|
|
||||
| **Purpose** | Delete query caches | Migrate extraction caches |
|
||||
| **Cache Types** | mix/hybrid/local/global | default:extract/summary |
|
||||
| **Modes** | query, keywords | extract, summary |
|
||||
| **Operation** | Deletion | Copy between storages |
|
||||
| **Reversible** | No | Yes (source unchanged) |
|
||||
| **Use Case** | Free storage, refresh caches | Change storage backend |
|
||||
|
||||
## Limitations
|
||||
|
||||
1. **Single Storage Operation**
|
||||
- Can only clean one storage type at a time
|
||||
- To clean multiple storages, run tool multiple times
|
||||
|
||||
2. **No Dry Run Mode**
|
||||
- Deletion is immediate after confirmation
|
||||
- No preview-only mode available
|
||||
- Test on non-production first
|
||||
|
||||
3. **No Selective Mode Cleanup**
|
||||
- Cannot clean only specific modes (e.g., only `mix`)
|
||||
- Cleanup applies to all modes for selected cache type
|
||||
- All-or-nothing per cache type
|
||||
|
||||
4. **No Scheduled Cleanup**
|
||||
- Manual execution required
|
||||
- No built-in scheduling
|
||||
- Use cron/scheduler if automation needed
|
||||
|
||||
5. **Verification Limitations**
|
||||
- Post-cleanup verification may fail in error scenarios
|
||||
- Manual verification recommended for critical operations
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
Potential improvements for future versions:
|
||||
|
||||
- Selective mode cleanup (e.g., clean only `mix` mode)
|
||||
- Age-based cleanup (delete caches older than X days)
|
||||
- Size-based cleanup (delete largest caches first)
|
||||
- Dry run mode for safe preview
|
||||
- Automated scheduling support
|
||||
- Cache statistics export
|
||||
- Incremental cleanup with pause/resume
|
||||
|
||||
## Support
|
||||
|
||||
For issues, questions, or feature requests:
|
||||
- Check the error details in the cleanup report
|
||||
- Review storage configuration
|
||||
- Verify workspace settings
|
||||
- Test with a small dataset first
|
||||
- Report bugs through project issue tracker
|
||||
@@ -0,0 +1,471 @@
|
||||
# LLM Cache Migration Tool - User Guide
|
||||
|
||||
## Overview
|
||||
|
||||
This tool migrates LightRAG's LLM response cache between different KV storage implementations. It specifically migrates caches generated during file extraction (mode `default`), including entity extraction and summary caches.
|
||||
|
||||
## Supported Storage Types
|
||||
|
||||
1. **JsonKVStorage** - File-based JSON storage
|
||||
2. **RedisKVStorage** - Redis database storage
|
||||
3. **PGKVStorage** - PostgreSQL database storage
|
||||
4. **MongoKVStorage** - MongoDB database storage
|
||||
|
||||
## Cache Types
|
||||
|
||||
The tool migrates the following cache types:
|
||||
- `default:extract:*` - Entity and relationship extraction caches
|
||||
- `default:summary:*` - Entity and relationship summary caches
|
||||
|
||||
**Note**: Query caches (modes like `mix`,`local`, `global`, etc.) are NOT migrated.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
The LLM Cache Migration Tool reads the storage configuration of the LightRAG Server and provides an LLM migration option to select source and destination storage. Ensure that both the source and destination storage have been correctly configured and are accessible via the LightRAG Server before cache migration.
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic Usage
|
||||
|
||||
Run from the LightRAG project root directory:
|
||||
|
||||
```bash
|
||||
python -m lightrag.tools.migrate_llm_cache
|
||||
# or
|
||||
python lightrag/tools/migrate_llm_cache.py
|
||||
```
|
||||
|
||||
### Interactive Workflow
|
||||
|
||||
The tool guides you through the following steps:
|
||||
|
||||
#### 1. Select Source Storage Type
|
||||
```
|
||||
Supported KV Storage Types:
|
||||
[1] JsonKVStorage
|
||||
[2] RedisKVStorage
|
||||
[3] PGKVStorage
|
||||
[4] MongoKVStorage
|
||||
|
||||
Select Source storage type (1-4) (Press Enter to exit): 1
|
||||
```
|
||||
|
||||
**Note**: You can press Enter or type `0` at any storage selection prompt to exit gracefully.
|
||||
|
||||
#### 2. Source Storage Validation
|
||||
The tool will:
|
||||
- Check required environment variables
|
||||
- Auto-detect workspace configuration
|
||||
- Initialize and connect to storage
|
||||
- Count cache records available for migration
|
||||
|
||||
```
|
||||
Checking environment variables...
|
||||
✓ All required environment variables are set
|
||||
|
||||
Initializing Source storage...
|
||||
- Storage Type: JsonKVStorage
|
||||
- Workspace: space1
|
||||
- Connection Status: ✓ Success
|
||||
|
||||
Counting cache records...
|
||||
- Total: 8,734 records
|
||||
```
|
||||
|
||||
**Progress Display by Storage Type:**
|
||||
- **JsonKVStorage**: Fast in-memory counting, displays final count without incremental progress
|
||||
```
|
||||
Counting cache records...
|
||||
- Total: 8,734 records
|
||||
```
|
||||
- **RedisKVStorage**: Real-time scanning progress with incremental counts
|
||||
```
|
||||
Scanning Redis keys... found 8,734 records
|
||||
```
|
||||
- **PostgreSQL**: Quick COUNT(*) query, shows timing only if operation takes >1 second
|
||||
```
|
||||
Counting PostgreSQL records... (took 2.3s)
|
||||
```
|
||||
- **MongoDB**: Fast count_documents(), shows timing only if operation takes >1 second
|
||||
```
|
||||
Counting MongoDB documents... (took 1.8s)
|
||||
```
|
||||
|
||||
#### 3. Select Target Storage Type
|
||||
|
||||
The tool automatically excludes the source storage type from the target selection and renumbers the remaining options sequentially:
|
||||
|
||||
```
|
||||
Available Storage Types for Target (source: JsonKVStorage excluded):
|
||||
[1] RedisKVStorage
|
||||
[2] PGKVStorage
|
||||
[3] MongoKVStorage
|
||||
|
||||
Select Target storage type (1-3) (Press Enter or 0 to exit): 1
|
||||
```
|
||||
|
||||
**Important Notes:**
|
||||
- You **cannot** select the same storage type for both source and target
|
||||
- Options are automatically renumbered (e.g., [1], [2], [3] instead of [2], [3], [4])
|
||||
- You can press Enter or type `0` to exit at this point as well
|
||||
|
||||
The tool then validates the target storage following the same process as the source (checking environment variables, initializing connection, counting records).
|
||||
|
||||
#### 4. Confirm Migration
|
||||
|
||||
```
|
||||
==================================================
|
||||
Migration Confirmation
|
||||
Source: JsonKVStorage (workspace: space1) - 8,734 records
|
||||
Target: MongoKVStorage (workspace: space1) - 0 records
|
||||
Batch Size: 1,000 records/batch
|
||||
Memory Mode: Streaming (memory-optimized)
|
||||
|
||||
⚠️ Warning: Target storage already has 0 records
|
||||
Migration will overwrite records with the same keys
|
||||
|
||||
Continue? (y/n): y
|
||||
```
|
||||
|
||||
#### 5. Execute Migration
|
||||
|
||||
The tool uses **streaming migration** by default for memory efficiency. Observe migration progress:
|
||||
|
||||
```
|
||||
=== Starting Streaming Migration ===
|
||||
💡 Memory-optimized mode: Processing 1,000 records at a time
|
||||
|
||||
Batch 1/9: ████████░░░░░░░░░░░░ 1000/8734 (11.4%) - default:extract ✓
|
||||
Batch 2/9: ████████████░░░░░░░░ 2000/8734 (22.9%) - default:extract ✓
|
||||
...
|
||||
Batch 9/9: ████████████████████ 8734/8734 (100.0%) - default:summary ✓
|
||||
|
||||
Persisting data to disk...
|
||||
✓ Data persisted successfully
|
||||
```
|
||||
|
||||
**Key Features:**
|
||||
- **Streaming mode**: Processes data in batches without loading entire dataset into memory
|
||||
- **Real-time progress**: Shows progress bar with precise percentage and cache type
|
||||
- **Success indicators**: ✓ for successful batches, ✗ for failed batches
|
||||
- **Constant memory usage**: Handles millions of records efficiently
|
||||
|
||||
#### 6. Review Migration Report
|
||||
|
||||
The tool provides a comprehensive final report showing statistics and any errors encountered:
|
||||
|
||||
**Successful Migration:**
|
||||
```
|
||||
Migration Complete - Final Report
|
||||
|
||||
📊 Statistics:
|
||||
Total source records: 8,734
|
||||
Total batches: 9
|
||||
Successful batches: 9
|
||||
Failed batches: 0
|
||||
Successfully migrated: 8,734
|
||||
Failed to migrate: 0
|
||||
Success rate: 100.00%
|
||||
|
||||
✓ SUCCESS: All records migrated successfully!
|
||||
```
|
||||
|
||||
**Migration with Errors:**
|
||||
```
|
||||
Migration Complete - Final Report
|
||||
|
||||
📊 Statistics:
|
||||
Total source records: 8,734
|
||||
Total batches: 9
|
||||
Successful batches: 8
|
||||
Failed batches: 1
|
||||
Successfully migrated: 7,734
|
||||
Failed to migrate: 1,000
|
||||
Success rate: 88.55%
|
||||
|
||||
⚠️ Errors encountered: 1
|
||||
|
||||
Error Details:
|
||||
------------------------------------------------------------
|
||||
|
||||
Error Summary:
|
||||
- ConnectionError: 1 occurrence(s)
|
||||
|
||||
First 5 errors:
|
||||
|
||||
1. Batch 2
|
||||
Type: ConnectionError
|
||||
Message: Connection timeout after 30s
|
||||
Records lost: 1,000
|
||||
|
||||
⚠️ WARNING: Migration completed with errors!
|
||||
Please review the error details above.
|
||||
```
|
||||
|
||||
## Technical Details
|
||||
|
||||
### Workspace Handling
|
||||
|
||||
The tool retrieves workspace in the following priority order:
|
||||
|
||||
1. **Storage-specific workspace environment variables**
|
||||
- PGKVStorage: `POSTGRES_WORKSPACE`
|
||||
- MongoKVStorage: `MONGODB_WORKSPACE`
|
||||
- RedisKVStorage: `REDIS_WORKSPACE`
|
||||
|
||||
2. **Generic workspace environment variable**
|
||||
- `WORKSPACE`
|
||||
|
||||
3. **Default value**
|
||||
- Empty string (uses storage's default workspace)
|
||||
|
||||
### Batch Migration
|
||||
|
||||
- Default batch size: 1000 records/batch
|
||||
- Avoids memory overflow from loading too much data at once
|
||||
- Each batch is committed independently, supporting resume capability
|
||||
|
||||
### Memory-Efficient Pagination
|
||||
|
||||
For large datasets, the tool implements storage-specific pagination strategies:
|
||||
|
||||
- **JsonKVStorage**: Direct in-memory access (data already loaded in shared storage)
|
||||
- **RedisKVStorage**: Cursor-based SCAN with pipeline batching (1000 keys/batch)
|
||||
- **PGKVStorage**: SQL LIMIT/OFFSET pagination (1000 records/batch)
|
||||
- **MongoKVStorage**: Cursor streaming with batch_size (1000 documents/batch)
|
||||
|
||||
This ensures the tool can handle millions of cache records without memory issues.
|
||||
|
||||
### Prefix Filtering Implementation
|
||||
|
||||
The tool uses optimized filtering methods for different storage types:
|
||||
|
||||
- **JsonKVStorage**: Direct dictionary iteration with lock protection
|
||||
- **RedisKVStorage**: SCAN command with namespace-prefixed patterns + pipeline for bulk GET
|
||||
- **PGKVStorage**: SQL LIKE queries with proper field mapping (id, return_value, etc.)
|
||||
- **MongoKVStorage**: MongoDB regex queries on `_id` field with cursor streaming
|
||||
|
||||
## Error Handling & Resilience
|
||||
|
||||
The tool implements comprehensive error tracking to ensure transparent and resilient migrations:
|
||||
|
||||
### Batch-Level Error Tracking
|
||||
- Each batch is independently error-checked
|
||||
- Failed batches are logged but don't stop the migration
|
||||
- Successful batches are committed even if later batches fail
|
||||
- Real-time progress shows ✓ (success) or ✗ (failed) for each batch
|
||||
|
||||
### Error Reporting
|
||||
After migration completes, a detailed report includes:
|
||||
- **Statistics**: Total records, success/failure counts, success rate
|
||||
- **Error Summary**: Grouped by error type with occurrence counts
|
||||
- **Error Details**: Batch number, error type, message, and records lost
|
||||
- **Recommendations**: Clear indication of success or need for review
|
||||
|
||||
### No Double Data Loading
|
||||
- Unlike traditional verification approaches, the tool does NOT reload all target data
|
||||
- Errors are detected during migration, not after
|
||||
- This eliminates memory overhead and handles pre-existing target data correctly
|
||||
|
||||
## Important Notes
|
||||
|
||||
1. **Data Overwrite Warning**
|
||||
- Migration will overwrite records with the same keys in the target storage
|
||||
- Tool displays a warning if target storage already has data
|
||||
- Data migration can be performed repeatedly
|
||||
- Pre-existing data in target storage is handled correctly
|
||||
3. **Interrupt and Resume**
|
||||
- Migration can be interrupted at any time (Ctrl+C)
|
||||
- Already migrated data will remain in target storage
|
||||
- Re-running will overwrite existing records
|
||||
- Failed batches can be manually retried
|
||||
4. **Performance Considerations**
|
||||
- Large data migration may take considerable time
|
||||
- Recommend migrating during off-peak hours
|
||||
- Ensure stable network connection (for remote databases)
|
||||
- Memory usage stays constant regardless of dataset size
|
||||
|
||||
## Storage Configuration
|
||||
|
||||
The tool supports multiple configuration methods with the following priority:
|
||||
|
||||
1. **Environment variables** (highest priority)
|
||||
2. **config.ini file** (medium priority)
|
||||
3. **Default values** (lowest priority)
|
||||
|
||||
#### Option A: Environment Variable Configuration
|
||||
|
||||
Configure storage settings in your `.env` file:
|
||||
|
||||
#### Workspace Configuration (Optional)
|
||||
|
||||
```bash
|
||||
# Generic workspace (shared by all storages)
|
||||
WORKSPACE=space1
|
||||
|
||||
# Or configure independent workspace for specific storage
|
||||
POSTGRES_WORKSPACE=pg_space
|
||||
MONGODB_WORKSPACE=mongo_space
|
||||
REDIS_WORKSPACE=redis_space
|
||||
```
|
||||
|
||||
**Workspace Priority**: Storage-specific > Generic WORKSPACE > Empty string
|
||||
|
||||
#### JsonKVStorage
|
||||
|
||||
```bash
|
||||
WORKING_DIR=./rag_storage
|
||||
```
|
||||
|
||||
#### RedisKVStorage
|
||||
|
||||
```bash
|
||||
REDIS_URI=redis://localhost:6379
|
||||
```
|
||||
|
||||
#### PGKVStorage
|
||||
|
||||
```bash
|
||||
POSTGRES_HOST=localhost
|
||||
POSTGRES_PORT=5432
|
||||
POSTGRES_USER=your_username
|
||||
POSTGRES_PASSWORD=your_password
|
||||
POSTGRES_DATABASE=your_database
|
||||
```
|
||||
|
||||
#### MongoKVStorage
|
||||
|
||||
```bash
|
||||
MONGO_URI=mongodb://root:root@localhost:27017/
|
||||
MONGO_DATABASE=LightRAG
|
||||
```
|
||||
|
||||
#### Option B: config.ini Configuration
|
||||
|
||||
Alternatively, create a `config.ini` file in the project root:
|
||||
|
||||
```ini
|
||||
[redis]
|
||||
uri = redis://localhost:6379
|
||||
|
||||
[postgres]
|
||||
host = localhost
|
||||
port = 5432
|
||||
user = postgres
|
||||
password = yourpassword
|
||||
database = lightrag
|
||||
|
||||
[mongodb]
|
||||
uri = mongodb://root:root@localhost:27017/
|
||||
database = LightRAG
|
||||
```
|
||||
|
||||
**Note**: Environment variables take precedence over config.ini settings. JsonKVStorage uses `WORKING_DIR` environment variable or defaults to `./rag_storage`.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Missing Environment Variables
|
||||
```
|
||||
✗ Missing required environment variables: POSTGRES_USER, POSTGRES_PASSWORD
|
||||
```
|
||||
**Solution**: Add missing variables to your `.env` file
|
||||
|
||||
### Connection Failed
|
||||
```
|
||||
✗ Initialization failed: Connection refused
|
||||
```
|
||||
**Solutions**:
|
||||
- Check if database service is running
|
||||
- Verify connection parameters (host, port, credentials)
|
||||
- Check firewall settings
|
||||
|
||||
**Solutions**:
|
||||
- Check migration process for error logs
|
||||
- Re-run migration tool
|
||||
- Check target storage capacity and permissions
|
||||
|
||||
## Example Scenarios
|
||||
|
||||
### Scenario 1: JSON to MongoDB Migration
|
||||
|
||||
Use case: Migrating from single-machine development to production
|
||||
|
||||
```bash
|
||||
# 1. Configure environment variables
|
||||
WORKSPACE=production
|
||||
MONGO_URI=mongodb://user:pass@prod-server:27017/
|
||||
MONGO_DATABASE=LightRAG
|
||||
|
||||
# 2. Run tool
|
||||
python -m lightrag.tools.migrate_llm_cache
|
||||
|
||||
# 3. Select: 1 (JsonKVStorage) -> 1 (MongoKVStorage - renumbered from 4)
|
||||
```
|
||||
|
||||
**Note**: After selecting JsonKVStorage as source, MongoKVStorage will be shown as option [1] in the target selection since options are renumbered after excluding the source.
|
||||
|
||||
### Scenario 2: Redis to PostgreSQL
|
||||
|
||||
Use case: Migrating from cache storage to relational database
|
||||
|
||||
```bash
|
||||
# 1. Ensure both databases are accessible
|
||||
REDIS_URI=redis://old-redis:6379
|
||||
POSTGRES_HOST=new-postgres-server
|
||||
# ... Other PostgreSQL configs
|
||||
|
||||
# 2. Run tool
|
||||
python -m lightrag.tools.migrate_llm_cache
|
||||
|
||||
# 3. Select: 2 (RedisKVStorage) -> 2 (PGKVStorage - renumbered from 3)
|
||||
```
|
||||
|
||||
**Note**: After selecting RedisKVStorage as source, PGKVStorage will be shown as option [2] in the target selection.
|
||||
|
||||
### Scenario 3: Different Workspaces Migration
|
||||
|
||||
Use case: Migrating data between different workspace environments
|
||||
|
||||
```bash
|
||||
# Configure separate workspaces for source and target
|
||||
POSTGRES_WORKSPACE=dev_workspace # For development environment
|
||||
MONGODB_WORKSPACE=prod_workspace # For production environment
|
||||
|
||||
# Run tool
|
||||
python -m lightrag.tools.migrate_llm_cache
|
||||
|
||||
# Select: 3 (PGKVStorage with dev_workspace) -> 3 (MongoKVStorage with prod_workspace)
|
||||
```
|
||||
|
||||
**Note**: This allows you to migrate between different logical data partitions while changing storage backends.
|
||||
|
||||
## Tool Limitations
|
||||
|
||||
1. **Same Storage Type Not Allowed**
|
||||
- You cannot migrate between the same storage type (e.g., PostgreSQL to PostgreSQL)
|
||||
- This is enforced by the tool automatically excluding the source storage type from target selection
|
||||
- For same-storage migrations (e.g., database switches), use database-native tools instead
|
||||
2. **Only Default Mode Caches**
|
||||
- Only migrates `default:extract:*` and `default:summary:*`
|
||||
- Query caches are not included
|
||||
4. **Network Dependency**
|
||||
- Tool requires stable network connection for remote databases
|
||||
- Large datasets may fail if connection is interrupted
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Backup Before Migration**
|
||||
- Always backup your data before migration
|
||||
- Test migration on non-production data first
|
||||
|
||||
2. **Verify Results**
|
||||
- Check the verification output after migration
|
||||
- Manually verify a few cache entries if needed
|
||||
|
||||
3. **Monitor Performance**
|
||||
- Watch database resource usage during migration
|
||||
- Consider migrating in smaller batches if needed
|
||||
|
||||
4. **Clean Old Data**
|
||||
- After successful migration, consider cleaning old cache data
|
||||
- Keep backups for a reasonable period before deletion
|
||||
@@ -0,0 +1,180 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Diagnostic tool to check LightRAG initialization status.
|
||||
|
||||
This tool helps developers verify that their LightRAG instance is properly
|
||||
initialized before use, preventing common initialization errors.
|
||||
|
||||
Usage:
|
||||
python -m lightrag.tools.check_initialization
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
# Add parent directory to path for imports
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent.parent))
|
||||
|
||||
from lightrag import LightRAG
|
||||
from lightrag.base import StoragesStatus
|
||||
|
||||
|
||||
async def check_lightrag_setup(rag_instance: LightRAG, verbose: bool = False) -> bool:
|
||||
"""
|
||||
Check if a LightRAG instance is properly initialized.
|
||||
|
||||
Args:
|
||||
rag_instance: The LightRAG instance to check
|
||||
verbose: If True, print detailed diagnostic information
|
||||
|
||||
Returns:
|
||||
True if properly initialized, False otherwise
|
||||
"""
|
||||
issues = []
|
||||
warnings = []
|
||||
|
||||
print("🔍 Checking LightRAG initialization status...\n")
|
||||
|
||||
# Check storage initialization status
|
||||
if not hasattr(rag_instance, "_storages_status"):
|
||||
issues.append("LightRAG instance missing _storages_status attribute")
|
||||
elif rag_instance._storages_status != StoragesStatus.INITIALIZED:
|
||||
issues.append(
|
||||
f"Storages not initialized (status: {rag_instance._storages_status.name})"
|
||||
)
|
||||
else:
|
||||
print("✅ Storage status: INITIALIZED")
|
||||
|
||||
# Check individual storage components
|
||||
storage_components = [
|
||||
("full_docs", "Document storage"),
|
||||
("text_chunks", "Text chunks storage"),
|
||||
("entities_vdb", "Entity vector database"),
|
||||
("relationships_vdb", "Relationship vector database"),
|
||||
("chunks_vdb", "Chunks vector database"),
|
||||
("doc_status", "Document status tracker"),
|
||||
("llm_response_cache", "LLM response cache"),
|
||||
("full_entities", "Entity storage"),
|
||||
("full_relations", "Relation storage"),
|
||||
("chunk_entity_relation_graph", "Graph storage"),
|
||||
]
|
||||
|
||||
if verbose:
|
||||
print("\n📦 Storage Components:")
|
||||
|
||||
for component, description in storage_components:
|
||||
if not hasattr(rag_instance, component):
|
||||
issues.append(f"Missing storage component: {component} ({description})")
|
||||
else:
|
||||
storage = getattr(rag_instance, component)
|
||||
if storage is None:
|
||||
warnings.append(f"Storage {component} is None (might be optional)")
|
||||
elif hasattr(storage, "_storage_lock"):
|
||||
if storage._storage_lock is None:
|
||||
issues.append(f"Storage {component} not initialized (lock is None)")
|
||||
elif verbose:
|
||||
print(f" ✅ {description}: Ready")
|
||||
elif verbose:
|
||||
print(f" ✅ {description}: Ready")
|
||||
|
||||
# Check pipeline status
|
||||
try:
|
||||
from lightrag.kg.shared_storage import get_namespace_data
|
||||
|
||||
get_namespace_data("pipeline_status")
|
||||
print("✅ Pipeline status: INITIALIZED")
|
||||
except KeyError:
|
||||
issues.append(
|
||||
"Pipeline status not initialized - call initialize_pipeline_status()"
|
||||
)
|
||||
except Exception as e:
|
||||
issues.append(f"Error checking pipeline status: {str(e)}")
|
||||
|
||||
# Print results
|
||||
print("\n" + "=" * 50)
|
||||
|
||||
if issues:
|
||||
print("❌ Issues found:\n")
|
||||
for issue in issues:
|
||||
print(f" • {issue}")
|
||||
|
||||
print("\n📝 To fix, run this initialization sequence:\n")
|
||||
print(" await rag.initialize_storages()")
|
||||
print(" from lightrag.kg.shared_storage import initialize_pipeline_status")
|
||||
print(" await initialize_pipeline_status()")
|
||||
print(
|
||||
"\n📚 Documentation: https://github.com/HKUDS/LightRAG#important-initialization-requirements"
|
||||
)
|
||||
|
||||
if warnings and verbose:
|
||||
print("\n⚠️ Warnings (might be normal):")
|
||||
for warning in warnings:
|
||||
print(f" • {warning}")
|
||||
|
||||
return False
|
||||
else:
|
||||
print("✅ LightRAG is properly initialized and ready to use!")
|
||||
|
||||
if warnings and verbose:
|
||||
print("\n⚠️ Warnings (might be normal):")
|
||||
for warning in warnings:
|
||||
print(f" • {warning}")
|
||||
|
||||
return True
|
||||
|
||||
|
||||
async def demo():
|
||||
"""Demonstrate the diagnostic tool with a test instance."""
|
||||
from lightrag.llm.openai import openai_embed, gpt_4o_mini_complete
|
||||
from lightrag.kg.shared_storage import initialize_pipeline_status
|
||||
|
||||
print("=" * 50)
|
||||
print("LightRAG Initialization Diagnostic Tool")
|
||||
print("=" * 50)
|
||||
|
||||
# Create test instance
|
||||
rag = LightRAG(
|
||||
working_dir="./test_diagnostic",
|
||||
embedding_func=openai_embed,
|
||||
llm_model_func=gpt_4o_mini_complete,
|
||||
)
|
||||
|
||||
print("\n🔴 BEFORE initialization:\n")
|
||||
await check_lightrag_setup(rag, verbose=True)
|
||||
|
||||
print("\n" + "=" * 50)
|
||||
print("\n🔄 Initializing...\n")
|
||||
await rag.initialize_storages()
|
||||
await initialize_pipeline_status()
|
||||
|
||||
print("\n🟢 AFTER initialization:\n")
|
||||
await check_lightrag_setup(rag, verbose=True)
|
||||
|
||||
# Cleanup
|
||||
import shutil
|
||||
|
||||
shutil.rmtree("./test_diagnostic", ignore_errors=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import argparse
|
||||
|
||||
parser = argparse.ArgumentParser(description="Check LightRAG initialization status")
|
||||
parser.add_argument(
|
||||
"--demo", action="store_true", help="Run a demonstration with a test instance"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--verbose",
|
||||
"-v",
|
||||
action="store_true",
|
||||
help="Show detailed diagnostic information",
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.demo:
|
||||
asyncio.run(demo())
|
||||
else:
|
||||
print("Run with --demo to see the diagnostic tool in action")
|
||||
print("Or import this module and use check_lightrag_setup() with your instance")
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,179 @@
|
||||
"""
|
||||
Download all necessary cache files for offline deployment.
|
||||
|
||||
This module provides a CLI command to download tiktoken model cache files
|
||||
for offline environments where internet access is not available.
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def download_tiktoken_cache(cache_dir: str = None, models: list = None):
|
||||
"""Download tiktoken models to local cache
|
||||
|
||||
Args:
|
||||
cache_dir: Directory to store the cache files. If None, uses default location.
|
||||
models: List of model names to download. If None, downloads common models.
|
||||
|
||||
Returns:
|
||||
Tuple of (success_count, failed_models)
|
||||
"""
|
||||
try:
|
||||
import tiktoken
|
||||
except ImportError:
|
||||
print("Error: tiktoken is not installed.")
|
||||
print("Install with: pip install tiktoken")
|
||||
sys.exit(1)
|
||||
|
||||
# Set cache directory if provided
|
||||
if cache_dir:
|
||||
cache_dir = os.path.abspath(cache_dir)
|
||||
os.environ["TIKTOKEN_CACHE_DIR"] = cache_dir
|
||||
cache_path = Path(cache_dir)
|
||||
cache_path.mkdir(parents=True, exist_ok=True)
|
||||
print(f"Using cache directory: {cache_dir}")
|
||||
else:
|
||||
cache_dir = os.environ.get(
|
||||
"TIKTOKEN_CACHE_DIR", str(Path.home() / ".tiktoken_cache")
|
||||
)
|
||||
print(f"Using default cache directory: {cache_dir}")
|
||||
|
||||
# Common models used by LightRAG and OpenAI
|
||||
if models is None:
|
||||
models = [
|
||||
"gpt-4o-mini", # Default model for LightRAG
|
||||
"gpt-4o", # GPT-4 Omni
|
||||
"gpt-4", # GPT-4
|
||||
"gpt-3.5-turbo", # GPT-3.5 Turbo
|
||||
"text-embedding-ada-002", # Legacy embedding model
|
||||
"text-embedding-3-small", # Small embedding model
|
||||
"text-embedding-3-large", # Large embedding model
|
||||
]
|
||||
|
||||
print(f"\nDownloading {len(models)} tiktoken models...")
|
||||
print("=" * 70)
|
||||
|
||||
success_count = 0
|
||||
failed_models = []
|
||||
|
||||
for i, model in enumerate(models, 1):
|
||||
try:
|
||||
print(f"[{i}/{len(models)}] Downloading {model}...", end=" ", flush=True)
|
||||
encoding = tiktoken.encoding_for_model(model)
|
||||
# Trigger download by encoding a test string
|
||||
encoding.encode("test")
|
||||
print("✓ Done")
|
||||
success_count += 1
|
||||
except KeyError as e:
|
||||
print(f"✗ Failed: Unknown model '{model}'")
|
||||
failed_models.append((model, str(e)))
|
||||
except Exception as e:
|
||||
print(f"✗ Failed: {e}")
|
||||
failed_models.append((model, str(e)))
|
||||
|
||||
print("=" * 70)
|
||||
print(f"\n✓ Successfully cached {success_count}/{len(models)} models")
|
||||
|
||||
if failed_models:
|
||||
print(f"\n✗ Failed to download {len(failed_models)} models:")
|
||||
for model, error in failed_models:
|
||||
print(f" - {model}: {error}")
|
||||
|
||||
print(f"\nCache location: {cache_dir}")
|
||||
print("\nFor offline deployment:")
|
||||
print(" 1. Copy directory to offline server:")
|
||||
print(f" tar -czf tiktoken_cache.tar.gz {cache_dir}")
|
||||
print(" scp tiktoken_cache.tar.gz user@offline-server:/path/to/")
|
||||
print("")
|
||||
print(" 2. On offline server, extract and set environment variable:")
|
||||
print(" tar -xzf tiktoken_cache.tar.gz")
|
||||
print(" export TIKTOKEN_CACHE_DIR=/path/to/tiktoken_cache")
|
||||
print("")
|
||||
print(" 3. Or copy to default location:")
|
||||
print(f" cp -r {cache_dir} ~/.tiktoken_cache/")
|
||||
|
||||
return success_count, failed_models
|
||||
|
||||
|
||||
def main():
|
||||
"""Main entry point for the CLI command"""
|
||||
import argparse
|
||||
|
||||
parser = argparse.ArgumentParser(
|
||||
prog="lightrag-download-cache",
|
||||
description="Download cache files for LightRAG offline deployment",
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
epilog="""
|
||||
Examples:
|
||||
# Download to default location (~/.tiktoken_cache)
|
||||
lightrag-download-cache
|
||||
|
||||
# Download to specific directory
|
||||
lightrag-download-cache --cache-dir ./offline_cache/tiktoken
|
||||
|
||||
# Download specific models only
|
||||
lightrag-download-cache --models gpt-4o-mini gpt-4
|
||||
|
||||
For more information, visit: https://github.com/HKUDS/LightRAG
|
||||
""",
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--cache-dir",
|
||||
help="Cache directory path (default: ~/.tiktoken_cache)",
|
||||
default=None,
|
||||
)
|
||||
parser.add_argument(
|
||||
"--models",
|
||||
nargs="+",
|
||||
help="Specific models to download (default: common models)",
|
||||
default=None,
|
||||
)
|
||||
parser.add_argument(
|
||||
"--version", action="version", version="%(prog)s (LightRAG cache downloader)"
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
print("=" * 70)
|
||||
print("LightRAG Offline Cache Downloader")
|
||||
print("=" * 70)
|
||||
|
||||
try:
|
||||
success_count, failed_models = download_tiktoken_cache(
|
||||
args.cache_dir, args.models
|
||||
)
|
||||
|
||||
print("\n" + "=" * 70)
|
||||
print("Download Complete")
|
||||
print("=" * 70)
|
||||
|
||||
# Exit with error code if all downloads failed
|
||||
if success_count == 0:
|
||||
print("\n✗ All downloads failed. Please check your internet connection.")
|
||||
sys.exit(1)
|
||||
# Exit with warning code if some downloads failed
|
||||
elif failed_models:
|
||||
print(
|
||||
f"\n⚠ Some downloads failed ({len(failed_models)}/{success_count + len(failed_models)})"
|
||||
)
|
||||
sys.exit(2)
|
||||
else:
|
||||
print("\n✓ All cache files downloaded successfully!")
|
||||
sys.exit(0)
|
||||
|
||||
except KeyboardInterrupt:
|
||||
print("\n\n✗ Download interrupted by user")
|
||||
sys.exit(130)
|
||||
except Exception as e:
|
||||
print(f"\n\n✗ Error: {e}")
|
||||
import traceback
|
||||
|
||||
traceback.print_exc()
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,95 @@
|
||||
# 3D GraphML Viewer
|
||||
|
||||
一个基于 Dear ImGui 和 ModernGL 的交互式 3D 图可视化工具。
|
||||
|
||||
## 功能特点
|
||||
|
||||
- **3D 交互式可视化**: 使用 ModernGL 实现高性能的 3D 图形渲染
|
||||
- **多种布局算法**: 支持多种图布局方式
|
||||
- Spring 布局
|
||||
- Circular 布局
|
||||
- Shell 布局
|
||||
- Random 布局
|
||||
- **社区检测**: 支持图社区结构的自动检测和可视化
|
||||
- **交互控制**:
|
||||
- WASD + QE 键控制相机移动
|
||||
- 鼠标右键拖拽控制视角
|
||||
- 节点选择和高亮
|
||||
- 可调节节点大小和边宽度
|
||||
- 可控制标签显示
|
||||
- 可在节点的Connections间快速跳转
|
||||
- **社区检测**: 支持图社区结构的自动检测和可视化
|
||||
- **交互控制**:
|
||||
- WASD + QE 键控制相机移动
|
||||
- 鼠标右键拖拽控制视角
|
||||
- 节点选择和高亮
|
||||
- 可调节节点大小和边宽度
|
||||
- 可控制标签显示
|
||||
|
||||
## 技术栈
|
||||
|
||||
- **imgui_bundle**: 用户界面
|
||||
- **ModernGL**: OpenGL 图形渲染
|
||||
- **NetworkX**: 图数据结构和算法
|
||||
- **NumPy**: 数值计算
|
||||
- **community**: 社区检测
|
||||
|
||||
## 使用方法
|
||||
|
||||
1. **启动程序**:
|
||||
```bash
|
||||
pip install lightrag-hku[tools]
|
||||
lightrag-viewer
|
||||
```
|
||||
|
||||
2. **加载字体**:
|
||||
- 将中文字体文件 `font.ttf` 放置在 `assets` 目录下
|
||||
- 或者修改 `CUSTOM_FONT` 常量来使用其他字体文件
|
||||
|
||||
3. **加载图文件**:
|
||||
- 点击界面上的 "Load GraphML" 按钮
|
||||
- 选择 GraphML 格式的图文件
|
||||
|
||||
4. **交互控制**:
|
||||
- **相机移动**:
|
||||
- W: 前进
|
||||
- S: 后退
|
||||
- A: 左移
|
||||
- D: 右移
|
||||
- Q: 上升
|
||||
- E: 下降
|
||||
- **视角控制**:
|
||||
- 按住鼠标右键拖动来旋转视角
|
||||
- **节点交互**:
|
||||
- 鼠标悬停可高亮节点
|
||||
- 点击可选中节点
|
||||
|
||||
5. **可视化设置**:
|
||||
- 可通过 UI 控制面板调整:
|
||||
- 布局类型
|
||||
- 节点大小
|
||||
- 边的宽度
|
||||
- 标签显示
|
||||
- 标签大小
|
||||
- 背景颜色
|
||||
|
||||
## 自定义设置
|
||||
|
||||
- **节点缩放**: 通过 `node_scale` 参数调整节点大小
|
||||
- **边宽度**: 通过 `edge_width` 参数调整边的宽度
|
||||
- **标签显示**: 可通过 `show_labels` 开关标签显示
|
||||
- **标签大小**: 使用 `label_size` 调整标签大小
|
||||
- **标签颜色**: 通过 `label_color` 设置标签颜色
|
||||
- **视距控制**: 使用 `label_culling_distance` 控制标签显示的最大距离
|
||||
|
||||
## 性能优化
|
||||
|
||||
- 使用 ModernGL 进行高效的图形渲染
|
||||
- 视距裁剪优化标签显示
|
||||
- 社区检测算法优化大规模图的可视化效果
|
||||
|
||||
## 系统要求
|
||||
|
||||
- Python 3.10+
|
||||
- OpenGL 3.3+ 兼容的显卡
|
||||
- 支持的操作系统：Windows/Linux/MacOS
|
||||
@@ -0,0 +1,136 @@
|
||||
# LightRAG 3D Graph Viewer
|
||||
|
||||
An interactive 3D graph visualization tool included in the LightRAG package for visualizing and analyzing RAG (Retrieval-Augmented Generation) graphs and other graph structures.
|
||||
|
||||
|
||||
|
||||
## Installation
|
||||
|
||||
### Quick Install
|
||||
```bash
|
||||
pip install lightrag-hku[tools] # Install with visualization tool only
|
||||
# or
|
||||
pip install lightrag-hku[api,tools] # Install with both API and visualization tools
|
||||
```
|
||||
|
||||
## Launch the Viewer
|
||||
```bash
|
||||
lightrag-viewer
|
||||
```
|
||||
|
||||
## Features
|
||||
|
||||
- **3D Interactive Visualization**: High-performance 3D graphics rendering using ModernGL
|
||||
- **Multiple Layout Algorithms**: Support for various graph layouts
|
||||
- Spring layout
|
||||
- Circular layout
|
||||
- Shell layout
|
||||
- Random layout
|
||||
- **Community Detection**: Automatic detection and visualization of graph community structures
|
||||
- **Interactive Controls**:
|
||||
- WASD + QE keys for camera movement
|
||||
- Right mouse drag for view angle control
|
||||
- Node selection and highlighting
|
||||
- Adjustable node size and edge width
|
||||
- Configurable label display
|
||||
- Quick navigation between node connections
|
||||
|
||||
## Tech Stack
|
||||
|
||||
- **imgui_bundle**: User interface
|
||||
- **ModernGL**: OpenGL graphics rendering
|
||||
- **NetworkX**: Graph data structures and algorithms
|
||||
- **NumPy**: Numerical computations
|
||||
- **community**: Community detection
|
||||
|
||||
## Interactive Controls
|
||||
|
||||
### Camera Movement
|
||||
- W: Move forward
|
||||
- S: Move backward
|
||||
- A: Move left
|
||||
- D: Move right
|
||||
- Q: Move up
|
||||
- E: Move down
|
||||
|
||||
### View Control
|
||||
- Hold right mouse button and drag to rotate view
|
||||
|
||||
### Node Interaction
|
||||
- Hover mouse to highlight nodes
|
||||
- Click to select nodes
|
||||
|
||||
## Visualization Settings
|
||||
|
||||
Adjustable via UI control panel:
|
||||
- Layout type
|
||||
- Node size
|
||||
- Edge width
|
||||
- Label visibility
|
||||
- Label size
|
||||
- Background color
|
||||
|
||||
## Customization Options
|
||||
|
||||
- **Node Scaling**: Adjust node size via `node_scale` parameter
|
||||
- **Edge Width**: Modify edge width using `edge_width` parameter
|
||||
- **Label Display**: Toggle label visibility with `show_labels`
|
||||
- **Label Size**: Adjust label size using `label_size`
|
||||
- **Label Color**: Set label color through `label_color`
|
||||
- **View Distance**: Control maximum label display distance with `label_culling_distance`
|
||||
|
||||
## System Requirements
|
||||
|
||||
- Python 3.9+
|
||||
- Graphics card with OpenGL 3.3+ support
|
||||
- Supported Operating Systems: Windows/Linux/MacOS
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **Command Not Found**
|
||||
```bash
|
||||
# Make sure you installed with the 'tools' option
|
||||
pip install lightrag-hku[tools]
|
||||
|
||||
# Verify installation
|
||||
pip list | grep lightrag-hku
|
||||
```
|
||||
|
||||
2. **ModernGL Initialization Failed**
|
||||
```bash
|
||||
# Check OpenGL version
|
||||
glxinfo | grep "OpenGL version"
|
||||
|
||||
# Update graphics drivers if needed
|
||||
```
|
||||
|
||||
3. **Font Loading Issues**
|
||||
- The required fonts are included in the package
|
||||
- If issues persist, check your graphics drivers
|
||||
|
||||
## Usage with LightRAG
|
||||
|
||||
The viewer is particularly useful for:
|
||||
- Visualizing RAG knowledge graphs
|
||||
- Analyzing document relationships
|
||||
- Exploring semantic connections
|
||||
- Debugging retrieval patterns
|
||||
|
||||
## Performance Optimizations
|
||||
|
||||
- Efficient graphics rendering using ModernGL
|
||||
- View distance culling for label display optimization
|
||||
- Community detection algorithms for optimized visualization of large-scale graphs
|
||||
|
||||
## Support
|
||||
|
||||
- GitHub Issues: [LightRAG Repository](https://github.com/HKUDS/LightRAG)
|
||||
- Documentation: [LightRAG Docs](https://URL-to-docs)
|
||||
|
||||
## License
|
||||
|
||||
This tool is part of LightRAG and is distributed under the MIT License. See `LICENSE` for more information.
|
||||
|
||||
Note: This visualization tool is an optional component of the LightRAG package. Install with the [tools] option to access the viewer functionality.
|
||||
Binary file not shown.
@@ -0,0 +1,92 @@
|
||||
Copyright (c) 2023 Vercel, in collaboration with basement.studio
|
||||
|
||||
This Font Software is licensed under the SIL Open Font License, Version 1.1.
|
||||
This license is copied below, and is also available with a FAQ at:
|
||||
http://scripts.sil.org/OFL
|
||||
|
||||
-----------------------------------------------------------
|
||||
SIL OPEN FONT LICENSE Version 1.1 - 26 February 2007
|
||||
-----------------------------------------------------------
|
||||
|
||||
PREAMBLE
|
||||
The goals of the Open Font License (OFL) are to stimulate worldwide
|
||||
development of collaborative font projects, to support the font creation
|
||||
efforts of academic and linguistic communities, and to provide a free and
|
||||
open framework in which fonts may be shared and improved in partnership
|
||||
with others.
|
||||
|
||||
The OFL allows the licensed fonts to be used, studied, modified and
|
||||
redistributed freely as long as they are not sold by themselves. The
|
||||
fonts, including any derivative works, can be bundled, embedded,
|
||||
redistributed and/or sold with any software provided that any reserved
|
||||
names are not used by derivative works. The fonts and derivatives,
|
||||
however, cannot be released under any other type of license. The
|
||||
requirement for fonts to remain under this license does not apply
|
||||
to any document created using the fonts or their derivatives.
|
||||
|
||||
DEFINITIONS
|
||||
"Font Software" refers to the set of files released by the Copyright
|
||||
Holder(s) under this license and clearly marked as such. This may
|
||||
include source files, build scripts and documentation.
|
||||
|
||||
"Reserved Font Name" refers to any names specified as such after the
|
||||
copyright statement(s).
|
||||
|
||||
"Original Version" refers to the collection of Font Software components as
|
||||
distributed by the Copyright Holder(s).
|
||||
|
||||
"Modified Version" refers to any derivative made by adding to, deleting,
|
||||
or substituting -- in part or in whole -- any of the components of the
|
||||
Original Version, by changing formats or by porting the Font Software to a
|
||||
new environment.
|
||||
|
||||
"Author" refers to any designer, engineer, programmer, technical
|
||||
writer or other person who contributed to the Font Software.
|
||||
|
||||
PERMISSION AND CONDITIONS
|
||||
Permission is hereby granted, free of charge, to any person obtaining
|
||||
a copy of the Font Software, to use, study, copy, merge, embed, modify,
|
||||
redistribute, and sell modified and unmodified copies of the Font
|
||||
Software, subject to the following conditions:
|
||||
|
||||
1) Neither the Font Software nor any of its individual components,
|
||||
in Original or Modified Versions, may be sold by itself.
|
||||
|
||||
2) Original or Modified Versions of the Font Software may be bundled,
|
||||
redistributed and/or sold with any software, provided that each copy
|
||||
contains the above copyright notice and this license. These can be
|
||||
included either as stand-alone text files, human-readable headers or
|
||||
in the appropriate machine-readable metadata fields within text or
|
||||
binary files as long as those fields can be easily viewed by the user.
|
||||
|
||||
3) No Modified Version of the Font Software may use the Reserved Font
|
||||
Name(s) unless explicit written permission is granted by the corresponding
|
||||
Copyright Holder. This restriction only applies to the primary font name as
|
||||
presented to the users.
|
||||
|
||||
4) The name(s) of the Copyright Holder(s) or the Author(s) of the Font
|
||||
Software shall not be used to promote, endorse or advertise any
|
||||
Modified Version, except to acknowledge the contribution(s) of the
|
||||
Copyright Holder(s) and the Author(s) or with their explicit written
|
||||
permission.
|
||||
|
||||
5) The Font Software, modified or unmodified, in part or in whole,
|
||||
must be distributed entirely under this license, and must not be
|
||||
distributed under any other license. The requirement for fonts to
|
||||
remain under this license does not apply to any document created
|
||||
using the Font Software.
|
||||
|
||||
TERMINATION
|
||||
This license becomes null and void if any of the above conditions are
|
||||
not met.
|
||||
|
||||
DISCLAIMER
|
||||
THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
||||
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF
|
||||
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT
|
||||
OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE
|
||||
COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
|
||||
INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL
|
||||
DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
|
||||
FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM
|
||||
OTHER DEALINGS IN THE FONT SOFTWARE.
|
||||
@@ -0,0 +1,93 @@
|
||||
Copyright (c) 2022--2024, atelierAnchor <https://atelier-anchor.com>,
|
||||
with Reserved Font Name <Smiley> and <得意黑>.
|
||||
|
||||
This Font Software is licensed under the SIL Open Font License, Version 1.1.
|
||||
This license is copied below, and is also available with a FAQ at:
|
||||
http://scripts.sil.org/OFL
|
||||
|
||||
-----------------------------------------------------------
|
||||
SIL OPEN FONT LICENSE Version 1.1 - 26 February 2007
|
||||
-----------------------------------------------------------
|
||||
|
||||
PREAMBLE
|
||||
The goals of the Open Font License (OFL) are to stimulate worldwide
|
||||
development of collaborative font projects, to support the font creation
|
||||
efforts of academic and linguistic communities, and to provide a free and
|
||||
open framework in which fonts may be shared and improved in partnership
|
||||
with others.
|
||||
|
||||
The OFL allows the licensed fonts to be used, studied, modified and
|
||||
redistributed freely as long as they are not sold by themselves. The
|
||||
fonts, including any derivative works, can be bundled, embedded,
|
||||
redistributed and/or sold with any software provided that any reserved
|
||||
names are not used by derivative works. The fonts and derivatives,
|
||||
however, cannot be released under any other type of license. The
|
||||
requirement for fonts to remain under this license does not apply
|
||||
to any document created using the fonts or their derivatives.
|
||||
|
||||
DEFINITIONS
|
||||
"Font Software" refers to the set of files released by the Copyright
|
||||
Holder(s) under this license and clearly marked as such. This may
|
||||
include source files, build scripts and documentation.
|
||||
|
||||
"Reserved Font Name" refers to any names specified as such after the
|
||||
copyright statement(s).
|
||||
|
||||
"Original Version" refers to the collection of Font Software components as
|
||||
distributed by the Copyright Holder(s).
|
||||
|
||||
"Modified Version" refers to any derivative made by adding to, deleting,
|
||||
or substituting -- in part or in whole -- any of the components of the
|
||||
Original Version, by changing formats or by porting the Font Software to a
|
||||
new environment.
|
||||
|
||||
"Author" refers to any designer, engineer, programmer, technical
|
||||
writer or other person who contributed to the Font Software.
|
||||
|
||||
PERMISSION & CONDITIONS
|
||||
Permission is hereby granted, free of charge, to any person obtaining
|
||||
a copy of the Font Software, to use, study, copy, merge, embed, modify,
|
||||
redistribute, and sell modified and unmodified copies of the Font
|
||||
Software, subject to the following conditions:
|
||||
|
||||
1) Neither the Font Software nor any of its individual components,
|
||||
in Original or Modified Versions, may be sold by itself.
|
||||
|
||||
2) Original or Modified Versions of the Font Software may be bundled,
|
||||
redistributed and/or sold with any software, provided that each copy
|
||||
contains the above copyright notice and this license. These can be
|
||||
included either as stand-alone text files, human-readable headers or
|
||||
in the appropriate machine-readable metadata fields within text or
|
||||
binary files as long as those fields can be easily viewed by the user.
|
||||
|
||||
3) No Modified Version of the Font Software may use the Reserved Font
|
||||
Name(s) unless explicit written permission is granted by the corresponding
|
||||
Copyright Holder. This restriction only applies to the primary font name as
|
||||
presented to the users.
|
||||
|
||||
4) The name(s) of the Copyright Holder(s) or the Author(s) of the Font
|
||||
Software shall not be used to promote, endorse or advertise any
|
||||
Modified Version, except to acknowledge the contribution(s) of the
|
||||
Copyright Holder(s) and the Author(s) or with their explicit written
|
||||
permission.
|
||||
|
||||
5) The Font Software, modified or unmodified, in part or in whole,
|
||||
must be distributed entirely under this license, and must not be
|
||||
distributed under any other license. The requirement for fonts to
|
||||
remain under this license does not apply to any document created
|
||||
using the Font Software.
|
||||
|
||||
TERMINATION
|
||||
This license becomes null and void if any of the above conditions are
|
||||
not met.
|
||||
|
||||
DISCLAIMER
|
||||
THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
||||
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF
|
||||
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT
|
||||
OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE
|
||||
COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
|
||||
INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL
|
||||
DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
|
||||
FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM
|
||||
OTHER DEALINGS IN THE FONT SOFTWARE.
|
||||
Binary file not shown.
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,8 @@
|
||||
imgui_bundle
|
||||
moderngl
|
||||
networkx
|
||||
numpy
|
||||
pyglm
|
||||
python-louvain
|
||||
scipy
|
||||
tk
|
||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user