[Feat] 텍스트 파싱 및 Chunk 저장 - #76
Merged
Merged
Conversation
|
Warning Review limit reached
Next review available in: 18 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (5)
📝 WalkthroughWalkthroughMinIO의 TXT·Markdown 원문을 엄격한 UTF-8로 정규화하고 Unicode Code Point 기준 Chunk로 분할해 Changes문서 Chunk 파이프라인
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant 관리자
participant IndexingJobAdminController
participant DocumentParsingService
participant MinioStorageService
participant DocumentChunkTransactionService
participant Database
관리자->>IndexingJobAdminController: POST /admin/indexing-jobs/{jobId}/attempts/{attemptId}/chunks
IndexingJobAdminController->>DocumentParsingService: createChunks(...)
DocumentParsingService->>DocumentChunkTransactionService: prepare(...)
DocumentChunkTransactionService-->>DocumentParsingService: FileSnapshot 또는 replay
DocumentParsingService->>MinioStorageService: read(StoredFile)
MinioStorageService-->>DocumentParsingService: 원문 bytes
DocumentParsingService->>DocumentChunkTransactionService: complete(..., drafts)
DocumentChunkTransactionService->>Database: Chunk·상태·이벤트 원자적 저장
Database-->>DocumentChunkTransactionService: 저장 결과
DocumentChunkTransactionService-->>IndexingJobAdminController: ChunkResult
IndexingJobAdminController-->>관리자: 201 생성 또는 200 재생 응답
Possibly related PRs
Suggested labels: 🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (3 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Gimini-3
marked this pull request as ready for review
July 29, 2026 06:05
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (2)
src/main/java/com/opensource/docgrid/domain/document/service/command/DocumentChunkTransactionService.java (1)
282-294: 🚀 Performance & Scalability | 🔵 Trivial | 💤 Low valueContent Hash 정규식 매번 컴파일
draft.contentHash().matches(...)는 호출마다Pattern을 새로 컴파일합니다. Chunk 수가 많은 대용량 문서(최대 10MB, 최대 약 1.2만개)에서는 Job/Version Lock을 쥔 채로 이 검증이 반복되므로,static final Pattern으로 미리 컴파일해두면 Lock 보유 시간을 소폭이나마 줄일 수 있습니다. 다만 실질적 영향은 크지 않아 급하지 않은 개선입니다.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/main/java/com/opensource/docgrid/domain/document/service/command/DocumentChunkTransactionService.java` around lines 282 - 294, Update DocumentChunkTransactionService by introducing a static final precompiled Pattern for the 64-character lowercase hexadecimal content hash, and replace the matches call in validateDraft with that Pattern. Preserve the existing validation and exception behavior.src/test/java/com/opensource/docgrid/domain/document/service/FixedSizeChunkerTest.java (1)
1-123: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win
overlap >= chunkSize경계에 대한 테스트가 없습니다.
FixedSizeChunker.chunk()는overlap >= chunkSize일 때 무한 루프 또는 예외를 유발할 수 있는데(별도 코멘트 참고), 이를 검증하는 테스트가 없습니다. 이 값에 대한 방어 로직을 추가한 뒤 회귀 테스트도 함께 추가하는 것을 권장합니다.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/test/java/com/opensource/docgrid/domain/document/service/FixedSizeChunkerTest.java` around lines 1 - 123, Extend FixedSizeChunkerTest to cover configurations where overlap is equal to or greater than chunkSize, and assert the intended guarded behavior after updating FixedSizeChunker.chunk() to reject or safely handle these invalid settings without looping or throwing unexpectedly. Add regression coverage for both boundary values while preserving existing valid-overlap behavior.Source: Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/design/text-parsing-chunk-storage.md`:
- Line 1: Rename the design document currently named
text-parsing-chunk-storage.md to follow the required
{github아이디}-#68-text-parsing-chunk-storage.md convention, replacing {github아이디}
with the actual author’s GitHub ID while preserving the existing document
contents.
In
`@src/main/java/com/opensource/docgrid/domain/document/service/FixedSizeChunker.java`:
- Around line 42-68: Validate at the start of FixedSizeChunker.chunk() that
overlap is non-negative and strictly less than chunkSize, failing fast with an
appropriate exception otherwise. Add regression coverage in FixedSizeChunkerTest
for overlap equal to chunkSize and greater than chunkSize, verifying both cases
throw.
In
`@src/main/java/com/opensource/docgrid/domain/document/storage/MinioStorageService.java`:
- Around line 63-67: Update logStorageReadFailure() to stop logging
storedFile.bucketName() and storedFile.objectKey() directly; remove those MinIO
location values from the error message or replace them with an approved
correlation identifier, while preserving the existing failure logging and
DocGridException behavior in the surrounding read flow.
In
`@src/test/java/com/opensource/docgrid/domain/embedding/controller/IndexingJobAdminControllerTest.java`:
- Around line 359-373: Extend invalidChunkRequests() with a case for a
whitespace-only claimToken, using {"workerId": 1, "claimToken": " "}, and retain
the expected invalid-request behavior so the /chunks contract test verifies a
400 response for `@NotBlank` validation.
---
Nitpick comments:
In
`@src/main/java/com/opensource/docgrid/domain/document/service/command/DocumentChunkTransactionService.java`:
- Around line 282-294: Update DocumentChunkTransactionService by introducing a
static final precompiled Pattern for the 64-character lowercase hexadecimal
content hash, and replace the matches call in validateDraft with that Pattern.
Preserve the existing validation and exception behavior.
In
`@src/test/java/com/opensource/docgrid/domain/document/service/FixedSizeChunkerTest.java`:
- Around line 1-123: Extend FixedSizeChunkerTest to cover configurations where
overlap is equal to or greater than chunkSize, and assert the intended guarded
behavior after updating FixedSizeChunker.chunk() to reject or safely handle
these invalid settings without looping or throwing unexpectedly. Add regression
coverage for both boundary values while preserving existing valid-overlap
behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 985e52c3-5084-4b5a-a4d9-ad952aff852a
📒 Files selected for processing (34)
.codex/hooks.json.env.exampledocs/design/text-parsing-chunk-storage.mdsrc/main/java/com/opensource/docgrid/domain/document/config/DocumentChunkingProperties.javasrc/main/java/com/opensource/docgrid/domain/document/entity/DocumentVersion.javasrc/main/java/com/opensource/docgrid/domain/document/repository/DocumentChunkRepository.javasrc/main/java/com/opensource/docgrid/domain/document/repository/DocumentVersionRepository.javasrc/main/java/com/opensource/docgrid/domain/document/service/DocumentChunkDraft.javasrc/main/java/com/opensource/docgrid/domain/document/service/DocumentParsingService.javasrc/main/java/com/opensource/docgrid/domain/document/service/FixedSizeChunker.javasrc/main/java/com/opensource/docgrid/domain/document/service/TextDocumentParser.javasrc/main/java/com/opensource/docgrid/domain/document/service/command/DocumentChunkTransactionService.javasrc/main/java/com/opensource/docgrid/domain/document/storage/FileStorageService.javasrc/main/java/com/opensource/docgrid/domain/document/storage/MinioStorageService.javasrc/main/java/com/opensource/docgrid/domain/embedding/controller/IndexingJobAdminController.javasrc/main/java/com/opensource/docgrid/domain/embedding/dto/request/CreateDocumentChunksRequest.javasrc/main/java/com/opensource/docgrid/domain/embedding/dto/response/DocumentChunksResponse.javasrc/main/java/com/opensource/docgrid/domain/embedding/service/command/EmbeddingJobAttemptService.javasrc/main/java/com/opensource/docgrid/domain/embedding/service/command/EmbeddingJobOwnershipValidator.javasrc/main/java/com/opensource/docgrid/global/exception/ErrorCode.javasrc/main/resources/application.ymlsrc/test/java/com/opensource/docgrid/domain/document/config/DocumentChunkingPropertiesTest.javasrc/test/java/com/opensource/docgrid/domain/document/entity/DocumentVersionTest.javasrc/test/java/com/opensource/docgrid/domain/document/integration/DocumentChunkingIntegrationTest.javasrc/test/java/com/opensource/docgrid/domain/document/integration/minio/MinioDocumentReadIntegrationTest.javasrc/test/java/com/opensource/docgrid/domain/document/integration/minio/MinioUploadConcurrencyIntegrationTest.javasrc/test/java/com/opensource/docgrid/domain/document/service/DocumentParsingServiceTest.javasrc/test/java/com/opensource/docgrid/domain/document/service/FixedSizeChunkerTest.javasrc/test/java/com/opensource/docgrid/domain/document/service/TextDocumentParserTest.javasrc/test/java/com/opensource/docgrid/domain/document/service/command/DocumentChunkTransactionServiceTest.javasrc/test/java/com/opensource/docgrid/domain/document/storage/MinioStorageServiceTest.javasrc/test/java/com/opensource/docgrid/domain/embedding/controller/IndexingJobAdminControllerTest.javasrc/test/java/com/opensource/docgrid/domain/embedding/service/command/EmbeddingJobAttemptServiceTest.javasrc/test/java/com/opensource/docgrid/domain/embedding/service/command/EmbeddingJobOwnershipValidatorTest.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🔍️ 작업 내용
현재 Worker가 소유한
PROCESSINGEmbedding Job과STARTEDAttempt를 실행 Context로 검증한 뒤,MinIO의 TXT·Markdown 원본을 결정적인 Unicode Code Point 기반 Chunk로 분할해
document_chunks에 저장합니다.201 Created200 OKPOST /admin/indexing-jobs/{jobId}/attempts/{attemptId}/chunks✨ 상세 설명
1. 전체 요청 흐름
PROCESSING상태, 소유권 필드,Worker, Claim Token과 Lease를 검증합니다.
STARTED상태인지 확인합니다.Document.currentVersion은 처리 대상 판단에 사용하지 않습니다.UPLOADED → PARSING으로 전환하고PARSE_STARTED이벤트를같은 Transaction에 저장합니다.
Stream을 닫습니다.
CRLF·CR → LF 정규화를 수행합니다. Markdown 문법과 일반 공백은 보존합니다.
소문자 SHA-256 Hash를 계산합니다.
준비 단계 Version ID를 재검증합니다.
DocumentChunk로 변환해saveAllAndFlush하고,Version을
CHUNKED로 전환하며CHUNKED이벤트를 저장합니다.2. 설계·동작 원리
짧은 두 Transaction과 외부 작업 분리
DocumentParsingService에는@Transactional을 적용하지 않았습니다. 준비와 완료는 별도 Spring Bean인DocumentChunkTransactionService의 서로 다른 Public 메서드가 담당합니다. 따라서 MinIO I/O와UTF-8 Decode·Hash 계산 동안 DB Connection과 Job·Version 행 잠금을 유지하지 않습니다.
준비 단계는 JPA Entity 대신 Version ID, Document Type과
StoredFile만 가진 불변 Snapshot을반환합니다. 외부 작업 결과도 Entity가 아닌
DocumentChunkDraft목록이므로 Persistence Context가Transaction 밖으로 새지 않습니다.
외부 작업 전후의 소유권 재검증
MinIO 읽기와 계산 중 Lease가 만료되거나 Claim Token이 교체될 수 있습니다. 완료 Transaction은 Lock을
다시 획득한 뒤 새 현재 시각으로 Worker·Token·Lease와 Attempt를 다시 검증합니다.
과거 Worker가 늦게 계산한 결과는 Chunk Insert 전에 거부됩니다.
Lock 순서와 원자성
모든 경로는 Embedding Job을 먼저, Document Version을 다음 순서로 잠급니다. Claim 교체와 같은 Job
중복 요청은 Job Lock으로, 동일 Version의 Chunk Set 경합은 Version Lock으로 직렬화합니다.
완료 Transaction의 Chunk 전체 Insert·Flush,
CHUNKED상태 전환과 완료 이벤트 중 하나라도 실패하면모두 Rollback됩니다. 기존
(document_version_id, chunk_index)Unique Constraint는 Service 경로를우회한 중복 Insert의 최종 방어선입니다.
멱등성과 동시성
UPLOADED → PARSING과PARSE_STARTED는 한 번만 기록합니다.PARSING상태 재개는 외부 계산을 다시 수행할 수 있지만 시작 이벤트를 추가하지 않습니다.CHUNKED재호출은 MinIO·Parser·Chunker를 호출하지 않고 저장된 Chunk 수를 반환합니다.CHUNKED를 확인하고 같은 결과를 재생합니다.결정적인 Text와 Chunk 규칙
[charStart, charEnd)nullJava UTF-16
char가 아니라 Code Point 배열로 범위를 계산하므로 Emoji의 Surrogate Pair를 중간에서자르지 않습니다. 같은 Canonical Text와 설정은 같은 Draft와 DB Row를 만듭니다.
Storage와 오류 분리
FileStorageService.read(StoredFile)계약을 추가했습니다.MinioStorageService는 전달된 Bucket과Object Key로
GetObject를 호출하고 try-with-resources 안에서 모든 Byte를 읽어 Stream을 닫습니다.NoSuchKey,NoSuchObject:DOCUMENT-STORAGE-002, 404DOCUMENT-STORAGE-001, 503보안과 외부 노출
기존
/admin/**ADMIN 정책을 재사용하므로SecurityConfig는 변경하지 않았습니다.Claim Token, 원문, Chunk 본문, Bucket과 Object Key는 성공 응답에 포함하지 않습니다.
Token과 본문은 이벤트와 일반 로그에도 남기지 않습니다.
3. 설정과 외부 연결
Chunk 크기는 양수, Overlap은 0 이상이면서 Chunk 크기보다 작아야 합니다.
같은 배포군의 Worker는 결정성을 위해 같은 설정을 사용해야 합니다.
기존 Spring Boot, Spring Data JPA, OpenSQL/PostgreSQL, Flyway와 MinIO Java SDK만 사용합니다.
새 Schema, Migration, Message Broker, Cache, Markdown Library와 Production Dependency는
추가하지 않았습니다.
4. 기능별 커밋과 코드 변경
7c71cd2— Embedding Job 소유권 검증기 추가EmbeddingJobOwnershipValidator를 추가했습니다.PROCESSING상태, 소유권 필드 완전성, Worker·Token 일치와 Lease를 순서대로 검증합니다.재사용할 수 있게 했습니다.
0a3cae8— 소유권 검증 계약 Test53ca125— Attempt 시작의 공통 Validator 위임EmbeddingJobAttemptService의 중복 Private 검증 메서드를 제거했습니다.d0b15ef— 프로젝트 경로 독립 Hookbackend절대 경로를 참조하던.codex/hooks.json을 저장소 상대 경로로 바꿨습니다.b7f8e6a— 결정적 Parser·Chunker 구현DocumentChunkingProperties,TextDocumentParser,FixedSizeChunker,DocumentChunkDraft를 추가했습니다.application.yml,.env.example과 Parser 관련 ErrorCode를 추가했습니다.d64f80c— Parser·Chunker 단위 Test74312d7— MinIO 원본 읽기FileStorageService에read(StoredFile)을 추가했습니다.MinioStorageService에GetObject, 전체 Byte 읽기, Stream 종료와Object 없음·일반 장애 분리를 구현했습니다.
109fa88— MinIO 읽기 단위 TestNoSuchKey와 일반 I/O 오류의 ErrorCode 변환을 검증했습니다.b09adbe— Version 상태와 Chunk RepositoryDocumentVersion.markParsing,markChunked에 허용 상태 Guard를 추가했습니다.DocumentVersionRepository.findByIdForUpdate를 추가했습니다.DocumentChunkRepository에 Version별 존재·개수 조회를 추가했습니다.43b028c— Version 상태 전이 TestUPLOADED → PARSING → CHUNKED, 잘못된 상태의 두 전이 거부와 기존 후속 전이를 포함한12개 Test를 추가했습니다.
45f9c9e— Chunk Transaction·Orchestration·관리자 APIDocumentChunkTransactionService에 준비·완료 Transaction을 구현했습니다.DocumentParsingService가 두 Transaction 사이의 MinIO·Parser·Chunker를 조정합니다.추가했습니다.
aa727f9— API·원자성·동시성·실제 MinIO Test9b9f176— 설계 문서 확정docs/design/text-parsing-chunk-storage.md에 범위, 전체 흐름, 상태, Transaction, Lock,오류, 보안, 설정, Test와 후속 경계를 기록했습니다.
develop기준과 실제 기능 커밋 구성을 반영했습니다.5. 검증 결과
./gradlew clean test: 319개 성공DocumentChunkingIntegrationTest: 2개 성공MinioDocumentReadIntegrationTest: 3개 성공git diff --check: 성공🛠️ 추후 리팩토링 및 고도화 계획
PARSE_FAILED, Attempt 종료와 오류 이력 기록📸 스크린샷 (선택)
💬 리뷰 요구사항
PARSING재개와CHUNKED재생의 이벤트·Row 비증가 계약을 확인 부탁드립니다.의견 부탁드립니다.
Summary by CodeRabbit