130 Commits

Author SHA1 Message Date
0c34ea7dcb hyperparam_with_modeltype 2026-02-12 18:48:14 +09:00
3547c28361 Merge pull request 'feat/training_260202' (#55) from feat/training_260202 into develop
Reviewed-on: #55
2026-02-12 16:56:23 +09:00
6c70bfed18 Merge remote-tracking branch 'origin/feat/training_260202' into feat/training_260202
# Conflicts:
#	src/main/java/com/kamco/cd/training/postgres/core/ModelTrainMngCoreService.java
2026-02-12 16:55:52 +09:00
95a75e63f4 임시폴더생성 api 추가 2026-02-12 16:55:10 +09:00
2a1dbee290 Merge pull request '모델학습 1단계 실행중인 것이 있는지 count API' (#54) from feat/training_260202 into develop
Reviewed-on: #54
2026-02-12 16:51:09 +09:00
384a321bf3 모델학습 1단계 실행중인 것이 있는지 count API 2026-02-12 16:50:40 +09:00
f4e97d389b Merge pull request 'file 확인 API 수정' (#53) from feat/training_260202 into develop
Reviewed-on: #53
2026-02-12 16:42:20 +09:00
590810ff0a file 확인 API 수정 2026-02-12 16:41:40 +09:00
a01c872982 Merge pull request 'feat/training_260202' (#52) from feat/training_260202 into develop
Reviewed-on: #52
2026-02-12 16:15:11 +09:00
905a245070 Merge branch 'feat/training_260202' of https://kamco.git.gs.dabeeo.com/MVPTeam/kamco-train-api into feat/training_260202 2026-02-12 16:14:45 +09:00
860ce35a8f docker mount 경로 추가 2026-02-12 16:14:19 +09:00
7f3f5dca40 Merge pull request 'feat/training_260202' (#51) from feat/training_260202 into develop
Reviewed-on: #51
2026-02-12 16:13:19 +09:00
4a0a4e35ed 학습 실행 수정 2026-02-12 16:12:58 +09:00
ae055dca1e 모델등록 수정 2026-02-12 16:01:14 +09:00
26e8e1492f Merge pull request 'feat/training_260202' (#50) from feat/training_260202 into develop
Reviewed-on: #50
2026-02-12 15:52:09 +09:00
8fa722011c 모델등록 수정 2026-02-12 15:51:54 +09:00
17d47d6200 Merge branch 'feat/training_260202' of https://kamco.git.gs.dabeeo.com/MVPTeam/kamco-train-api into feat/training_260202 2026-02-12 15:47:10 +09:00
e178f58fe2 chunk save log 추가 2026-02-12 15:47:06 +09:00
cd0cf5726d Merge pull request 'feat/training_260202' (#49) from feat/training_260202 into develop
Reviewed-on: #49
2026-02-12 15:44:11 +09:00
8e4bea53da 모델등록 수정 2026-02-12 15:43:52 +09:00
7a22d8ba73 containerName 생성 변경 2026-02-12 15:39:12 +09:00
2df4a7a80b csv 파일 읽는 경로 읽어서 수정, train은 epoch + 1 해서 저장 2026-02-12 15:24:30 +09:00
b451f697bc 모델 마스터 테이블 request,response 경로 추가 2026-02-12 14:59:35 +09:00
7e9c867f34 Merge pull request '모델 등록할 때 step1State를 READY로 업데이트' (#48) from feat/training_260202 into develop
Reviewed-on: #48
2026-02-12 14:35:52 +09:00
130e85f8a1 모델 등록할 때 step1State를 READY로 업데이트 2026-02-12 14:35:17 +09:00
9e713cb49d Merge pull request '업로드 로직 재수정' (#47) from feat/training_260202 into develop
Reviewed-on: #47
2026-02-12 14:21:57 +09:00
51dfa97900 업로드 로직 재수정 2026-02-12 14:21:08 +09:00
87c6b599b4 Merge pull request 'feat/training_260202' (#46) from feat/training_260202 into develop
Reviewed-on: #46
2026-02-12 12:10:04 +09:00
f50855a822 Merge branch 'feat/training_260202' of https://kamco.git.gs.dabeeo.com/MVPTeam/kamco-train-api into feat/training_260202 2026-02-12 12:08:04 +09:00
8d416317a8 베스트 에폭 API, 2단계 실행 시 best epoch 업데이트 2026-02-12 12:07:44 +09:00
22aa071476 Merge pull request 'feat/training_260202' (#45) from feat/training_260202 into develop
Reviewed-on: #45
2026-02-12 12:06:04 +09:00
a83bd09f8f containerName 생성 변경 2026-02-12 12:05:30 +09:00
96035f864a containerName 생성 변경 2026-02-12 11:42:38 +09:00
fd7dfd7e7f containerName 생성 변경 2026-02-12 11:10:28 +09:00
190b93bee8 실행 오류 수정 2026-02-12 10:58:51 +09:00
c5f19cc961 실행 오류 수정 2026-02-12 10:58:32 +09:00
c56c0ca605 실행 오류 수정 2026-02-12 10:58:26 +09:00
c6e721aa37 실행 오류 수정 2026-02-12 10:58:12 +09:00
6572e17f00 실행 오류 수정 2026-02-12 10:51:15 +09:00
be6365807c Merge pull request '실행 오류 수정' (#43) from feat/training_260202 into develop
Reviewed-on: #43
2026-02-12 10:20:05 +09:00
d2fff7dfde 실행 오류 수정 2026-02-12 10:19:44 +09:00
f66bc22c95 Merge pull request '실행 오류 수정' (#42) from feat/training_260202 into develop
Reviewed-on: #42
2026-02-12 10:14:54 +09:00
3367d0e7be 실행 오류 수정 2026-02-12 10:14:32 +09:00
352ec6ccb0 Merge pull request 'feat/training_260202' (#41) from feat/training_260202 into develop
Reviewed-on: #41
2026-02-12 09:53:02 +09:00
6a989255a3 모델별 데이터셋 목록 - G2,G3 dataTypeName 추가 2026-02-12 09:52:24 +09:00
878b21573f 테스트 실행 추가 2026-02-11 22:00:35 +09:00
0602db1436 Merge pull request '테스트 실행 추가' (#40) from feat/training_260202 into develop
Reviewed-on: #40
2026-02-11 21:58:58 +09:00
2f8bd1f98c 테스트 실행 추가 2026-02-11 21:58:25 +09:00
75231ccbba Merge pull request '추론 실행 추가' (#39) from feat/training_260202 into develop
Reviewed-on: #39
2026-02-11 20:22:01 +09:00
1249a80da5 추론 실행 추가 2026-02-11 20:21:25 +09:00
00c78eb42f Merge pull request '성능정보 그래프 데이터 API 추가' (#38) from feat/training_260202 into develop
Reviewed-on: #38
2026-02-11 19:52:23 +09:00
35767adba1 성능정보 그래프 데이터 API 추가 2026-02-11 19:52:00 +09:00
47a2a159ef Merge pull request 'test metrics 스케줄 추가' (#37) from feat/training_260202 into develop
Reviewed-on: #37
2026-02-11 19:10:37 +09:00
95548223cd test metrics 스케줄 추가 2026-02-11 19:09:58 +09:00
2debdc5312 Merge pull request 'feat/training_260202' (#36) from feat/training_260202 into develop
Reviewed-on: #36
2026-02-11 18:51:01 +09:00
207cc47f1b 스케줄 주석 2026-02-11 18:50:43 +09:00
b6338bce8e 테이블 구조 변경 2026-02-11 18:49:59 +09:00
2cfa2adcf5 tb_model_master 컬럼 추가 2026-02-11 17:21:48 +09:00
d7e19abfc9 uploadRate 로직 수정 2026-02-11 17:06:02 +09:00
c843703ee7 Merge pull request 'file 가져오기 86 호출하는 거로 추가' (#35) from feat/training_260202 into develop
Reviewed-on: #35
2026-02-11 16:53:25 +09:00
133ea6b1ba file 가져오기 86 호출하는 거로 추가 2026-02-11 16:49:48 +09:00
0df977ae81 Merge pull request '업로드 로직 86으로 수행하기 수정' (#34) from feat/training_260202 into develop
Reviewed-on: #34
2026-02-11 16:33:03 +09:00
3e39006822 업로드 로직 86으로 수행하기 수정 2026-02-11 16:32:40 +09:00
3ec1a71406 Merge pull request '업로드 로직 수정' (#33) from feat/training_260202 into develop
Reviewed-on: #33
2026-02-11 15:53:21 +09:00
16009f1623 업로드 로직 수정 2026-02-11 15:52:57 +09:00
41911014c9 Merge pull request '업로드 로직 수정' (#32) from feat/training_260202 into develop
Reviewed-on: #32
2026-02-11 15:44:54 +09:00
8ea32ce675 업로드 로직 수정 2026-02-11 15:44:18 +09:00
a4ac80c787 Merge pull request '업로드 경로 수정' (#31) from feat/training_260202 into develop
Reviewed-on: #31
2026-02-11 15:11:02 +09:00
3a5d136d34 업로드 경로 수정 2026-02-11 15:10:37 +09:00
2f63b9ddcd Merge pull request 'feat/training_260202' (#30) from feat/training_260202 into develop
Reviewed-on: #30
2026-02-11 14:08:58 +09:00
92de48b55e 전이학습 상세 로직 수정 2026-02-11 14:08:21 +09:00
224ddae68b 전이학습 상세 수정 2026-02-11 14:05:15 +09:00
885b72a0c6 Merge pull request '모델별 데이터셋 목록 조회 수정' (#29) from feat/training_260202 into develop
Reviewed-on: #29
2026-02-11 12:29:08 +09:00
9ac00d37c5 모델별 데이터셋 목록 조회 수정 2026-02-11 12:28:38 +09:00
fbb5a34867 Merge pull request '업로드 경로 원복' (#28) from feat/training_260202 into develop
Reviewed-on: #28
2026-02-11 12:12:43 +09:00
e25fc01b25 업로드 경로 원복 2026-02-11 12:12:08 +09:00
6b3f22dd66 Merge pull request '업로드 파일 max 수정' (#27) from feat/training_260202 into develop
Reviewed-on: #27
2026-02-11 11:53:04 +09:00
abc2c8e806 업로드 파일 max 수정 2026-02-11 11:52:41 +09:00
29e1d0ec7e Merge pull request '업로드 경로 수정' (#26) from feat/training_260202 into develop
Reviewed-on: #26
2026-02-11 11:41:49 +09:00
8d379d064c 업로드 경로 수정 2026-02-11 11:38:55 +09:00
a2072e0148 Merge pull request 'feat/training_260202' (#25) from feat/training_260202 into develop
Reviewed-on: #25
2026-02-11 10:28:38 +09:00
6352f74b08 업로드 성공 후 COMPLETED 해주기 2026-02-10 18:28:42 +09:00
025b573859 학습데이터 obj-list API geojson 로직 수정 2026-02-10 16:36:07 +09:00
0e9fa80092 학습데이터 등록 로직 커밋 2026-02-10 16:15:21 +09:00
2d5de88a6b Merge pull request '학습데이터 업로드, unzip 로직 진행중' (#24) from feat/training_260202 into develop
Reviewed-on: #24
2026-02-10 10:44:28 +09:00
89744d2aa1 학습데이터 업로드, unzip 로직 진행중 2026-02-10 10:43:40 +09:00
eda1d19942 Merge pull request '스웨거 로그인 설정 수정' (#23) from feat/training_260202 into develop
Reviewed-on: #23
2026-02-06 14:54:56 +09:00
b4a4486560 스웨거 로그인 설정 수정 2026-02-06 14:51:45 +09:00
653717a074 Merge pull request 'feat/training_260202' (#22) from feat/training_260202 into develop
Reviewed-on: #22
2026-02-06 11:09:36 +09:00
7cc3392856 사용 가능 공간 조회 API 추가 2026-02-06 11:09:13 +09:00
def84d2b1c 학습데이터 obj 삭제 스웨거 문구 수정 2026-02-06 10:33:12 +09:00
679795d14d Merge pull request '전이학습 추가' (#21) from feat/training_260202 into develop
Reviewed-on: #21
2026-02-05 18:23:30 +09:00
0a7f01a2f5 전이학습 추가 2026-02-05 18:23:07 +09:00
9655c62d35 Merge pull request '하이퍼 파라미터 최적값 조회 수정' (#20) from feat/training_260202 into develop
Reviewed-on: #20
2026-02-05 15:22:56 +09:00
db6844f0e7 하이퍼 파라미터 최적값 조회 수정 2026-02-05 15:22:34 +09:00
af16933378 Merge pull request 'feat/training_260202' (#19) from feat/training_260202 into develop
Reviewed-on: #19
2026-02-05 15:08:41 +09:00
29b653a4e9 데이터셋 상세조회 class 조회 추가 2026-02-05 15:08:22 +09:00
693e3ef3ab 모델학습 데이터셋 선택 목록 수정 2026-02-05 14:22:37 +09:00
03135a972a Merge pull request '모델학습 데이터셋 선택 목록 수정' (#18) from feat/training_260202 into develop
Reviewed-on: #18
2026-02-05 13:59:00 +09:00
381b7d7e0b 모델학습 데이터셋 선택 목록 수정 2026-02-05 13:58:31 +09:00
947cba2742 Merge pull request 'feat/training_260202' (#17) from feat/training_260202 into develop
Reviewed-on: #17
2026-02-04 19:54:23 +09:00
f038fdd1db 모델 상세 추가 API 커밋 2026-02-04 19:54:00 +09:00
474a3c119e 모델학습 config 정보 조회 추가 2026-02-04 19:51:43 +09:00
b2be43a76e 모델 상세 API 커밋 2026-02-04 19:46:57 +09:00
ce69bacb01 모델학습 데이터 등록 수정 2026-02-04 19:15:10 +09:00
200b384e19 모델 종류 이름 변경 2026-02-04 18:41:34 +09:00
350d622e5a 미사용 소스 정리 2026-02-04 18:26:02 +09:00
b25fc6fe68 Merge pull request 'feat/training_260202' (#16) from feat/training_260202 into develop
Reviewed-on: #16
2026-02-04 18:01:15 +09:00
6cdf4efda6 데이터셋 등록 추가 2026-02-04 18:00:56 +09:00
7d866e5869 데이터셋 테이블 수정 2026-02-04 15:13:13 +09:00
5bc59c0e0b Merge pull request '데이터셋 조회 수정' (#15) from feat/training_260202 into develop
Reviewed-on: #15
2026-02-04 14:11:00 +09:00
3c0a12da4e 데이터셋 조회 수정 2026-02-04 14:10:46 +09:00
cdac9d6148 Merge pull request '모델학습 설정 dto 수정' (#14) from feat/training_260202 into develop
Reviewed-on: #14
2026-02-04 14:03:41 +09:00
abe4272227 모델학습 설정 dto 수정 2026-02-04 14:03:25 +09:00
d238debcc8 Merge pull request '모델학습 설정 dto 수정' (#13) from feat/training_260202 into develop
Reviewed-on: #13
2026-02-04 13:57:00 +09:00
fdfda049f8 모델학습 설정 dto 수정 2026-02-04 13:56:32 +09:00
3e780ef007 Merge pull request 'feat/training_260202' (#12) from feat/training_260202 into develop
Reviewed-on: #12
2026-02-04 12:32:36 +09:00
2c825b14ee 모델학습 설정 dto 수정 2026-02-04 12:32:22 +09:00
f9d081970d 모델학습 설정 dto 수정 2026-02-04 12:29:09 +09:00
2110f395b7 Merge pull request 'dataset 테이블 수정, 모델학습 설정 dto 추가' (#11) from feat/training_260202 into develop
Reviewed-on: #11
2026-02-04 12:25:19 +09:00
60d45ee2ce dataset 테이블 수정, 모델학습 설정 dto 추가 2026-02-04 12:24:55 +09:00
50464c1aa8 Merge pull request 'feat/training_260202' (#10) from feat/training_260202 into develop
Reviewed-on: #10
2026-02-03 18:52:24 +09:00
f1ad59d0b1 데이터셋 API 커밋 2026-02-03 18:51:56 +09:00
19644e5c9f dataset 테이블 수정 2026-02-03 18:51:35 +09:00
4c80017fc5 dataset 테이블 수정 2026-02-03 18:46:00 +09:00
5dfcd3d181 Merge pull request '하이퍼파라미터 , 모델관리 수정' (#9) from feat/training_260202 into develop
Reviewed-on: #9
2026-02-03 18:25:13 +09:00
6e99c209d6 하이퍼파라미터 , 모델관리 수정 2026-02-03 18:24:49 +09:00
38b037da31 Merge pull request 'feat/training_260202' (#8) from feat/training_260202 into develop
Reviewed-on: #8
2026-02-03 16:32:02 +09:00
d66711e4f4 하이퍼파라미터 기능 추가 2026-02-03 16:31:43 +09:00
44878e9c37 하이퍼파라미터 기능 추가 2026-02-03 15:15:25 +09:00
111 changed files with 7822 additions and 2102 deletions

View File

@@ -3,6 +3,7 @@ plugins {
id 'org.springframework.boot' version '3.5.7'
id 'io.spring.dependency-management' version '1.1.7'
id 'com.diffplug.spotless' version '6.25.0'
id 'idea'
}
group = 'com.kamco.cd'
@@ -21,11 +22,23 @@ configurations {
}
}
// QueryDSL 생성된 소스 디렉토리 정의
def generatedSourcesDir = file("$buildDir/generated/sources/annotationProcessor/java/main")
repositories {
mavenCentral()
maven { url "https://repo.osgeo.org/repository/release/" }
}
// Gradle이 생성된 소스를 컴파일 경로에 포함하도록 설정
sourceSets {
main {
java {
srcDirs += generatedSourcesDir
}
}
}
dependencies {
implementation 'org.springframework.boot:spring-boot-starter-data-jpa'
implementation 'org.springframework.boot:spring-boot-starter-web'
@@ -83,6 +96,23 @@ dependencies {
implementation 'io.hypersistence:hypersistence-utils-hibernate-63:3.7.0'
implementation 'org.reflections:reflections:0.10.2'
implementation 'com.jcraft:jsch:0.1.55'
implementation 'org.apache.commons:commons-csv:1.10.0'
}
// IntelliJ가 생성된 소스를 인식하도록 설정
idea {
module {
// 소스 디렉토리로 인식
sourceDirs += generatedSourcesDir
// Generated Sources Root로 마킹 (IntelliJ에서 특별 처리)
generatedSourceDirs += generatedSourcesDir
// 소스 및 Javadoc 다운로드
downloadJavadoc = true
downloadSources = true
}
}
configurations.configureEach {
@@ -93,6 +123,21 @@ tasks.named('test') {
useJUnitPlatform()
}
// 컴파일 전 생성된 소스 디렉토리 생성 보장
tasks.named('compileJava') {
doFirst {
generatedSourcesDir.mkdirs()
}
}
// 생성된 소스 정리 태스크
tasks.register('cleanGeneratedSources', Delete) {
delete generatedSourcesDir
}
tasks.named('clean') {
dependsOn 'cleanGeneratedSources'
}
bootJar {
archiveFileName = 'ROOT.jar'

View File

@@ -14,6 +14,7 @@ services:
- /mnt/nfs_share/images:/app/original-images
- /mnt/nfs_share/model_output:/app/model-outputs
- /mnt/nfs_share/train_dataset:/app/train-dataset
- /home/kcomu/data:/home/kcomu/data
networks:
- kamco-cds
restart: unless-stopped

View File

@@ -2,8 +2,10 @@ package com.kamco.cd.training;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.scheduling.annotation.EnableAsync;
import org.springframework.scheduling.annotation.EnableScheduling;
@EnableAsync
@SpringBootApplication
@EnableScheduling
public class KamcoTrainingApplication {

View File

@@ -0,0 +1,159 @@
package com.kamco.cd.training.common.dto;
import com.kamco.cd.training.common.enums.ModelType;
import io.swagger.v3.oas.annotations.media.Schema;
import lombok.AllArgsConstructor;
import lombok.Getter;
import lombok.NoArgsConstructor;
import lombok.Setter;
@Getter
@Setter
@AllArgsConstructor
@NoArgsConstructor
public class HyperParam {
// -------------------------
// Important
// -------------------------
@Schema(description = "모델", example = "large")
private ModelType model; // backbone
@Schema(description = "백본 네트워크", example = "large")
private String backbone; // backbone
@Schema(description = "입력 이미지 크기(H,W)", example = "512,512")
private String inputSize; // input_size
@Schema(description = "크롭 크기(H,W 또는 단일값)", example = "256,256")
private String cropSize; // crop_size
@Schema(description = "배치 크기(Per GPU)", example = "16")
private Integer batchSize; // batch_size
// -------------------------
// Data
// -------------------------
@Schema(description = "Train dataloader workers", example = "16")
private Integer trainNumWorkers; // train_num_workers
@Schema(description = "Val dataloader workers", example = "8")
private Integer valNumWorkers; // val_num_workers
@Schema(description = "Test dataloader workers", example = "8")
private Integer testNumWorkers; // test_num_workers
@Schema(description = "Train shuffle 여부", example = "true")
private Boolean trainShuffle; // train_shuffle
@Schema(description = "Train persistent workers 여부", example = "true")
private Boolean trainPersistent; // train_persistent
@Schema(description = "Val persistent workers 여부", example = "true")
private Boolean valPersistent; // val_persistent
// -------------------------
// Model Architecture
// -------------------------
@Schema(description = "Drop Path 비율", example = "0.3")
private Double dropPathRate; // drop_path_rate
@Schema(description = "Freeze 단계(-1:None)", example = "-1")
private Integer frozenStages; // frozen_stages
@Schema(description = "Neck 결합 정책", example = "abs_diff")
private String neckPolicy; // neck_policy
@Schema(description = "디코더 채널 구성", example = "512,256,128,64")
private String decoderChannels; // decoder_channels
@Schema(description = "클래스별 가중치", example = "1,10")
private String classWeight; // class_weight
// -------------------------
// Loss & Optimization
// -------------------------
@Schema(description = "학습률", example = "0.00006")
private Double learningRate; // learning_rate
@Schema(description = "Weight Decay", example = "0.05")
private Double weightDecay; // weight_decay
@Schema(description = "Layer Decay Rate", example = "0.9")
private Double layerDecayRate; // layer_decay_rate
@Schema(description = "DDP unused params 탐색 여부", example = "true")
private Boolean ddpFindUnusedParams; // ddp_find_unused_params
@Schema(description = "Loss 계산 제외 인덱스", example = "255")
private Integer ignoreIndex; // ignore_index
@Schema(description = "레이어 깊이", example = "24")
private Integer numLayers; // num_layers
// -------------------------
// Evaluation
// -------------------------
@Schema(description = "평가 지표 목록", example = "mFscore,mIoU")
private String metrics; // metrics
@Schema(description = "Best 모델 선정 기준 지표", example = "changed_fscore")
private String saveBest; // save_best
@Schema(description = "Best 모델 선정 규칙", example = "less")
private String saveBestRule; // save_best_rule
@Schema(description = "검증 수행 주기(Epoch)", example = "10")
private Integer valInterval; // val_interval
@Schema(description = "로그 기록 주기(Iteration)", example = "400")
private Integer logInterval; // log_interval
@Schema(description = "시각화 저장 주기(Epoch)", example = "1")
private Integer visInterval; // vis_interval
// -------------------------
// Augmentation
// -------------------------
@Schema(description = "회전 적용 확률", example = "0.5")
private Double rotProb; // rot_prob
@Schema(description = "회전 각도 범위(Min,Max)", example = "-20,20")
private String rotDegree; // rot_degree
@Schema(description = "반전 적용 확률", example = "0.5")
private Double flipProb; // flip_prob
@Schema(description = "채널 교환 확률", example = "0.5")
private Double exchangeProb; // exchange_prob
@Schema(description = "밝기 변화량", example = "10")
private Integer brightnessDelta; // brightness_delta
@Schema(description = "대비 범위(Min,Max)", example = "0.8,1.2")
private String contrastRange; // contrast_range
@Schema(description = "채도 범위(Min,Max)", example = "0.8,1.2")
private String saturationRange; // saturation_range
@Schema(description = "색조 변화량", example = "10")
private Integer hueDelta; // hue_delta
// -------------------------
// Hardware
// -------------------------
@Schema(description = "사용 GPU 개수", example = "4")
private Integer gpuCnt; // gpu_cnt
@Schema(description = "사용 GPU ID 목록", example = "0,1,2,3")
private String gpuIds; // gpu_ids
@Schema(description = "분산학습 마스터 포트", example = "1122")
private Integer masterPort; // master_port
// -------------------------
// Memo
// -------------------------
@Schema(description = "메모", example = "하이퍼파라미터 신규등록")
private String memo; // memo
}

View File

@@ -6,21 +6,22 @@ import lombok.Getter;
@Getter
@AllArgsConstructor
public enum DetectionClassification {
BUILDING("building", "건물", 10),
CONTAINER("container", "컨테이너", 20),
FIELD("field", "경작지", 30),
FOREST("forest", "", 40),
GRASS("grass", "초지", 50),
GREENHOUSE("greenhouse", "비닐하우스", 60),
LAND("land", "일반토지", 70),
ORCHARD("orchard", "과수원", 80),
ROAD("road", "도로", 90),
STONE("stone", "모래/자갈", 100),
TANK("tank", "물탱크", 110),
TUMULUS("tumulus", "토분(무덤)", 120),
WASTE("waste", "폐기물", 130),
WATER("water", "", 140),
ETC("ETC", "기타", 200); // For 'etc' (miscellaneous/other)
ROAD("road", "도로", 10),
BUILDING("building", "건물", 20),
GREENHOUSE("greenhouse", "비닐하우스", 30),
FIELD("field", "논/밭", 40),
ORCHARD("orchard", "과수원", 50),
GRASS("grass", "초지", 60),
FOREST("forest", "", 70),
WATER("water", "", 80),
STONE("stone", "모래/자갈", 90),
WASTE("waste", "적치물", 100),
CONTAINER("container", "컨테이너", 110),
LAND("land", "일반토지", 120),
SOLAR("solar", "태양광", 130),
TANK("tank", "탱크", 140),
NDC("NDC", "미분류", 150),
ETC("ETC", "기타", 160);
private final String id;
private final String desc;

View File

@@ -0,0 +1,27 @@
package com.kamco.cd.training.common.enums;
import com.kamco.cd.training.common.utils.enums.CodeExpose;
import com.kamco.cd.training.common.utils.enums.EnumType;
import lombok.AllArgsConstructor;
import lombok.Getter;
@CodeExpose
@Getter
@AllArgsConstructor
public enum HyperParamSelectType implements EnumType {
OPTIMIZED("최적화 파라미터"),
EXISTING("기존 파라미터"),
NEW("신규 파라미터");
private final String desc;
@Override
public String getId() {
return name();
}
@Override
public String getText() {
return desc;
}
}

View File

@@ -0,0 +1,32 @@
package com.kamco.cd.training.common.enums;
import com.kamco.cd.training.common.utils.enums.CodeExpose;
import com.kamco.cd.training.common.utils.enums.EnumType;
import java.util.Arrays;
import lombok.AllArgsConstructor;
import lombok.Getter;
@CodeExpose
@Getter
@AllArgsConstructor
public enum ModelType implements EnumType {
G1("G1"),
G2("G2"),
G3("G3");
private String desc;
public static ModelType getValueData(String modelNo) {
return Arrays.stream(ModelType.values()).filter(m -> m.getId().equals(modelNo)).findFirst().orElse(G1);
}
@Override
public String getId() {
return name();
}
@Override
public String getText() {
return desc;
}
}

View File

@@ -10,13 +10,21 @@ import lombok.Getter;
@AllArgsConstructor
public enum TrainStatusType implements EnumType {
// @formatter:off
READY("READY", "대기"),
ING("ING", "진행중"),
COMPLETED("COMPLETED", "완료"),
STOPPED("STOPPED", "중단됨"),
ERROR("ERROR", "오류");
// @formatter:on
READY("대기"),
IN_PROGRESS("진행중"),
COMPLETED("완료"),
STOPPED("중단됨"),
ERROR("오류");
private final String id;
private final String text;
private final String desc;
@Override
public String getId() {
return name();
}
@Override
public String getText() {
return desc;
}
}

View File

@@ -0,0 +1,26 @@
package com.kamco.cd.training.common.enums;
import com.kamco.cd.training.common.utils.enums.CodeExpose;
import com.kamco.cd.training.common.utils.enums.EnumType;
import lombok.AllArgsConstructor;
import lombok.Getter;
@CodeExpose
@Getter
@AllArgsConstructor
public enum TrainType implements EnumType {
GENERAL("일반"),
TRANSFER("전이");
private final String desc;
@Override
public String getId() {
return name();
}
@Override
public String getText() {
return desc;
}
}

View File

@@ -0,0 +1,50 @@
package com.kamco.cd.training.common.service;
import java.nio.file.FileStore;
import java.nio.file.Files;
import java.nio.file.Path;
public class FormatStorage {
private FormatStorage() {}
/** 디스크 사용량 조회 */
public static DiskUsage getDiskUsage(Path path) throws Exception {
FileStore store = Files.getFileStore(path);
long total = store.getTotalSpace();
long usable = store.getUsableSpace();
return new DiskUsage(path.toString(), total, usable);
}
/** 디스크 사용량 DTO */
public record DiskUsage(String path, long totalBytes, long usableBytes) {
public long usedBytes() {
return totalBytes - usableBytes;
}
public double usedPercent() {
return totalBytes == 0 ? 0.0 : (usedBytes() * 100.0) / totalBytes;
}
public String totalText() {
return formatStorageSize(totalBytes);
}
public String usableText() {
return formatStorageSize(usableBytes);
}
/** 저장공간을 사람이 읽기 좋은 단위로 변환 (GB / MB) */
private static String formatStorageSize(long bytes) {
double gb = bytes / 1024.0 / 1024 / 1024;
if (gb >= 1) {
return String.format("%.1f GB", gb);
}
double mb = bytes / 1024.0 / 1024;
return String.format("%.0f MB", mb);
}
}
}

View File

@@ -2,9 +2,15 @@ package com.kamco.cd.training.common.utils;
import static java.lang.String.CASE_INSENSITIVE_ORDER;
import com.jcraft.jsch.ChannelExec;
import com.jcraft.jsch.ChannelSftp;
import com.jcraft.jsch.JSch;
import com.jcraft.jsch.Session;
import io.swagger.v3.oas.annotations.media.Schema;
import java.io.BufferedReader;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.FileReader;
import java.io.IOException;
import java.io.InputStream;
@@ -21,17 +27,22 @@ import java.util.Arrays;
import java.util.Comparator;
import java.util.Date;
import java.util.List;
import java.util.Properties;
import java.util.Set;
import java.util.function.Predicate;
import java.util.stream.Collectors;
import java.util.stream.Stream;
import java.util.zip.ZipEntry;
import java.util.zip.ZipInputStream;
import lombok.Getter;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.io.FilenameUtils;
import org.geotools.coverage.grid.GridCoverage2D;
import org.geotools.gce.geotiff.GeoTiffReader;
import org.springframework.util.FileSystemUtils;
import org.springframework.web.multipart.MultipartFile;
@Slf4j
public class FIleChecker {
static SimpleDateFormat dttmFormat = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
@@ -492,16 +503,37 @@ public class FIleChecker {
public static boolean multipartChunkSaveTo(
MultipartFile mfile, String targetPath, int chunkIndex) {
File dest = new File(targetPath, String.valueOf(chunkIndex));
boolean fileUpload = true;
try {
File dir = new File(targetPath);
log.info("targetPath={}", targetPath);
log.info("absolute targetPath={}", dir.getAbsolutePath());
if (!dir.exists()) {
dir.mkdirs();
}
File dest = new File(dir, String.valueOf(chunkIndex));
log.info("real save path = {}", dest.getAbsolutePath());
log.info("chunkIndex={}, uploadSize={}", chunkIndex, mfile.getSize());
log.info("savedSize={}", dest.length());
// ⭐ 핵심
if (dest.exists()) {
dest.delete();
}
log.info("after delete={}", dest.length());
mfile.transferTo(dest);
log.info("after transfer size={}", dest.length());
log.info("after transfer exists={}", dest.exists());
return true;
} catch (IOException e) {
log.error("chunk save error", e);
return false;
}
return true;
}
public static boolean deleteFolder(String path) {
@@ -680,4 +712,198 @@ public class FIleChecker {
this.lastModified = lastModified;
}
}
public static void unzip(String fileName, String destDirectory) throws IOException {
String zipFilePath = destDirectory + File.separator + fileName;
// zip 이름으로 폴더 생성 (확장자 제거)
String folderName =
fileName.endsWith(".zip") ? fileName.substring(0, fileName.length() - 4) : fileName;
File destDir = new File(destDirectory, folderName);
if (!destDir.exists()) {
destDir.mkdirs();
}
try (ZipInputStream zis = new ZipInputStream(new FileInputStream(zipFilePath))) {
ZipEntry zipEntry = zis.getNextEntry();
while (zipEntry != null) {
File newFile = newFile(destDir, zipEntry);
if (zipEntry.isDirectory()) {
if (!newFile.isDirectory() && !newFile.mkdirs()) {
throw new IOException("디렉토리 생성 실패: " + newFile);
}
} else {
// 상위 디렉토리가 없는 경우 생성
File parent = newFile.getParentFile();
if (!parent.exists() && !parent.mkdirs()) {
throw new IOException("상위 디렉토리 생성 실패: " + parent);
}
// 실제 파일 쓰기
try (FileOutputStream fos = new FileOutputStream(newFile)) {
byte[] buffer = new byte[1024];
int len;
while ((len = zis.read(buffer)) > 0) {
fos.write(buffer, 0, len);
}
}
}
zipEntry = zis.getNextEntry();
}
zis.closeEntry();
}
}
public static File newFile(File destinationDir, ZipEntry zipEntry) throws IOException {
File destFile = new File(destinationDir, zipEntry.getName());
String destDirPath = destinationDir.getCanonicalPath();
String destFilePath = destFile.getCanonicalPath();
if (!destFilePath.startsWith(destDirPath + File.separator)) {
throw new IOException("엔트리가 대상 디렉토리를 벗어남: " + zipEntry.getName());
}
return destFile;
}
public static void uploadTo86(Path localFile) {
String host = "192.168.2.86";
int port = 22;
String username = "kcomu";
String password = "Kamco2025!";
String remoteDir = "/home/kcomu/data/request";
Session session = null;
ChannelSftp channel = null;
try {
JSch jsch = new JSch();
session = jsch.getSession(username, host, port);
session.setPassword(password);
Properties config = new Properties();
config.put("StrictHostKeyChecking", "no");
session.setConfig(config);
session.connect(10_000);
channel = (ChannelSftp) session.openChannel("sftp");
channel.connect(10_000);
// 목적지 디렉토리 이동
channel.cd(remoteDir);
// 업로드
channel.put(localFile.toString(), localFile.getFileName().toString());
} catch (Exception e) {
throw new RuntimeException("SFTP upload failed", e);
} finally {
if (channel != null) channel.disconnect();
if (session != null) session.disconnect();
}
}
public static void unzipOn86Server(String zipPath, String targetDir) {
String host = "192.168.2.86";
String user = "kcomu";
String password = "Kamco2025!";
Session session = null;
ChannelExec channel = null;
try {
JSch jsch = new JSch();
session = jsch.getSession(user, host, 22);
session.setPassword(password);
Properties config = new Properties();
config.put("StrictHostKeyChecking", "no");
session.setConfig(config);
session.connect(10_000);
String command = "unzip -o " + zipPath + " -d " + targetDir;
channel = (ChannelExec) session.openChannel("exec");
channel.setCommand(command);
channel.setErrStream(System.err);
InputStream in = channel.getInputStream();
channel.connect();
// 출력 읽기(선택)
try (BufferedReader br = new BufferedReader(new InputStreamReader(in))) {
while (br.readLine() != null) {
// 필요하면 로그
}
}
} catch (Exception e) {
throw new RuntimeException(e);
} finally {
if (channel != null) channel.disconnect();
if (session != null) session.disconnect();
}
}
public static List<String> execCommandAndReadLines(String command) {
List<String> result = new ArrayList<>();
String host = "192.168.2.86";
String user = "kcomu";
String password = "Kamco2025!";
Session session = null;
ChannelExec channel = null;
try {
JSch jsch = new JSch();
session = jsch.getSession(user, host, 22);
session.setPassword(password);
Properties config = new Properties();
config.put("StrictHostKeyChecking", "no");
session.setConfig(config);
session.connect(10_000);
channel = (ChannelExec) session.openChannel("exec");
channel.setCommand(command);
channel.setInputStream(null);
InputStream in = channel.getInputStream();
channel.connect();
try (BufferedReader br = new BufferedReader(new InputStreamReader(in))) {
String line;
while ((line = br.readLine()) != null) {
result.add(line);
}
}
return result;
} catch (Exception e) {
throw new RuntimeException("remote command failed : " + command, e);
} finally {
if (channel != null) channel.disconnect();
if (session != null) session.disconnect();
}
}
}

View File

@@ -164,6 +164,7 @@ public class ApiResponseDto<T> {
NOT_FOUND_USER_FOR_EMAIL("이메일로 유저를 찾을 수 없습니다."),
NOT_FOUND_USER("사용자를 찾을 수 없습니다."),
UNPROCESSABLE_ENTITY("이 데이터는 삭제할 수 없습니다."),
UNPROCESSABLE_ENTITY_UPDATE("이 데이터는 수정할 수 없습니다."),
LOGIN_ID_NOT_FOUND("아이디를 잘못 입력하셨습니다."),
LOGIN_PASSWORD_MISMATCH("비밀번호를 잘못 입력하셨습니다."),
LOGIN_PASSWORD_EXCEEDED("비밀번호 오류 횟수를 초과하여 이용하실 수 없습니다.\n로그인 오류에 대해 관리자에게 문의하시기 바랍니다."),

View File

@@ -1,4 +1,4 @@
package com.kamco.cd.training.config;
package com.kamco.cd.training.config.swagger;
import io.swagger.v3.oas.annotations.enums.SecuritySchemeType;
import io.swagger.v3.oas.annotations.security.SecurityScheme;

View File

@@ -0,0 +1,97 @@
package com.kamco.cd.training.config.swagger;
import jakarta.servlet.http.HttpServletRequest;
import java.nio.charset.StandardCharsets;
import org.springdoc.core.properties.SwaggerUiConfigProperties;
import org.springdoc.core.properties.SwaggerUiOAuthProperties;
import org.springdoc.core.providers.ObjectMapperProvider;
import org.springdoc.webmvc.ui.SwaggerIndexPageTransformer;
import org.springdoc.webmvc.ui.SwaggerIndexTransformer;
import org.springdoc.webmvc.ui.SwaggerWelcomeCommon;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Primary;
import org.springframework.context.annotation.Profile;
import org.springframework.core.io.Resource;
import org.springframework.web.servlet.resource.ResourceTransformerChain;
import org.springframework.web.servlet.resource.TransformedResource;
@Profile({"local", "dev"})
@Configuration
public class SwaggerUiAutoAuthConfig {
@Bean
@Primary
public SwaggerIndexTransformer swaggerIndexTransformer(
SwaggerUiConfigProperties swaggerUiConfigProperties,
SwaggerUiOAuthProperties swaggerUiOAuthProperties,
SwaggerWelcomeCommon swaggerWelcomeCommon,
ObjectMapperProvider objectMapperProvider) {
SwaggerIndexPageTransformer delegate =
new SwaggerIndexPageTransformer(
swaggerUiConfigProperties,
swaggerUiOAuthProperties,
swaggerWelcomeCommon,
objectMapperProvider);
return new SwaggerIndexTransformer() {
private static final String TOKEN_KEY = "SWAGGER_ACCESS_TOKEN";
@Override
public Resource transform(
HttpServletRequest request, Resource resource, ResourceTransformerChain chain) {
try {
// 1) springdoc 기본 변환 먼저 적용
Resource transformed = delegate.transform(request, resource, chain);
String html =
new String(transformed.getInputStream().readAllBytes(), StandardCharsets.UTF_8);
String loginPathContains = "/api/auth/signin";
String inject =
"""
tagsSorter: (a, b) => {
const TOP = '인증(Auth)';
if (a === TOP && b !== TOP) return -1;
if (b === TOP && a !== TOP) return 1;
return a.localeCompare(b);
},
requestInterceptor: (req) => {
const token = localStorage.getItem('%s');
if (token) {
req.headers = req.headers || {};
req.headers['Authorization'] = 'Bearer ' + token;
}
return req;
},
responseInterceptor: async (res) => {
try {
const isLogin = (res?.url?.includes('%s') && res?.status === 200);
if (isLogin) {
const text = (typeof res.data === 'string') ? res.data : JSON.stringify(res.data);
const json = JSON.parse(text);
const token = json?.data?.accessToken;
if (token) {
localStorage.setItem('%s', token);
}
}
} catch (e) {}
return res;
},
"""
.formatted(TOKEN_KEY, loginPathContains, TOKEN_KEY);
html = html.replace("SwaggerUIBundle({", "SwaggerUIBundle({\n" + inject);
return new TransformedResource(transformed, html.getBytes(StandardCharsets.UTF_8));
} catch (Exception e) {
// 실패 시 원본 반환(문서 깨짐 방지)
return resource;
}
}
};
}
}

View File

@@ -2,6 +2,9 @@ package com.kamco.cd.training.dataset;
import com.kamco.cd.training.config.api.ApiResponseDto;
import com.kamco.cd.training.dataset.dto.DatasetDto;
import com.kamco.cd.training.dataset.dto.DatasetObjDto;
import com.kamco.cd.training.dataset.dto.DatasetObjDto.DatasetClass;
import com.kamco.cd.training.dataset.dto.DatasetObjDto.DatasetStorage;
import com.kamco.cd.training.dataset.service.DatasetService;
import io.swagger.v3.oas.annotations.Operation;
import io.swagger.v3.oas.annotations.Parameter;
@@ -11,9 +14,12 @@ import io.swagger.v3.oas.annotations.responses.ApiResponse;
import io.swagger.v3.oas.annotations.responses.ApiResponses;
import io.swagger.v3.oas.annotations.tags.Tag;
import jakarta.validation.Valid;
import java.util.List;
import java.util.UUID;
import lombok.RequiredArgsConstructor;
import org.springframework.core.io.Resource;
import org.springframework.data.domain.Page;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.*;
@Tag(name = "학습데이터 관리", description = "어드민 홈 > 학습데이터관리 > 전체데이터 API")
@@ -39,9 +45,12 @@ public class DatasetApiController {
})
@GetMapping
public ApiResponseDto<Page<DatasetDto.Basic>> searchDatasets(
@Parameter(description = "구분", example = "DELIVER(납품), PRODUCTION(제작)")
@Parameter(
description = "구분",
example = "",
schema = @Schema(allowableValues = {"DELIVER", "PRODUCTION"}))
@RequestParam(required = false)
String groupTitle,
String dataType,
@Parameter(description = "제목", example = "") @RequestParam(required = false) String title,
@Parameter(description = "페이지 번호 (0부터 시작)", example = "0") @RequestParam(defaultValue = "0")
int page,
@@ -49,7 +58,7 @@ public class DatasetApiController {
int size) {
DatasetDto.SearchReq searchReq = new DatasetDto.SearchReq();
searchReq.setTitle(title);
searchReq.setGroupTitle(groupTitle);
searchReq.setDataType(dataType);
searchReq.setPage(page);
searchReq.setSize(size);
return ApiResponseDto.ok(datasetService.searchDatasets(searchReq));
@@ -73,26 +82,6 @@ public class DatasetApiController {
return ApiResponseDto.ok(datasetService.getDatasetDetail(uuid));
}
@Operation(summary = "학습데이터 등록", description = "학습데이터 파일 업로드")
@ApiResponses(
value = {
@ApiResponse(
responseCode = "201",
description = "등록 성공",
content =
@Content(
mediaType = "application/json",
schema = @Schema(implementation = Long.class))),
@ApiResponse(responseCode = "400", description = "잘못된 요청 데이터", content = @Content),
@ApiResponse(responseCode = "500", description = "서버 오류", content = @Content)
})
@PostMapping("/register")
public ApiResponseDto<Long> registerDataset(
@RequestBody @Valid DatasetDto.RegisterReq registerReq) {
Long id = datasetService.registerDataset(registerReq);
return ApiResponseDto.createOK(id);
}
@Operation(summary = "학습데이터 수정", description = "학습데이터 제목, 메모 수정")
@ApiResponses(
value = {
@@ -129,8 +118,7 @@ public class DatasetApiController {
return ApiResponseDto.ok(uuid);
}
/*
@Operation(summary = "데이터셋 통계 요약", description = "선택 데이터셋의 통계를 요약합니다.")
@Operation(summary = "학습데이터 관리 목록 조회", description = "학습데이터 목록을 조회합니다.")
@ApiResponses(
value = {
@ApiResponse(
@@ -139,16 +127,114 @@ public class DatasetApiController {
content =
@Content(
mediaType = "application/json",
schema = @Schema(implementation = DatasetDto.Summary.class))),
@ApiResponse(responseCode = "400", description = "잘못된 요청", content = @Content),
schema = @Schema(implementation = Page.class))),
@ApiResponse(responseCode = "400", description = "잘못된 검색 조건", content = @Content),
@ApiResponse(responseCode = "500", description = "서버 오류", content = @Content)
})
@PostMapping("/summary")
public ApiResponseDto<DatasetDto.Summary> getDatasetSummary(
@RequestBody @Valid DatasetDto.SummaryReq summaryReq) {
return ApiResponseDto.ok(datasetService.getDatasetSummary(summaryReq));
@GetMapping("/obj-list")
public ApiResponseDto<Page<DatasetObjDto.Basic>> searchDatasetObjectList(
@Parameter(description = "회차 uuid", example = "e9a6774b-4f81-4402-b080-51d27fac1f01")
@RequestParam(required = true)
UUID uuid,
@Parameter(description = "비교년도분류", example = "container") @RequestParam(required = false)
String compareClassCd,
@Parameter(description = "기준년도분류", example = "waste") @RequestParam(required = false)
String targetClassCd,
@Parameter(description = "도엽번호", example = "36713060") @RequestParam(required = false)
String mapSheetNum,
@RequestParam(defaultValue = "0") int page,
@RequestParam(defaultValue = "20") int size) {
DatasetObjDto.SearchReq searchReq = new DatasetObjDto.SearchReq();
searchReq.setUuid(uuid);
searchReq.setCompareClassCd(compareClassCd);
searchReq.setTargetClassCd(targetClassCd);
searchReq.setMapSheetNum(mapSheetNum);
searchReq.setPage(page);
searchReq.setSize(size);
return ApiResponseDto.ok(datasetService.searchDatasetObjectList(searchReq));
}
*/
@Operation(summary = "학습데이터 관리 obj 삭제", description = "학습데이터 관리 obj 삭제 API")
@ApiResponses(
value = {
@ApiResponse(
responseCode = "200",
description = "삭제 성공",
content =
@Content(
mediaType = "application/json",
schema = @Schema(implementation = Page.class))),
@ApiResponse(responseCode = "400", description = "잘못된 검색 조건", content = @Content),
@ApiResponse(responseCode = "500", description = "서버 오류", content = @Content)
})
@DeleteMapping("/obj/{uuid}")
public ApiResponseDto<UUID> deleteDatasetObjByUuid(@PathVariable UUID uuid) {
return ApiResponseDto.ok(datasetService.deleteDatasetObjByUuid(uuid));
}
@Operation(summary = "학습데이터 결과 class 조회", description = "학습데이터 결과 class 조회 API")
@ApiResponses(
value = {
@ApiResponse(
responseCode = "200",
description = "조회 성공",
content =
@Content(
mediaType = "application/json",
schema = @Schema(implementation = Page.class))),
@ApiResponse(responseCode = "400", description = "잘못된 검색 조건", content = @Content),
@ApiResponse(responseCode = "500", description = "서버 오류", content = @Content)
})
@GetMapping("/class/{uuid}")
public ApiResponseDto<List<DatasetClass>> getDatasetObjByUuid(
@Parameter(description = "dataset uuid", example = "e1416f32-769f-495c-a883-3ebfacef4bac")
@PathVariable
UUID uuid,
@Parameter(description = "compare, target", example = "compare") @RequestParam String type) {
return ApiResponseDto.ok(datasetService.getDatasetObjByUuid(uuid, type));
}
@Operation(summary = "남은 저장공간 조회", description = "남은 저장공간 조회 API")
@ApiResponses(
value = {
@ApiResponse(
responseCode = "200",
description = "조회 성공",
content =
@Content(
mediaType = "application/json",
schema = @Schema(implementation = Page.class))),
@ApiResponse(responseCode = "404", description = "저장 공간 조회 오류", content = @Content),
@ApiResponse(responseCode = "500", description = "서버 오류", content = @Content)
})
@GetMapping("/usable-bytes")
public ApiResponseDto<DatasetStorage> getUsableBytes() {
return ApiResponseDto.ok(datasetService.getUsableBytes());
}
@Operation(summary = "학습데이터 zip파일 등록", description = "학습데이터 zip파일 등록 합니다.")
@PostMapping
public ApiResponseDto<ApiResponseDto.ResponseObj> insertDataset(
@RequestBody @Valid DatasetDto.AddReq addReq) {
return ApiResponseDto.ok(datasetService.insertDataset(addReq));
}
@Operation(summary = "객체별 파일 Path 조회", description = "파일 Path 조회")
@GetMapping("/files")
public ResponseEntity<Resource> getFile(@RequestParam UUID uuid, @RequestParam String pathType)
throws Exception {
String path = datasetService.getFilePathByUUIDPathType(uuid, pathType);
return datasetService.getFilePathByFile(path);
}
@Operation(summary = "객체별 파일 Path 조회", description = "파일 Path 조회")
@GetMapping("/files-to86")
public ResponseEntity<Resource> getFileTo86(
@RequestParam UUID uuid, @RequestParam String pathType) throws Exception {
String path = datasetService.getFilePathByUUIDPathType(uuid, pathType);
return datasetService.getFilePathByFile(path);
}
}

View File

@@ -1,56 +0,0 @@
package com.kamco.cd.training.dataset;
import com.kamco.cd.training.config.api.ApiResponseDto;
import com.kamco.cd.training.dataset.dto.MapSheetDto;
import com.kamco.cd.training.dataset.service.MapSheetService;
import io.swagger.v3.oas.annotations.Operation;
import io.swagger.v3.oas.annotations.media.Content;
import io.swagger.v3.oas.annotations.media.Schema;
import io.swagger.v3.oas.annotations.responses.ApiResponse;
import io.swagger.v3.oas.annotations.responses.ApiResponses;
import io.swagger.v3.oas.annotations.tags.Tag;
import jakarta.validation.Valid;
import lombok.RequiredArgsConstructor;
import org.springframework.data.domain.Page;
import org.springframework.web.bind.annotation.*;
@Tag(name = "도엽 관리", description = "도엽(MapSheet) 관리 API")
@RestController
@RequiredArgsConstructor
public class MapSheetApiController {
private final MapSheetService mapSheetService;
@Operation(summary = "도엽 목록 조회", description = "데이터셋의 도엽 목록을 조회합니다.")
@ApiResponses(
value = {
@ApiResponse(
responseCode = "200",
description = "조회 성공",
content =
@Content(
mediaType = "application/json",
schema = @Schema(implementation = Page.class))),
@ApiResponse(responseCode = "400", description = "잘못된 검색 조건", content = @Content),
@ApiResponse(responseCode = "500", description = "서버 오류", content = @Content)
})
@PostMapping("/api/datasets/items/search")
public ApiResponseDto<Page<MapSheetDto.Basic>> searchMapSheets(
@RequestBody @Valid MapSheetDto.SearchReq searchReq) {
return ApiResponseDto.ok(mapSheetService.searchMapSheets(searchReq));
}
@Operation(summary = "도엽 삭제", description = "도엽을 삭제합니다 (다건 지원).")
@ApiResponses(
value = {
@ApiResponse(responseCode = "200", description = "삭제 성공", content = @Content),
@ApiResponse(responseCode = "400", description = "잘못된 요청", content = @Content),
@ApiResponse(responseCode = "404", description = "도엽을 찾을 수 없음", content = @Content),
@ApiResponse(responseCode = "500", description = "서버 오류", content = @Content)
})
@PostMapping("/api/datasets/items/delete")
public ApiResponseDto<Void> deleteMapSheets(@RequestBody @Valid MapSheetDto.DeleteReq deleteReq) {
mapSheetService.deleteMapSheets(deleteReq);
return ApiResponseDto.ok(null);
}
}

View File

@@ -1,7 +1,10 @@
package com.kamco.cd.training.dataset.dto;
import com.fasterxml.jackson.annotation.JsonIgnore;
import com.fasterxml.jackson.annotation.JsonInclude;
import com.kamco.cd.training.common.enums.LearnDataRegister;
import com.kamco.cd.training.common.enums.LearnDataType;
import com.kamco.cd.training.common.enums.ModelType;
import com.kamco.cd.training.common.utils.enums.Enums;
import com.kamco.cd.training.common.utils.interfaces.JsonFormatDttm;
import io.swagger.v3.oas.annotations.media.Schema;
@@ -12,13 +15,16 @@ import java.time.ZonedDateTime;
import java.util.List;
import java.util.UUID;
import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Getter;
import lombok.NoArgsConstructor;
import lombok.Setter;
import lombok.extern.slf4j.Slf4j;
import org.springframework.data.domain.PageRequest;
import org.springframework.data.domain.Pageable;
import org.springframework.data.domain.Sort;
@Slf4j
public class DatasetDto {
@Schema(name = "Dataset Basic", description = "데이터셋 기본 정보")
@@ -30,40 +36,44 @@ public class DatasetDto {
private Long id;
private UUID uuid;
private String groupTitle;
private String groupTitleCd;
private String title;
private Long roundNo;
private Integer compareYyyy;
private Integer targetYyyy;
private String totalSize;
private String memo;
@JsonFormatDttm private ZonedDateTime createdDttm;
private String status;
private String statusCd;
private Boolean deleted;
private String dataType;
public Basic(
Long id,
UUID uuid,
String groupTitle,
String title,
Long roundNo,
Integer compareYyyy,
Integer targetYyyy,
Long totalSize,
String memo,
ZonedDateTime createdDttm,
String status,
Boolean deleted) {
Boolean deleted,
String dataType) {
this.id = id;
this.uuid = uuid;
this.groupTitle = getGroupTitle(groupTitle);
this.groupTitleCd = groupTitle;
this.title = title;
this.roundNo = roundNo;
this.compareYyyy = compareYyyy;
this.targetYyyy = targetYyyy;
this.totalSize = getTotalSize(totalSize);
this.memo = memo;
this.createdDttm = createdDttm;
this.status = getStatus(status);
this.statusCd = status;
this.deleted = deleted;
this.dataType = dataType;
}
public String getTotalSize(Long totalSize) {
@@ -72,13 +82,17 @@ public class DatasetDto {
return String.format("%.2fG", giga);
}
public String getGroupTitle(String groupTitleCd) {
LearnDataType type = Enums.fromId(LearnDataType.class, groupTitleCd);
public String getStatus(String status) {
LearnDataRegister type = Enums.fromId(LearnDataRegister.class, status);
return type == null ? null : type.getText();
}
public String getStatus(String status) {
LearnDataRegister type = Enums.fromId(LearnDataRegister.class, status);
public String getYear() {
return this.compareYyyy + "-" + this.targetYyyy;
}
public String getDataTypeName() {
LearnDataType type = Enums.fromId(LearnDataType.class, this.dataType);
return type == null ? null : type.getText();
}
}
@@ -108,8 +122,8 @@ public class DatasetDto {
@AllArgsConstructor
public static class SearchReq {
@Schema(description = "구분", example = "DELIVER(납품), PRODUCTION(제작)")
private String groupTitle;
@Schema(description = "구분")
private String dataType;
@Schema(description = "제목 (부분 검색)", example = "1차")
private String title;
@@ -121,9 +135,7 @@ public class DatasetDto {
private int size = 20;
public Pageable toPageable() {
// API에서는 1부터 시작하지만 내부적으로는 0부터 시작
int pageIndex = Math.max(0, page - 1);
return PageRequest.of(pageIndex, size, Sort.by(Sort.Direction.DESC, "createdDttm"));
return PageRequest.of(page, size, Sort.by(Sort.Direction.DESC, "createdDttm"));
}
}
@@ -151,10 +163,11 @@ public class DatasetDto {
@Schema(description = "제목", example = "1차 제작")
private String title;
@NotBlank(message = "연도는 필수입니다")
@Size(max = 4, message = "연도는 4자리입니다")
@Schema(description = "연도 (YYYY)", example = "2024")
private String year;
@Schema(description = "비교연도 (YYYY)", example = "2023")
private Integer compareYear;
@Schema(description = "기준연도 (YYYY)", example = "2024")
private Integer targetYyyy;
@Schema(description = "회차", example = "1")
private Long roundNo;
@@ -209,4 +222,132 @@ public class DatasetDto {
@Schema(description = "평균 도엽 수", example = "750")
private double averageMapSheets;
}
@Schema(name = "SelectDataSet", description = "데이터셋 선택 리스트")
@Getter
@Setter
@NoArgsConstructor
@JsonInclude(JsonInclude.Include.NON_NULL)
public static class SelectDataSet {
private String modelNo; // G1, G2, G3 모델 타입
private Long datasetId;
private UUID uuid;
private String dataType;
private String title;
private Long roundNo;
private Integer compareYyyy;
private Integer targetYyyy;
private String memo;
@JsonIgnore private Long classCount;
private Integer buildingCnt;
private Integer containerCnt;
private String dataTypeName;
private Long wasteCnt;
private Long landCoverCnt;
public SelectDataSet(
String modelNo,
Long datasetId,
UUID uuid,
String dataType,
String title,
Long roundNo,
Integer compareYyyy,
Integer targetYyyy,
String memo,
Long classCount) {
this.datasetId = datasetId;
this.uuid = uuid;
this.dataType = dataType;
this.dataTypeName = getDataTypeName(dataType);
this.title = title;
this.roundNo = roundNo;
this.compareYyyy = compareYyyy;
this.targetYyyy = targetYyyy;
this.memo = memo;
this.classCount = classCount;
if (modelNo.equals(ModelType.G2.getId())) {
this.wasteCnt = classCount;
} else if (modelNo.equals(ModelType.G3.getId())) {
this.landCoverCnt = classCount;
}
}
public SelectDataSet(
String modelNo,
Long datasetId,
UUID uuid,
String dataType,
String title,
Long roundNo,
Integer compareYyyy,
Integer targetYyyy,
String memo,
Integer buildingCnt,
Integer containerCnt) {
this.datasetId = datasetId;
this.uuid = uuid;
this.dataType = dataType;
this.dataTypeName = getDataTypeName(dataType);
this.title = title;
this.roundNo = roundNo;
this.compareYyyy = compareYyyy;
this.targetYyyy = targetYyyy;
this.memo = memo;
this.buildingCnt = buildingCnt;
this.containerCnt = containerCnt;
}
public String getDataTypeName(String groupTitleCd) {
LearnDataType type = Enums.fromId(LearnDataType.class, groupTitleCd);
return type == null ? null : type.getText();
}
public String getYear() {
return this.compareYyyy + "-" + this.targetYyyy;
}
}
@Getter
@Setter
@NoArgsConstructor
@AllArgsConstructor
public static class DatasetReq {
String modelNo;
String dataType;
UUID uuid;
Long id;
List<Long> ids;
}
@Getter
@Setter
@NoArgsConstructor
@AllArgsConstructor
public static class AddReq {
private String fileName;
private String filePath;
private Long fileSize;
private String memo;
}
@Getter
@Setter
@NoArgsConstructor
@AllArgsConstructor
@Builder
public static class DatasetMngRegDto {
private String uid;
private String dataType;
private Integer compareYyyy;
private Integer targetYyyy;
private Long roundNo;
private String title;
private String memo;
private Long totalSize;
private Long totalObjectCount;
private String datasetPath;
}
}

View File

@@ -0,0 +1,163 @@
package com.kamco.cd.training.dataset.dto;
import com.fasterxml.jackson.annotation.JsonIgnore;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.kamco.cd.training.common.enums.DetectionClassification;
import com.kamco.cd.training.common.utils.interfaces.JsonFormatDttm;
import io.swagger.v3.oas.annotations.media.Schema;
import java.time.ZonedDateTime;
import java.util.UUID;
import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Getter;
import lombok.NoArgsConstructor;
import lombok.Setter;
import lombok.extern.slf4j.Slf4j;
import org.springframework.data.domain.PageRequest;
import org.springframework.data.domain.Pageable;
import org.springframework.data.domain.Sort;
@Slf4j
public class DatasetObjDto {
@Schema(name = "DatasetObj Basic", description = "데이터셋 객체 Obj 기본 정보")
@Getter
@Setter
@NoArgsConstructor
public static class Basic {
private Long objId;
private Long datasetUid;
private Integer targetYyyy;
private String targetClassCd;
private Integer compareYyyy;
private String compareClassCd;
private String targetPath;
private String comparePath;
private String labelPath;
private String geojsonPath;
private String mapSheetNum;
@JsonFormatDttm private ZonedDateTime createdDttm;
private Long createdUid;
private Boolean deleted;
private UUID uuid;
@JsonIgnore private String geoJsonb;
private JsonNode geoJson;
public Basic(
Long objId,
Long datasetUid,
Integer targetYyyy,
String targetClassCd,
Integer compareYyyy,
String compareClassCd,
String targetPath,
String comparePath,
String labelPath,
String geojsonPath,
String mapSheetNum,
ZonedDateTime createdDttm,
Long createdUid,
Boolean deleted,
UUID uuid,
String geoJsonb) {
this.objId = objId;
this.datasetUid = datasetUid;
this.targetYyyy = targetYyyy;
this.targetClassCd = targetClassCd;
this.compareYyyy = compareYyyy;
this.compareClassCd = compareClassCd;
this.targetPath = targetPath;
this.comparePath = comparePath;
this.labelPath = labelPath;
this.geojsonPath = geojsonPath;
this.mapSheetNum = mapSheetNum;
this.createdDttm = createdDttm;
this.createdUid = createdUid;
this.deleted = deleted;
this.uuid = uuid;
this.geoJsonb = geoJsonb;
JsonNode geoJsonNode = null;
ObjectMapper mapper = new ObjectMapper();
if (geoJsonb != null) {
try {
geoJsonNode = mapper.readTree(geoJsonb);
} catch (JsonProcessingException e) {
throw new RuntimeException(e);
}
}
this.geoJson = geoJsonNode;
}
}
@Schema(name = "DatasetSearchReq", description = "데이터셋 상세 도엽목록 조회 요청")
@Getter
@Setter
@NoArgsConstructor
@AllArgsConstructor
public static class SearchReq {
@Schema(description = "회차 uuid", example = "e9a6774b-4f81-4402-b080-51d27fac1f01")
private UUID uuid;
@Schema(description = "비교년도분류", example = "waste")
private String compareClassCd;
@Schema(description = "기준년도분류", example = "land")
private String targetClassCd;
@Schema(description = "도엽번호", example = "36713060")
private String mapSheetNum;
@Schema(description = "페이지 번호 (0부터 시작)", example = "0")
private int page = 0;
@Schema(description = "페이지 크기", example = "20")
private int size = 20;
public Pageable toPageable() {
return PageRequest.of(page, size, Sort.by(Sort.Direction.DESC, "createdDttm"));
}
}
@Getter
@Setter
@AllArgsConstructor
@NoArgsConstructor
public static class DatasetClass {
private String classCd;
public String getClassName() {
return DetectionClassification.valueOf(classCd.toUpperCase()).getDesc();
}
}
@Getter
@Setter
public static class DatasetStorage {
private String usableBytes;
}
@Getter
@Setter
@NoArgsConstructor
@AllArgsConstructor
@Builder
public static class DatasetObjRegDto {
private Long datasetUid;
private Integer compareYyyy;
private String compareClassCd;
private Integer targetYyyy;
private String targetClassCd;
private String comparePath;
private String targetPath;
private String labelPath;
private String geojsonPath;
private String mapSheetNum;
private JsonNode geojson;
private String fileName;
}
}

View File

@@ -1,22 +1,62 @@
package com.kamco.cd.training.dataset.service;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.node.ArrayNode;
import com.fasterxml.jackson.databind.node.ObjectNode;
import com.kamco.cd.training.common.enums.LearnDataType;
import com.kamco.cd.training.common.exception.CustomApiException;
import com.kamco.cd.training.common.service.FormatStorage;
import com.kamco.cd.training.common.utils.FIleChecker;
import com.kamco.cd.training.config.api.ApiResponseDto.ApiResponseCode;
import com.kamco.cd.training.config.api.ApiResponseDto.ResponseObj;
import com.kamco.cd.training.dataset.dto.DatasetDto;
import com.kamco.cd.training.dataset.dto.DatasetDto.AddReq;
import com.kamco.cd.training.dataset.dto.DatasetDto.DatasetMngRegDto;
import com.kamco.cd.training.dataset.dto.DatasetObjDto;
import com.kamco.cd.training.dataset.dto.DatasetObjDto.DatasetClass;
import com.kamco.cd.training.dataset.dto.DatasetObjDto.DatasetObjRegDto;
import com.kamco.cd.training.dataset.dto.DatasetObjDto.DatasetStorage;
import com.kamco.cd.training.dataset.dto.DatasetObjDto.SearchReq;
import com.kamco.cd.training.postgres.core.DatasetCoreService;
import jakarta.validation.Valid;
import java.io.IOException;
import java.io.InputStream;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.UUID;
import java.util.stream.Stream;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.core.io.InputStreamResource;
import org.springframework.core.io.Resource;
import org.springframework.data.domain.Page;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpStatus;
import org.springframework.http.MediaType;
import org.springframework.http.ResponseEntity;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
@Slf4j
@Service
@RequiredArgsConstructor
@Transactional(readOnly = true)
@Transactional
public class DatasetService {
private final DatasetCoreService datasetCoreService;
@Value("${file.dataset-dir}")
private String datasetDir;
private static final List<String> LABEL_DIRS = List.of("label-json", "label", "input1", "input2");
/**
* 데이터셋 목록 조회
*
@@ -83,4 +123,402 @@ public class DatasetService {
log.info("데이터셋 통계 요약 - 요청: {}", summaryReq);
return datasetCoreService.getDatasetSummary(summaryReq);
}
public Page<DatasetObjDto.Basic> searchDatasetObjectList(SearchReq searchReq) {
return datasetCoreService.searchDatasetObjectList(searchReq);
}
public UUID deleteDatasetObjByUuid(UUID uuid) {
return datasetCoreService.deleteDatasetObjByUuid(uuid);
}
/**
* 데이터셋 object class 조회
*
* @param uuid dataset uuid
* @param type compare, target
*/
public List<DatasetClass> getDatasetObjByUuid(UUID uuid, String type) {
return datasetCoreService.findDatasetObjClassByUuid(uuid, type);
}
/**
* 사용 가능 공간 조회
*
* @return
*/
public DatasetStorage getUsableBytes() {
// 현재 실행 위치가 속한 디스크 기준
try {
FormatStorage.DiskUsage usage = FormatStorage.getDiskUsage(Path.of("."));
log.debug("경로 : {}", usage.path());
log.debug("총 저장공간 : {}", usage.totalText());
log.debug("남은 저장공간 : {}", usage.usableText());
log.debug("사용률 : {}", usage.usedPercent());
DatasetStorage datasetStorage = new DatasetStorage();
datasetStorage.setUsableBytes(usage.usableText());
return datasetStorage;
} catch (Exception e) {
throw new CustomApiException("NOT_FOUND", HttpStatus.NOT_FOUND);
}
}
@Deprecated
@Transactional
public ResponseObj insertDatasetTo86(@Valid AddReq addReq) {
Long datasetUid = null; // master id 값, 등록하면서 가져올 예정
// 압축 해제
FIleChecker.unzipOn86Server(
addReq.getFilePath() + addReq.getFileName(),
addReq.getFilePath() + addReq.getFileName().replace(".zip", ""));
// 해제한 폴더 읽어서 데이터 저장
List<Map<String, Object>> list =
getUnzipDatasetFilesTo86(
addReq.getFilePath() + addReq.getFileName().replace(".zip", ""), "train");
int idx = 0;
for (Map<String, Object> map : list) {
datasetUid =
this.insertTrainTestData(map, addReq, idx, datasetUid, "train"); // train 데이터 insert
idx++;
}
List<Map<String, Object>> testList =
getUnzipDatasetFilesTo86(
addReq.getFilePath() + addReq.getFileName().replace(".zip", ""), "test");
int testIdx = 0;
for (Map<String, Object> test : testList) {
datasetUid =
this.insertTrainTestData(test, addReq, testIdx, datasetUid, "test"); // test 데이터 insert
testIdx++;
}
datasetCoreService.updateDatasetUploadStatus(datasetUid);
return new ResponseObj(ApiResponseCode.OK, "업로드 성공하였습니다.");
}
@Transactional
public ResponseObj insertDataset(@Valid AddReq addReq) {
Long datasetUid = null; // master id 값, 등록하면서 가져올 예정
try {
// 압축 해제
FIleChecker.unzip(addReq.getFileName(), addReq.getFilePath());
// 해제한 폴더 읽어서 데이터 저장
List<Map<String, Object>> list =
getUnzipDatasetFiles(
addReq.getFilePath() + addReq.getFileName().replace(".zip", ""), "train");
int idx = 0;
for (Map<String, Object> map : list) {
datasetUid =
this.insertTrainTestData(map, addReq, idx, datasetUid, "train"); // train 데이터 insert
idx++;
}
List<Map<String, Object>> valList =
getUnzipDatasetFiles(
addReq.getFilePath() + addReq.getFileName().replace(".zip", ""), "val");
int valIdx = 0;
for (Map<String, Object> valid : valList) {
datasetUid =
this.insertTrainTestData(valid, addReq, valIdx, datasetUid, "val"); // val 데이터 insert
valIdx++;
}
List<Map<String, Object>> testList =
getUnzipDatasetFiles(
addReq.getFilePath() + addReq.getFileName().replace(".zip", ""), "test");
int testIdx = 0;
for (Map<String, Object> test : testList) {
datasetUid =
this.insertTrainTestData(test, addReq, testIdx, datasetUid, "test"); // test 데이터 insert
testIdx++;
}
} catch (IOException e) {
log.error(e.getMessage());
return new ResponseObj(ApiResponseCode.INTERNAL_SERVER_ERROR, e.getMessage());
}
datasetCoreService.updateDatasetUploadStatus(datasetUid);
return new ResponseObj(ApiResponseCode.OK, "업로드 성공하였습니다.");
}
@Transactional
public Long insertTrainTestData(
Map<String, Object> map, AddReq addReq, int idx, Long datasetUid, String subDir) {
ObjectMapper mapper = new ObjectMapper();
String comparePath = (String) map.get("input1");
String targetPath = (String) map.get("input2");
String labelPath = (String) map.get("label");
String geojsonPath = (String) map.get("geojson_path");
Object labelJson = map.get("label-json");
JsonNode json;
if (labelJson instanceof JsonNode jn) {
json = jn;
} else {
try {
json = mapper.readTree(labelJson.toString());
} catch (IOException e) {
throw new RuntimeException("label_json parse error", e);
}
}
String fileName = Paths.get(comparePath).getFileName().toString();
String[] fileNameStr = fileName.split("_");
String compareYyyy = fileNameStr[1];
String targetYyyy = fileNameStr[2];
String mapSheetNum = fileNameStr[3];
if (idx == 0 && subDir.equals("train")) {
String title = compareYyyy + "-" + targetYyyy; // 제목 : 비교년도-기준년도
String dataType = LearnDataType.PRODUCTION.getId(); // 만들어 넣는 건 다 제작
Long stage =
datasetCoreService.getDatasetMaxStage(
Integer.parseInt(compareYyyy), Integer.parseInt(targetYyyy))
+ 1;
String uid = addReq.getFileName().replace(".zip", "");
DatasetMngRegDto mngRegDto =
DatasetMngRegDto.builder()
.uid(uid)
.dataType(dataType)
.compareYyyy(Integer.parseInt(compareYyyy))
.targetYyyy(Integer.parseInt(targetYyyy))
.title(title)
.memo(addReq.getMemo())
.roundNo(stage)
.totalSize(addReq.getFileSize())
.datasetPath(addReq.getFilePath())
.build();
datasetUid = datasetCoreService.insertDatasetMngData(mngRegDto); // tb_dataset 에 insert
}
// datasetUid 로 obj 도 등록하기
// Json 갯수만큼 for문 돌려서 insert 해야 함, features에 빈값
if (json != null && json.path("features") != null && !json.path("features").isEmpty()) {
for (JsonNode feature : json.path("features")) {
JsonNode prop = feature.path("properties");
String compareClassCd = prop.path("before").asText(null);
String targetClassCd = prop.path("after").asText(null);
// 한 개씩 자른 geojson을 FeatureCollection 으로 만들어서 넣기
ObjectNode root = mapper.createObjectNode();
root.put("type", "FeatureCollection");
ArrayNode features = mapper.createArrayNode();
features.add(feature);
root.set("features", features);
DatasetObjRegDto objRegDto =
DatasetObjRegDto.builder()
.datasetUid(datasetUid)
.compareYyyy(Integer.parseInt(compareYyyy))
.compareClassCd(compareClassCd)
.targetYyyy(Integer.parseInt(targetYyyy))
.targetClassCd(targetClassCd)
.comparePath(comparePath)
.targetPath(targetPath)
.labelPath(labelPath)
.mapSheetNum(mapSheetNum)
.geojson(root)
.geojsonPath(geojsonPath)
.fileName(fileName)
.build();
if (subDir.equals("train")) {
datasetCoreService.insertDatasetObj(objRegDto);
} else if (subDir.equals("val")) {
datasetCoreService.insertDatasetValObj(objRegDto);
} else {
datasetCoreService.insertDatasetTestObj(objRegDto);
}
}
}
return datasetUid;
}
private List<Map<String, Object>> getUnzipDatasetFiles(String unzipRootPath, String subDir) {
Path root = Paths.get(unzipRootPath).resolve(subDir);
Map<String, Map<String, Object>> grouped = new HashMap<>();
for (String dirName : LABEL_DIRS) {
Path dir = root.resolve(dirName);
if (!Files.isDirectory(dir)) {
throw new IllegalStateException("폴더가 존재하지 않습니다 : " + dir);
}
try (Stream<Path> stream = Files.list(dir)) {
stream
.filter(Files::isRegularFile)
.forEach(
path -> {
String fileName = path.getFileName().toString();
String baseName = getBaseName(fileName);
// baseName 기준 Map 생성
Map<String, Object> data =
grouped.computeIfAbsent(baseName, k -> new HashMap<>());
// 공통 메타
data.put("baseName", baseName);
// 폴더별 처리
if ("label-json".equals(dirName)) {
// json 파일이면 파싱
data.put("label-json", readJson(path));
data.put("geojson_path", path.toAbsolutePath().toString());
} else {
data.put(dirName, path.toAbsolutePath().toString());
}
});
} catch (IOException e) {
throw new RuntimeException(e);
}
}
return new ArrayList<>(grouped.values());
}
private String getBaseName(String fileName) {
int idx = fileName.lastIndexOf('.');
return (idx > 0) ? fileName.substring(0, idx) : fileName;
}
private JsonNode readJson(Path jsonPath) {
try {
ObjectMapper mapper = new ObjectMapper();
return mapper.readTree(jsonPath.toFile());
} catch (IOException e) {
throw new RuntimeException("JSON 읽기 실패: " + jsonPath, e);
}
}
public String getFilePathByUUIDPathType(UUID uuid, String pathType) {
return datasetCoreService.getFilePathByUUIDPathType(uuid, pathType);
}
@Deprecated
private List<Map<String, Object>> getUnzipDatasetFilesTo86(String unzipRootPath, String subDir) {
// String root = Paths.get(unzipRootPath)
// .resolve(subDir)
// .toString();
//
String root = normalizeLinuxPath(unzipRootPath + "/" + subDir);
Map<String, Map<String, Object>> grouped = new HashMap<>();
for (String dirName : LABEL_DIRS) {
String remoteDir = root + "/" + dirName;
// 1. 86 서버에서 해당 디렉토리의 파일 목록 조회
List<String> files = listFilesOn86Server(remoteDir);
if (files.isEmpty()) {
throw new IllegalStateException("폴더가 존재하지 않거나 파일이 없습니다 : " + remoteDir);
}
for (String fullPath : files) {
String fileName = Paths.get(fullPath).getFileName().toString();
String baseName = getBaseName(fileName);
Map<String, Object> data = grouped.computeIfAbsent(baseName, k -> new HashMap<>());
data.put("baseName", baseName);
if ("label-json".equals(dirName)) {
// 2. json 내용도 86 서버에서 읽어서 가져와야 함
String json = readRemoteFileAsString(fullPath);
data.put("label-json", parseJson(json));
data.put("geojson_path", fullPath);
} else {
data.put(dirName, fullPath);
}
}
}
return new ArrayList<>(grouped.values());
}
private List<String> listFilesOn86Server(String remoteDir) {
String command = "find " + escape(remoteDir) + " -maxdepth 1 -type f";
return FIleChecker.execCommandAndReadLines(command);
}
private String readRemoteFileAsString(String remoteFilePath) {
String command = "cat " + escape(remoteFilePath);
List<String> lines = FIleChecker.execCommandAndReadLines(command);
return String.join("\n", lines);
}
private JsonNode parseJson(String json) {
try {
ObjectMapper mapper = new ObjectMapper();
return mapper.readTree(json);
} catch (IOException e) {
throw new RuntimeException("JSON 파싱 실패", e);
}
}
private String escape(String path) {
return "'" + path.replace("'", "'\"'\"'") + "'";
}
private static String normalizeLinuxPath(String path) {
return path.replace("\\", "/");
}
public ResponseEntity<Resource> getFilePathByFile(String remoteFilePath) {
try {
Path path = Paths.get(remoteFilePath);
InputStream inputStream = Files.newInputStream(path);
InputStreamResource resource =
new InputStreamResource(inputStream) {
@Override
public long contentLength() {
return -1; // 알 수 없으면 -1
}
};
String fileName = Paths.get(remoteFilePath.replace("\\", "/")).getFileName().toString();
return ResponseEntity.ok()
.header(HttpHeaders.CONTENT_DISPOSITION, "attachment; filename=\"" + fileName + "\"")
.contentType(MediaType.APPLICATION_OCTET_STREAM)
.body(resource);
} catch (Exception e) {
throw new RuntimeException(e);
}
}
}

View File

@@ -1,10 +1,11 @@
package com.kamco.cd.training.hyperparam;
import com.kamco.cd.training.common.dto.HyperParam;
import com.kamco.cd.training.common.enums.ModelType;
import com.kamco.cd.training.config.api.ApiResponseDto;
import com.kamco.cd.training.hyperparam.dto.HyperParamDto;
import com.kamco.cd.training.hyperparam.dto.HyperParamDto.List;
import com.kamco.cd.training.hyperparam.service.HyperParamService;
import com.kamco.cd.training.model.dto.ModelMngDto;
import io.swagger.v3.oas.annotations.Operation;
import io.swagger.v3.oas.annotations.Parameter;
import io.swagger.v3.oas.annotations.media.Content;
@@ -49,8 +50,7 @@ public class HyperParamApiController {
@ApiResponse(responseCode = "500", description = "서버 오류", content = @Content)
})
@PostMapping
public ApiResponseDto<String> createHyperParam(
@Valid @RequestBody HyperParamDto.HyperParamCreateReq createReq) {
public ApiResponseDto<String> createHyperParam(@Valid @RequestBody HyperParam createReq) {
String newVersion = hyperParamService.createHyperParam(createReq);
return ApiResponseDto.ok(newVersion);
}
@@ -66,11 +66,12 @@ public class HyperParamApiController {
mediaType = "application/json",
schema = @Schema(implementation = String.class))),
@ApiResponse(responseCode = "400", description = "잘못된 요청", content = @Content),
@ApiResponse(responseCode = "422", description = "default는 삭제불가", content = @Content),
@ApiResponse(responseCode = "500", description = "서버 오류", content = @Content)
})
@PutMapping("/{uuid}")
public ApiResponseDto<String> updateHyperParam(
@PathVariable UUID uuid, @Valid @RequestBody HyperParamDto.HyperParamCreateReq createReq) {
@PathVariable UUID uuid, @Valid @RequestBody HyperParam createReq) {
return ApiResponseDto.ok(hyperParamService.updateHyperParam(uuid, createReq));
}
@@ -87,8 +88,9 @@ public class HyperParamApiController {
@ApiResponse(responseCode = "404", description = "하이퍼파라미터를 찾을 수 없음", content = @Content),
@ApiResponse(responseCode = "500", description = "서버 오류", content = @Content)
})
@GetMapping("/list")
@GetMapping("{model}/list")
public ApiResponseDto<Page<List>> getHyperParam(
@PathVariable ModelType model,
@Parameter(
description = "구분 CREATE_DATE(생성일), LAST_USED_DATE(최근사용일)",
example = "CREATE_DATE")
@@ -98,7 +100,7 @@ public class HyperParamApiController {
LocalDate startDate,
@Parameter(description = "종료일", example = "2026-02-28") @RequestParam(required = false)
LocalDate endDate,
@Parameter(description = "버전명", example = "HPs_0001") @RequestParam(required = false)
@Parameter(description = "버전명", example = "G_000001") @RequestParam(required = false)
String hyperVer,
@Parameter(
description = "정렬",
@@ -124,7 +126,7 @@ public class HyperParamApiController {
searchReq.setSort(sort);
searchReq.setPage(page);
searchReq.setSize(size);
Page<List> list = hyperParamService.getHyperParamList(searchReq);
Page<List> list = hyperParamService.getHyperParamList(model, searchReq);
return ApiResponseDto.ok(list);
}
@@ -133,12 +135,12 @@ public class HyperParamApiController {
@ApiResponses(
value = {
@ApiResponse(responseCode = "200", description = "삭제 성공", content = @Content),
@ApiResponse(responseCode = "409", description = "HPs_0001 삭제 불가", content = @Content),
@ApiResponse(responseCode = "422", description = "default 삭제 불가", content = @Content),
@ApiResponse(responseCode = "404", description = "하이퍼파라미터를 찾을 수 없음", content = @Content),
})
@DeleteMapping("/{uuid}")
public ApiResponseDto<Void> deleteHyperParam(
@Parameter(description = "하이퍼파라미터 uuid", example = "7966dd64-004a-4596-89ef-001664bc4de2")
@Parameter(description = "하이퍼파라미터 uuid", example = "c3b5a285-8f68-42af-84f0-e6d09162deb5")
@PathVariable
UUID uuid) {
hyperParamService.deleteHyperParam(uuid);
@@ -154,13 +156,13 @@ public class HyperParamApiController {
content =
@Content(
mediaType = "application/json",
schema = @Schema(implementation = ModelMngDto.HyperParamInfo.class))),
schema = @Schema(implementation = HyperParamDto.Basic.class))),
@ApiResponse(responseCode = "404", description = "하이퍼파라미터를 찾을 수 없음", content = @Content),
@ApiResponse(responseCode = "500", description = "서버 오류", content = @Content)
})
@GetMapping("/{uuid}")
public ApiResponseDto<HyperParamDto.Basic> getHyperParam(
@Parameter(description = "하이퍼파라미터 uuid", example = "9c91a20c-71e7-4e5f-a860-9626d2b2059c")
@Parameter(description = "하이퍼파라미터 uuid", example = "c3b5a285-8f68-42af-84f0-e6d09162deb5")
@PathVariable
UUID uuid) {
return ApiResponseDto.ok(hyperParamService.getHyperParam(uuid));
@@ -179,8 +181,11 @@ public class HyperParamApiController {
@ApiResponse(responseCode = "404", description = "하이퍼파라미터를 찾을 수 없음", content = @Content),
@ApiResponse(responseCode = "500", description = "서버 오류", content = @Content)
})
@GetMapping("/init")
public ApiResponseDto<HyperParamDto.Basic> getInitHyperParam() {
return ApiResponseDto.ok(hyperParamService.getInitHyperParam());
@GetMapping("/init/{model}")
public ApiResponseDto<HyperParamDto.Basic> getInitHyperParam(
@PathVariable ModelType model
) {
return ApiResponseDto.ok(hyperParamService.getInitHyperParam(model));
}
}

View File

@@ -1,5 +1,6 @@
package com.kamco.cd.training.hyperparam.dto;
import com.kamco.cd.training.common.enums.ModelType;
import com.kamco.cd.training.common.utils.enums.CodeExpose;
import com.kamco.cd.training.common.utils.enums.EnumType;
import com.kamco.cd.training.common.utils.interfaces.JsonFormatDttm;
@@ -24,7 +25,10 @@ public class HyperParamDto {
@AllArgsConstructor
public static class Basic {
private ModelType model; // 20250212 modeltype추가
private UUID uuid;
private String hyperVer;
@JsonFormatDttm private ZonedDateTime createdDttm;
// -------------------------
// Important
@@ -96,6 +100,8 @@ public class HyperParamDto {
private Integer gpuCnt;
private String gpuIds;
private Integer masterPort;
private Boolean isDefault;
}
@Getter
@@ -113,155 +119,6 @@ public class HyperParamDto {
private Long totalCnt;
}
@Schema(name = "HyperParamCreateReq", description = "하이퍼파라미터 등록 요청")
@Getter
@Setter
@NoArgsConstructor
@AllArgsConstructor
public static class HyperParamCreateReq {
// -------------------------
// Important
// -------------------------
@Schema(description = "백본 네트워크", example = "large")
private String backbone; // backbone
@Schema(description = "입력 이미지 크기(H,W)", example = "512,512")
private String inputSize; // input_size
@Schema(description = "크롭 크기(H,W 또는 단일값)", example = "256,256")
private String cropSize; // crop_size
@Schema(description = "배치 크기(Per GPU)", example = "16")
private Integer batchSize; // batch_size
// -------------------------
// Data
// -------------------------
@Schema(description = "Train dataloader workers", example = "16")
private Integer trainNumWorkers; // train_num_workers
@Schema(description = "Val dataloader workers", example = "8")
private Integer valNumWorkers; // val_num_workers
@Schema(description = "Test dataloader workers", example = "8")
private Integer testNumWorkers; // test_num_workers
@Schema(description = "Train shuffle 여부", example = "true")
private Boolean trainShuffle; // train_shuffle
@Schema(description = "Train persistent workers 여부", example = "true")
private Boolean trainPersistent; // train_persistent
@Schema(description = "Val persistent workers 여부", example = "true")
private Boolean valPersistent; // val_persistent
// -------------------------
// Model Architecture
// -------------------------
@Schema(description = "Drop Path 비율", example = "0.3")
private Double dropPathRate; // drop_path_rate
@Schema(description = "Freeze 단계(-1:None)", example = "-1")
private Integer frozenStages; // frozen_stages
@Schema(description = "Neck 결합 정책", example = "abs_diff")
private String neckPolicy; // neck_policy
@Schema(description = "디코더 채널 구성", example = "512,256,128,64")
private String decoderChannels; // decoder_channels
@Schema(description = "클래스별 가중치", example = "1,10")
private String classWeight; // class_weight
// -------------------------
// Loss & Optimization
// -------------------------
@Schema(description = "학습률", example = "0.00006")
private Double learningRate; // learning_rate
@Schema(description = "Weight Decay", example = "0.05")
private Double weightDecay; // weight_decay
@Schema(description = "Layer Decay Rate", example = "0.9")
private Double layerDecayRate; // layer_decay_rate
@Schema(description = "DDP unused params 탐색 여부", example = "true")
private Boolean ddpFindUnusedParams; // ddp_find_unused_params
@Schema(description = "Loss 계산 제외 인덱스", example = "255")
private Integer ignoreIndex; // ignore_index
@Schema(description = "레이어 깊이", example = "24")
private Integer numLayers; // num_layers
// -------------------------
// Evaluation
// -------------------------
@Schema(description = "평가 지표 목록", example = "mFscore,mIoU")
private String metrics; // metrics
@Schema(description = "Best 모델 선정 기준 지표", example = "changed_fscore")
private String saveBest; // save_best
@Schema(description = "Best 모델 선정 규칙", example = "less")
private String saveBestRule; // save_best_rule
@Schema(description = "검증 수행 주기(Epoch)", example = "10")
private Integer valInterval; // val_interval
@Schema(description = "로그 기록 주기(Iteration)", example = "400")
private Integer logInterval; // log_interval
@Schema(description = "시각화 저장 주기(Epoch)", example = "1")
private Integer visInterval; // vis_interval
// -------------------------
// Augmentation
// -------------------------
@Schema(description = "회전 적용 확률", example = "0.5")
private Double rotProb; // rot_prob
@Schema(description = "회전 각도 범위(Min,Max)", example = "-20,20")
private String rotDegree; // rot_degree
@Schema(description = "반전 적용 확률", example = "0.5")
private Double flipProb; // flip_prob
@Schema(description = "채널 교환 확률", example = "0.5")
private Double exchangeProb; // exchange_prob
@Schema(description = "밝기 변화량", example = "10")
private Integer brightnessDelta; // brightness_delta
@Schema(description = "대비 범위(Min,Max)", example = "0.8,1.2")
private String contrastRange; // contrast_range
@Schema(description = "채도 범위(Min,Max)", example = "0.8,1.2")
private String saturationRange; // saturation_range
@Schema(description = "색조 변화량", example = "10")
private Integer hueDelta; // hue_delta
// -------------------------
// Hardware
// -------------------------
@Schema(description = "사용 GPU 개수", example = "4")
private Integer gpuCnt; // gpu_cnt
@Schema(description = "사용 GPU ID 목록", example = "0,1,2,3")
private String gpuIds; // gpu_ids
@Schema(description = "분산학습 마스터 포트", example = "1122")
private Integer masterPort; // master_port
// -------------------------
// Memo
// -------------------------
@Schema(description = "메모", example = "하이퍼파라미터 신규등록")
private String memo; // memo
}
@Getter
@Setter
@NoArgsConstructor

View File

@@ -1,7 +1,10 @@
package com.kamco.cd.training.hyperparam.service;
import com.kamco.cd.training.common.dto.HyperParam;
import com.kamco.cd.training.common.enums.ModelType;
import com.kamco.cd.training.hyperparam.dto.HyperParamDto;
import com.kamco.cd.training.hyperparam.dto.HyperParamDto.List;
import com.kamco.cd.training.hyperparam.dto.HyperParamDto.SearchReq;
import com.kamco.cd.training.postgres.core.HyperParamCoreService;
import java.util.UUID;
import lombok.RequiredArgsConstructor;
@@ -19,11 +22,12 @@ public class HyperParamService {
/**
* 하이퍼 파라미터 목록 조회
*
* @param model
* @param req
* @return 목록
*/
public Page<List> getHyperParamList(HyperParamDto.SearchReq req) {
return hyperParamCoreService.findByHyperVerList(req);
public Page<List> getHyperParamList(ModelType model, SearchReq req) {
return hyperParamCoreService.findByHyperVerList(model, req);
}
/**
@@ -33,8 +37,8 @@ public class HyperParamService {
* @return 생성된 버전명
*/
@Transactional
public String createHyperParam(HyperParamDto.HyperParamCreateReq createReq) {
return hyperParamCoreService.createHyperParam(createReq);
public String createHyperParam(HyperParam createReq) {
return hyperParamCoreService.createHyperParam(createReq).getHyperVer();
}
/**
@@ -44,7 +48,7 @@ public class HyperParamService {
* @return
*/
@Transactional
public String updateHyperParam(UUID uuid, HyperParamDto.HyperParamCreateReq createReq) {
public String updateHyperParam(UUID uuid, HyperParam createReq) {
return hyperParamCoreService.updateHyperParam(uuid, createReq);
}
@@ -58,8 +62,8 @@ public class HyperParamService {
}
/** 하이퍼파라미터 최적화 설정값 조회 */
public HyperParamDto.Basic getInitHyperParam() {
return hyperParamCoreService.getInitHyperParam();
public HyperParamDto.Basic getInitHyperParam(ModelType model) {
return hyperParamCoreService.getInitHyperParam(model);
}
/**

View File

@@ -1,231 +0,0 @@
package com.kamco.cd.training.model;
import com.kamco.cd.training.config.api.ApiResponseDto;
import com.kamco.cd.training.model.dto.ModelMngDto;
import com.kamco.cd.training.model.dto.ModelMngDto.Basic;
import com.kamco.cd.training.model.service.ModelMngService;
import com.kamco.cd.training.model.service.ModelTrainService;
import io.swagger.v3.oas.annotations.Operation;
import io.swagger.v3.oas.annotations.Parameter;
import io.swagger.v3.oas.annotations.media.Content;
import io.swagger.v3.oas.annotations.media.Schema;
import io.swagger.v3.oas.annotations.responses.ApiResponse;
import io.swagger.v3.oas.annotations.responses.ApiResponses;
import io.swagger.v3.oas.annotations.tags.Tag;
import jakarta.validation.Valid;
import java.util.List;
import lombok.RequiredArgsConstructor;
import org.springframework.data.domain.Page;
import org.springframework.web.bind.annotation.DeleteMapping;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
@RestController
@RequiredArgsConstructor
@Tag(name = "모델관리", description = "어드민 홈 > 모델학습관리 > 모델관리 > 목록")
@RequestMapping("/api/models")
public class ModelMngApiController {
private final ModelMngService modelMngService;
private final ModelTrainService modelTrainService;
@Operation(summary = "학습 모델 목록 조회", description = "학습 모델 목록을 조회합니다")
@ApiResponses(
value = {
@ApiResponse(
responseCode = "200",
description = "검색 성공",
content =
@Content(
mediaType = "application/json",
schema = @Schema(implementation = Page.class))),
@ApiResponse(responseCode = "400", description = "잘못된 검색 조건", content = @Content),
@ApiResponse(responseCode = "500", description = "서버 오류", content = @Content)
})
@GetMapping
public ApiResponseDto<Page<Basic>> findByModels(
@Parameter(description = "상태 코드") @RequestParam(required = false) String status,
@Parameter(description = "페이지 번호") @RequestParam(defaultValue = "0") int page,
@Parameter(description = "페이지 크기") @RequestParam(defaultValue = "20") int size) {
ModelMngDto.SearchReq searchReq = new ModelMngDto.SearchReq(status, page, size);
return ApiResponseDto.ok(modelMngService.findByModels(searchReq));
}
@Operation(summary = "학습 모델 상세 조회", description = "학습 모델의 상세 정보를 UUID로 조회합니다")
@ApiResponses(
value = {
@ApiResponse(
responseCode = "200",
description = "조회 성공",
content =
@Content(
mediaType = "application/json",
schema = @Schema(implementation = ModelMngDto.Detail.class))),
@ApiResponse(responseCode = "404", description = "모델을 찾을 수 없음", content = @Content),
@ApiResponse(responseCode = "500", description = "서버 오류", content = @Content)
})
@GetMapping("/{uuid}")
public ApiResponseDto<ModelMngDto.Detail> getModelDetail(
@Parameter(description = "모델 UUID", example = "b7e99739-6736-45f9-a224-8161ecddf287")
@PathVariable
String uuid) {
return ApiResponseDto.ok(modelMngService.getModelDetailByUuid(uuid));
}
// ==================== 학습 모델학습관리 API (5종) ====================
@Operation(summary = "학습 모델 통합 조회", description = "학습 관리 화면에서 학습 이력 리스트와 현재 상태를 조회합니다")
@ApiResponses(
value = {
@ApiResponse(
responseCode = "200",
description = "조회 성공",
content =
@Content(
mediaType = "application/json",
schema = @Schema(implementation = List.class))),
@ApiResponse(responseCode = "500", description = "서버 오류", content = @Content)
})
@GetMapping("/train")
public ApiResponseDto<List<ModelMngDto.TrainListRes>> getTrainModelList() {
return ApiResponseDto.ok(modelTrainService.getTrainModelList());
}
@Operation(summary = "학습 설정 통합 조회", description = "학습 실행 팝업 구성에 필요한 모든 데이터를 한 번에 반환합니다")
@ApiResponses(
value = {
@ApiResponse(
responseCode = "200",
description = "조회 성공",
content =
@Content(
mediaType = "application/json",
schema = @Schema(implementation = ModelMngDto.FormConfigRes.class))),
@ApiResponse(responseCode = "500", description = "서버 오류", content = @Content)
})
@GetMapping("/train/form-config")
public ApiResponseDto<ModelMngDto.FormConfigRes> getFormConfig() {
return ApiResponseDto.ok(modelTrainService.getFormConfig());
}
@Operation(summary = "학습 시작", description = "모든 설정(Step 1~3)을 마치고 최종적으로 학습 프로세스를 시작합니다")
@ApiResponses(
value = {
@ApiResponse(
responseCode = "200",
description = "학습 시작 성공",
content =
@Content(
mediaType = "application/json",
schema = @Schema(implementation = ModelMngDto.TrainStartRes.class))),
@ApiResponse(responseCode = "400", description = "잘못된 요청", content = @Content),
@ApiResponse(responseCode = "500", description = "서버 오류", content = @Content)
})
@PostMapping("/train")
public ApiResponseDto<ModelMngDto.TrainStartRes> startTraining(
@Valid @RequestBody ModelMngDto.TrainStartReq trainReq) {
return ApiResponseDto.ok(modelTrainService.startTraining(trainReq));
}
@Operation(summary = "학습 모델 삭제", description = "목록에서 특정 학습 모델을 삭제합니다")
@ApiResponses(
value = {
@ApiResponse(responseCode = "200", description = "삭제 성공", content = @Content),
@ApiResponse(responseCode = "400", description = "진행 중인 모델은 삭제 불가", content = @Content),
@ApiResponse(responseCode = "404", description = "모델을 찾을 수 없음", content = @Content),
@ApiResponse(responseCode = "500", description = "서버 오류", content = @Content)
})
@DeleteMapping("/train/{uuid}")
public ApiResponseDto<Void> deleteTrainModel(
@Parameter(description = "모델 UUID") @PathVariable String uuid) {
modelTrainService.deleteTrainModel(uuid);
return ApiResponseDto.ok(null);
}
// ==================== Resume Training (학습 재시작) ====================
@Operation(summary = "학습 재시작 정보 조회", description = "중단된 학습의 재시작 가능 여부와 Checkpoint 정보를 조회합니다")
@ApiResponses(
value = {
@ApiResponse(
responseCode = "200",
description = "조회 성공",
content =
@Content(
mediaType = "application/json",
schema = @Schema(implementation = ModelMngDto.ResumeInfo.class))),
@ApiResponse(responseCode = "404", description = "모델을 찾을 수 없음", content = @Content),
@ApiResponse(responseCode = "500", description = "서버 오류", content = @Content)
})
@GetMapping("/train/{uuid}/resume-info")
public ApiResponseDto<ModelMngDto.ResumeInfo> getResumeInfo(
@Parameter(description = "모델 UUID") @PathVariable String uuid) {
return ApiResponseDto.ok(modelTrainService.getResumeInfo(uuid));
}
@Operation(summary = "학습 재시작", description = "중단된 지점(Checkpoint)부터 학습을 재개합니다")
@ApiResponses(
value = {
@ApiResponse(
responseCode = "200",
description = "재시작 성공",
content =
@Content(
mediaType = "application/json",
schema = @Schema(implementation = ModelMngDto.ResumeResponse.class))),
@ApiResponse(responseCode = "400", description = "재시작 불가능한 상태", content = @Content),
@ApiResponse(responseCode = "404", description = "모델을 찾을 수 없음", content = @Content),
@ApiResponse(responseCode = "500", description = "서버 오류", content = @Content)
})
@PostMapping("/train/{uuid}/resume")
public ApiResponseDto<ModelMngDto.ResumeResponse> resumeTraining(
@Parameter(description = "모델 UUID") @PathVariable String uuid,
@Valid @RequestBody ModelMngDto.ResumeRequest resumeReq) {
return ApiResponseDto.ok(modelTrainService.resumeTraining(uuid, resumeReq));
}
// ==================== Best Epoch Setting (Best Epoch 설정) ====================
@Operation(summary = "Best Epoch 설정", description = "사용자가 직접 Best Epoch를 선택하여 설정합니다")
@ApiResponses(
value = {
@ApiResponse(
responseCode = "200",
description = "설정 성공",
content =
@Content(
mediaType = "application/json",
schema = @Schema(implementation = ModelMngDto.BestEpochResponse.class))),
@ApiResponse(responseCode = "404", description = "모델을 찾을 수 없음", content = @Content),
@ApiResponse(responseCode = "500", description = "서버 오류", content = @Content)
})
@PostMapping("/train/{uuid}/best-epoch")
public ApiResponseDto<ModelMngDto.BestEpochResponse> setBestEpoch(
@Parameter(description = "모델 UUID") @PathVariable String uuid,
@Valid @RequestBody ModelMngDto.BestEpochRequest bestEpochReq) {
return ApiResponseDto.ok(modelTrainService.setBestEpoch(uuid, bestEpochReq));
}
@Operation(summary = "Epoch별 성능 지표 조회", description = "학습된 모델의 Epoch별 성능 지표를 조회합니다")
@ApiResponses(
value = {
@ApiResponse(
responseCode = "200",
description = "조회 성공",
content =
@Content(
mediaType = "application/json",
schema = @Schema(implementation = List.class))),
@ApiResponse(responseCode = "404", description = "모델을 찾을 수 없음", content = @Content),
@ApiResponse(responseCode = "500", description = "서버 오류", content = @Content)
})
@GetMapping("/train/{uuid}/epoch-metrics")
public ApiResponseDto<List<ModelMngDto.EpochMetric>> getEpochMetrics(
@Parameter(description = "모델 UUID") @PathVariable String uuid) {
return ApiResponseDto.ok(modelTrainService.getEpochMetrics(uuid));
}
}

View File

@@ -0,0 +1,225 @@
package com.kamco.cd.training.model;
import com.kamco.cd.training.config.api.ApiResponseDto;
import com.kamco.cd.training.model.dto.ModelTrainDetailDto;
import com.kamco.cd.training.model.dto.ModelTrainDetailDto.MappingDataset;
import com.kamco.cd.training.model.dto.ModelTrainDetailDto.ModelBestEpoch;
import com.kamco.cd.training.model.dto.ModelTrainDetailDto.ModelTestMetrics;
import com.kamco.cd.training.model.dto.ModelTrainDetailDto.ModelTrainMetrics;
import com.kamco.cd.training.model.dto.ModelTrainDetailDto.ModelValidationMetrics;
import com.kamco.cd.training.model.dto.ModelTrainDetailDto.TransferDetailDto;
import com.kamco.cd.training.model.dto.ModelTrainMngDto.Basic;
import com.kamco.cd.training.model.service.ModelTrainDetailService;
import io.swagger.v3.oas.annotations.Operation;
import io.swagger.v3.oas.annotations.Parameter;
import io.swagger.v3.oas.annotations.media.ArraySchema;
import io.swagger.v3.oas.annotations.media.Content;
import io.swagger.v3.oas.annotations.media.Schema;
import io.swagger.v3.oas.annotations.responses.ApiResponse;
import io.swagger.v3.oas.annotations.responses.ApiResponses;
import io.swagger.v3.oas.annotations.tags.Tag;
import java.util.List;
import java.util.UUID;
import lombok.RequiredArgsConstructor;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
@RestController
@RequiredArgsConstructor
@Tag(name = "모델학습 관리", description = "어드민 홈 > 모델학습관리 > 모델관리 > 목록")
@RequestMapping("/api/models")
public class ModelTrainDetailApiController {
private final ModelTrainDetailService modelTrainDetailService;
@Operation(summary = "모델학습관리> 모델관리 > 상세정보탭 > 학습 진행정보", description = "학습 진행정보, 모델학습 정보 API")
@ApiResponses(
value = {
@ApiResponse(
responseCode = "200",
description = "조회 성공",
content =
@Content(
mediaType = "application/json",
schema = @Schema(implementation = Basic.class))),
@ApiResponse(responseCode = "404", description = "데이터셋을 찾을 수 없음", content = @Content),
@ApiResponse(responseCode = "500", description = "서버 오류", content = @Content)
})
@GetMapping("/detail/{uuid}")
public ApiResponseDto<Basic> findByModelByUUID(
@Parameter(description = "모델학습 uuid", example = "7fbdff54-ea87-4b02-90d1-955fa2a3457e")
@PathVariable
UUID uuid) {
return ApiResponseDto.ok(modelTrainDetailService.findByModelByUUID(uuid));
}
@Operation(summary = "모델학습관리> 모델관리 > 상세정보탭 > 요약정보", description = "상세정보 탭 요약정보 API")
@ApiResponses(
value = {
@ApiResponse(
responseCode = "200",
description = "조회 성공",
content =
@Content(
mediaType = "application/json",
schema = @Schema(implementation = ModelTrainDetailDto.DetailSummary.class))),
@ApiResponse(responseCode = "404", description = "데이터셋을 찾을 수 없음", content = @Content),
@ApiResponse(responseCode = "500", description = "서버 오류", content = @Content)
})
@GetMapping("/summary/{uuid}")
public ApiResponseDto<ModelTrainDetailDto.DetailSummary> getModelDetailSummary(
@Parameter(description = "모델학습 uuid", example = "7fbdff54-ea87-4b02-90d1-955fa2a3457e")
@PathVariable
UUID uuid) {
return ApiResponseDto.ok(modelTrainDetailService.getModelDetailSummary(uuid));
}
@Operation(summary = "모델학습관리> 모델관리 > 상세정보탭 > 하이퍼파라미터 요약 정보", description = "하이퍼파라미터 요약 정보 API")
@ApiResponses(
value = {
@ApiResponse(
responseCode = "200",
description = "조회 성공",
content =
@Content(
mediaType = "application/json",
schema = @Schema(implementation = ModelTrainDetailDto.HyperSummary.class))),
@ApiResponse(responseCode = "404", description = "데이터셋을 찾을 수 없음", content = @Content),
@ApiResponse(responseCode = "500", description = "서버 오류", content = @Content)
})
@GetMapping("/hyper-summary/{uuid}")
public ApiResponseDto<ModelTrainDetailDto.HyperSummary> getByModelHyperParamSummary(
@Parameter(description = "모델학습 uuid", example = "7fbdff54-ea87-4b02-90d1-955fa2a3457e")
@PathVariable
UUID uuid) {
return ApiResponseDto.ok(modelTrainDetailService.getByModelHyperParamSummary(uuid));
}
@Operation(summary = "모델학습관리> 모델관리 > 상세정보탭 > 데이터셋 정보", description = "모델학습 상세 데이터셋 정보 API")
@ApiResponses({
@ApiResponse(
responseCode = "200",
description = "조회 성공",
content =
@Content(
mediaType = "application/json",
array = @ArraySchema(schema = @Schema(implementation = MappingDataset.class)))),
@ApiResponse(responseCode = "404", description = "데이터셋을 찾을 수 없음"),
@ApiResponse(responseCode = "500", description = "서버 오류")
})
@GetMapping("/mapp-dataset/{uuid}")
public ApiResponseDto<List<MappingDataset>> getByModelMappingDataset(
@Parameter(description = "모델학습 uuid", example = "7fbdff54-ea87-4b02-90d1-955fa2a3457e")
@PathVariable
UUID uuid) {
return ApiResponseDto.ok(modelTrainDetailService.getByModelMappingDataset(uuid));
}
@Operation(summary = "모델관리 > 전이 학습 실행설정 > 모델선택", description = "모델선택 정보 API")
@ApiResponses(
value = {
@ApiResponse(
responseCode = "200",
description = "조회 성공",
content =
@Content(
mediaType = "application/json",
schema = @Schema(implementation = TransferDetailDto.class))),
@ApiResponse(responseCode = "404", description = "데이터셋을 찾을 수 없음", content = @Content),
@ApiResponse(responseCode = "500", description = "서버 오류", content = @Content)
})
@GetMapping("/transfer/detail/{uuid}")
public ApiResponseDto<TransferDetailDto> getTransferDetail(
@Parameter(description = "모델 uuid", example = "7fbdff54-ea87-4b02-90d1-955fa2a3457e")
@PathVariable
UUID uuid) {
return ApiResponseDto.ok(modelTrainDetailService.getTransferDetail(uuid));
}
@Operation(summary = "모델관리 > 모델 상세 > 성능 정보 (Train)", description = "모델 상세 > 성능 정보 (Train) API")
@ApiResponses(
value = {
@ApiResponse(
responseCode = "200",
description = "조회 성공",
content =
@Content(
mediaType = "application/json",
schema = @Schema(implementation = TransferDetailDto.class))),
@ApiResponse(responseCode = "404", description = "데이터셋을 찾을 수 없음", content = @Content),
@ApiResponse(responseCode = "500", description = "서버 오류", content = @Content)
})
@GetMapping("/metrics/train/{uuid}")
public ApiResponseDto<List<ModelTrainMetrics>> getModelTrainMetricResult(
@Parameter(description = "모델 uuid", example = "95cb116c-380a-41c0-98d8-4d1142f15bbf")
@PathVariable
UUID uuid) {
return ApiResponseDto.ok(modelTrainDetailService.getModelTrainMetricResult(uuid));
}
@Operation(
summary = "모델관리 > 모델 상세 > 성능 정보 (Validation)",
description = "모델 상세 > 성능 정보 (Validation) API")
@ApiResponses(
value = {
@ApiResponse(
responseCode = "200",
description = "조회 성공",
content =
@Content(
mediaType = "application/json",
schema = @Schema(implementation = TransferDetailDto.class))),
@ApiResponse(responseCode = "404", description = "데이터셋을 찾을 수 없음", content = @Content),
@ApiResponse(responseCode = "500", description = "서버 오류", content = @Content)
})
@GetMapping("/metrics/validation/{uuid}")
public ApiResponseDto<List<ModelValidationMetrics>> getModelValidationMetricResult(
@Parameter(description = "모델 uuid", example = "95cb116c-380a-41c0-98d8-4d1142f15bbf")
@PathVariable
UUID uuid) {
return ApiResponseDto.ok(modelTrainDetailService.getModelValidationMetricResult(uuid));
}
@Operation(summary = "모델관리 > 모델 상세 > 성능 정보 (Test)", description = "모델 상세 > 성능 정보 (Test) API")
@ApiResponses(
value = {
@ApiResponse(
responseCode = "200",
description = "조회 성공",
content =
@Content(
mediaType = "application/json",
schema = @Schema(implementation = TransferDetailDto.class))),
@ApiResponse(responseCode = "404", description = "데이터셋을 찾을 수 없음", content = @Content),
@ApiResponse(responseCode = "500", description = "서버 오류", content = @Content)
})
@GetMapping("/metrics/test/{uuid}")
public ApiResponseDto<List<ModelTestMetrics>> getModelTestMetricResult(
@Parameter(description = "모델 uuid", example = "95cb116c-380a-41c0-98d8-4d1142f15bbf")
@PathVariable
UUID uuid) {
return ApiResponseDto.ok(modelTrainDetailService.getModelTestMetricResult(uuid));
}
@Operation(summary = "모델관리 > 모델 상세 > 성능 정보 (Test)", description = "모델 상세 > 성능 정보 (Test) API")
@ApiResponses(
value = {
@ApiResponse(
responseCode = "200",
description = "조회 성공",
content =
@Content(
mediaType = "application/json",
schema = @Schema(implementation = TransferDetailDto.class))),
@ApiResponse(responseCode = "404", description = "데이터셋을 찾을 수 없음", content = @Content),
@ApiResponse(responseCode = "500", description = "서버 오류", content = @Content)
})
@GetMapping("/best-epoch/{uuid}")
public ApiResponseDto<ModelBestEpoch> getModelTrainBestEpoch(
@Parameter(description = "모델 uuid", example = "95cb116c-380a-41c0-98d8-4d1142f15bbf")
@PathVariable
UUID uuid) {
return ApiResponseDto.ok(modelTrainDetailService.getModelTrainBestEpoch(uuid));
}
}

View File

@@ -0,0 +1,169 @@
package com.kamco.cd.training.model;
import com.kamco.cd.training.config.api.ApiResponseDto;
import com.kamco.cd.training.dataset.dto.DatasetDto;
import com.kamco.cd.training.dataset.dto.DatasetDto.DatasetReq;
import com.kamco.cd.training.dataset.dto.DatasetDto.SelectDataSet;
import com.kamco.cd.training.model.dto.ModelConfigDto;
import com.kamco.cd.training.model.dto.ModelTrainMngDto;
import com.kamco.cd.training.model.dto.ModelTrainMngDto.Basic;
import com.kamco.cd.training.model.service.ModelTrainMngService;
import io.swagger.v3.oas.annotations.Operation;
import io.swagger.v3.oas.annotations.Parameter;
import io.swagger.v3.oas.annotations.media.Content;
import io.swagger.v3.oas.annotations.media.Schema;
import io.swagger.v3.oas.annotations.responses.ApiResponse;
import io.swagger.v3.oas.annotations.responses.ApiResponses;
import io.swagger.v3.oas.annotations.tags.Tag;
import jakarta.validation.Valid;
import java.util.List;
import java.util.UUID;
import lombok.RequiredArgsConstructor;
import org.springframework.data.domain.Page;
import org.springframework.web.bind.annotation.DeleteMapping;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
@RestController
@RequiredArgsConstructor
@Tag(name = "모델학습 관리", description = "어드민 홈 > 모델학습관리 > 모델관리 > 목록")
@RequestMapping("/api/models")
public class ModelTrainMngApiController {
private final ModelTrainMngService modelTrainMngService;
@Operation(summary = "모델학습 목록 조회", description = "모델학습 목록 조회 API")
@ApiResponses(
value = {
@ApiResponse(
responseCode = "200",
description = "검색 성공",
content =
@Content(
mediaType = "application/json",
schema = @Schema(implementation = Page.class))),
@ApiResponse(responseCode = "400", description = "잘못된 검색 조건", content = @Content),
@ApiResponse(responseCode = "500", description = "서버 오류", content = @Content)
})
@GetMapping("/list")
public ApiResponseDto<Page<Basic>> findByModelList(
@Parameter(
description = "상태코드",
example = "IN_PROGRESS",
schema = @Schema(allowableValues = {"", "IN_PROGRESS", "COMPLETED"}))
@RequestParam(required = false)
String status,
@Parameter(
description = "모델",
example = "G1",
schema = @Schema(allowableValues = {"G1", "G2", "G3"}))
@RequestParam(required = false)
String modelNo,
@Parameter(description = "페이지 번호") @RequestParam(defaultValue = "0") int page,
@Parameter(description = "페이지 크기") @RequestParam(defaultValue = "20") int size) {
ModelTrainMngDto.SearchReq searchReq =
new ModelTrainMngDto.SearchReq(status, modelNo, page, size);
return ApiResponseDto.ok(modelTrainMngService.getModelList(searchReq));
}
@Operation(summary = "학습 모델 삭제", description = "학습 모델 삭제 API")
@ApiResponses(
value = {
@ApiResponse(responseCode = "200", description = "삭제 성공", content = @Content),
@ApiResponse(responseCode = "409", description = "G1_000001 삭제 불가", content = @Content)
})
@DeleteMapping("/{uuid}")
public ApiResponseDto<Void> deleteModelTrain(
@Parameter(description = "학습 모델 uuid", example = "f2b02229-90f2-45f5-92ea-c56cf1c29f79")
@PathVariable UUID uuid) {
modelTrainMngService.deleteModelTrain(uuid);
return ApiResponseDto.ok(null);
}
@Operation(summary = "학습 모델 등록", description = "학습 모델 등록 API")
@ApiResponses(
value = {
@ApiResponse(responseCode = "200", description = "등록 성공", content = @Content),
@ApiResponse(responseCode = "500", description = "서버 오류", content = @Content)
})
@PostMapping
public ApiResponseDto<UUID> createModelTrain(@Valid @RequestBody ModelTrainMngDto.AddReq req) {
return ApiResponseDto.ok(modelTrainMngService.createModelTrain(req));
}
@Operation(summary = "모델학습 config 정보 조회", description = "모델학습 config 정보 조회 API")
@ApiResponses(
value = {
@ApiResponse(
responseCode = "200",
description = "검색 성공",
content =
@Content(
mediaType = "application/json",
schema = @Schema(implementation = ModelConfigDto.Basic.class))),
@ApiResponse(responseCode = "400", description = "잘못된 검색 조건", content = @Content),
@ApiResponse(responseCode = "500", description = "서버 오류", content = @Content)
})
@GetMapping("/config/{uuid}")
public ApiResponseDto<ModelConfigDto.Basic> updateModelTrain(
@Parameter(description = "모델학습 uuid", example = "7fbdff54-ea87-4b02-90d1-955fa2a3457e")
@PathVariable
UUID uuid) {
return ApiResponseDto.ok(modelTrainMngService.getModelConfigByModelId(uuid));
}
@Operation(summary = "모델별 데이터셋 목록 조회", description = "모델별 데이터셋 목록 조회 API")
@ApiResponses(
value = {
@ApiResponse(
responseCode = "200",
description = "조회 성공",
content =
@Content(
mediaType = "application/json",
schema = @Schema(implementation = DatasetDto.Basic.class))),
@ApiResponse(responseCode = "404", description = "데이터셋을 찾을 수 없음", content = @Content),
@ApiResponse(responseCode = "500", description = "서버 오류", content = @Content)
})
@GetMapping("/select-dataset-list")
public ApiResponseDto<List<SelectDataSet>> getDatasetSelectList(
@Parameter(
description = "모델 구분",
example = "",
schema = @Schema(allowableValues = {"G1", "G2", "G3"}))
@RequestParam
String modelType,
@Parameter(
description = "선택 구분",
example = "",
schema = @Schema(allowableValues = {"CURRENT", "DELIVER", "PRODUCTION"}))
@RequestParam
String selectType) {
DatasetReq req = new DatasetReq();
req.setModelNo(modelType);
req.setDataType(selectType);
return ApiResponseDto.ok(modelTrainMngService.getDatasetSelectList(req));
}
@Operation(summary = "모델학습 1단계 실행중인 것이 있는지 count", description = "모델학습 1단계 실행중인 것이 있는지 count")
@ApiResponses(
value = {
@ApiResponse(
responseCode = "200",
description = "검색 성공",
content =
@Content(
mediaType = "application/json",
schema = @Schema(implementation = Long.class))),
@ApiResponse(responseCode = "400", description = "잘못된 검색 조건", content = @Content),
@ApiResponse(responseCode = "500", description = "서버 오류", content = @Content)
})
@GetMapping("/ing-training-cnt")
public ApiResponseDto<Long> findModelStep1InProgressCnt() {
return ApiResponseDto.ok(modelTrainMngService.findModelStep1InProgressCnt());
}
}

View File

@@ -0,0 +1,23 @@
package com.kamco.cd.training.model.dto;
import lombok.AllArgsConstructor;
import lombok.Getter;
import lombok.NoArgsConstructor;
import lombok.Setter;
public class ModelConfigDto {
@Getter
@Setter
@NoArgsConstructor
@AllArgsConstructor
public static class Basic {
private Long configId;
private Long modelId;
private Integer epochCount;
private Float trainPercent;
private Float validationPercent;
private Float testPercent;
private String memo;
}
}

View File

@@ -1,415 +0,0 @@
package com.kamco.cd.training.model.dto;
import com.kamco.cd.training.common.utils.interfaces.JsonFormatDttm;
import io.swagger.v3.oas.annotations.media.Schema;
import jakarta.validation.constraints.NotBlank;
import jakarta.validation.constraints.NotEmpty;
import jakarta.validation.constraints.NotNull;
import java.time.ZonedDateTime;
import java.util.List;
import java.util.Map;
import java.util.UUID;
import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Getter;
import lombok.NoArgsConstructor;
import lombok.Setter;
import org.springframework.data.domain.PageRequest;
import org.springframework.data.domain.Pageable;
public class ModelMngDto {
@Schema(name = "모델관리 목록 조회", description = "모델관리 목록 조회")
@Getter
@Setter
@NoArgsConstructor
@AllArgsConstructor
@Builder
public static class Basic {
private Long id;
private UUID uuid;
private String modelVer;
@JsonFormatDttm private ZonedDateTime startDttm;
@JsonFormatDttm private ZonedDateTime step1EndDttm;
@JsonFormatDttm private ZonedDateTime step1Duration;
@JsonFormatDttm private ZonedDateTime step2EndDttm;
@JsonFormatDttm private ZonedDateTime step2Duration;
private String step1Status;
private String step2Status;
private String transferStatus;
}
@Schema(name = "searchReq", description = "모델 관리 목록조회 파라미터")
@Getter
@Setter
@NoArgsConstructor
@AllArgsConstructor
public static class SearchReq {
private String status;
// 페이징 파라미터
private int page = 0;
private int size = 20;
public Pageable toPageable() {
return PageRequest.of(page, size);
}
}
@Schema(name = "Detail", description = "모델 상세 정보")
@Getter
@Builder
public static class Detail {
private String uuid;
private String modelVer;
private String hyperVer;
private String epochVer;
private String processStep;
private String statusCd;
private String statusText;
@JsonFormatDttm private ZonedDateTime trainStartDttm;
private Integer epochCnt;
private String datasetRatio;
private Integer bestEpoch;
private Integer confirmedBestEpoch;
@JsonFormatDttm private ZonedDateTime step1EndDttm;
private ZonedDateTime step1Duration;
@JsonFormatDttm private ZonedDateTime step2EndDttm;
private ZonedDateTime step2Duration;
private Integer progressRate;
@JsonFormatDttm private ZonedDateTime createdDttm;
@JsonFormatDttm private ZonedDateTime updatedDttm;
private String modelPath;
private String errorMsg;
}
@Schema(name = "TrainListRes", description = "학습 모델 목록 응답")
@Getter
@Setter
@NoArgsConstructor
@AllArgsConstructor
@Builder
public static class TrainListRes {
private String uuid;
private String modelVer;
private String status;
private String processStep;
@JsonFormatDttm private ZonedDateTime trainStartDttm;
private Integer progressRate;
private Integer epochCnt;
@JsonFormatDttm private ZonedDateTime step1EndDttm;
private ZonedDateTime step1Duration;
@JsonFormatDttm private ZonedDateTime step2EndDttm;
private ZonedDateTime step2Duration;
@JsonFormatDttm private ZonedDateTime createdDttm;
private String errorMsg;
private Boolean canResume;
private Integer lastCheckpointEpoch;
}
@Schema(name = "FormConfigRes", description = "학습 설정 통합 조회 응답")
@Getter
@Setter
@NoArgsConstructor
@AllArgsConstructor
@Builder
public static class FormConfigRes {
private Boolean isTrainAvailable;
private List<HyperParamInfo> hyperParams;
private List<DatasetInfo> datasets;
private String runningModelUuid;
}
@Schema(name = "HyperParamInfo", description = "하이퍼파라미터 정보")
@Getter
@Setter
@NoArgsConstructor
@AllArgsConstructor
@Builder
public static class HyperParamInfo {
@Schema(description = "하이퍼파라미터 버전", example = "V3.99.251221.120518")
private String hyperVer;
// Important
@Schema(description = "백본", example = "large")
private String backbone;
@Schema(description = "입력 사이즈", example = "256,256")
private String inputSize;
@Schema(description = "크롭 사이즈", example = "256,256")
private String cropSize;
@Schema(description = "에폭 수", example = "200")
private Integer epochCnt;
@Schema(description = "배치 사이즈", example = "16")
private Integer batchSize;
// Architecture
@Schema(description = "Drop Path Rate", example = "0.3")
private Double dropPathRate;
@Schema(description = "Frozen Stages", example = "-1")
private Integer frozenStages;
@Schema(description = "Neck Policy", example = "abs_diff")
private String neckPolicy;
@Schema(description = "Decoder Channels", example = "512,256,128,64")
private String decoderChannels;
@Schema(description = "Class Weight", example = "1,1")
private String classWeight;
@Schema(description = "레이어 수", example = "24")
private Integer numLayers;
// Optimization
@Schema(description = "Learning Rate", example = "0.00006")
private Double learningRate;
@Schema(description = "Weight Decay", example = "0.05")
private Double weightDecay;
@Schema(description = "Layer Decay Rate", example = "0.9")
private Double layerDecayRate;
@Schema(description = "DDP Unused Params 찾기", example = "true")
private Boolean ddpFindUnusedParams;
@Schema(description = "Ignore Index", example = "255")
private Integer ignoreIndex;
// Data
@Schema(description = "Train Workers", example = "16")
private Integer trainNumWorkers;
@Schema(description = "Val Workers", example = "8")
private Integer valNumWorkers;
@Schema(description = "Test Workers", example = "8")
private Integer testNumWorkers;
@Schema(description = "Train Shuffle", example = "true")
private Boolean trainShuffle;
@Schema(description = "Train Persistent", example = "true")
private Boolean trainPersistent;
@Schema(description = "Val Persistent", example = "true")
private Boolean valPersistent;
// Evaluation
@Schema(description = "Metrics", example = "mFscore,mIoU")
private String metrics;
@Schema(description = "Save Best", example = "changed_fscore")
private String saveBest;
@Schema(description = "Save Best Rule", example = "greater")
private String saveBestRule;
@Schema(description = "Val Interval", example = "10")
private Integer valInterval;
@Schema(description = "Log Interval", example = "400")
private Integer logInterval;
@Schema(description = "Vis Interval", example = "1")
private Integer visInterval;
// Hardware
@Schema(description = "GPU 수", example = "4")
private Integer gpuCnt;
@Schema(description = "GPU IDs", example = "0,1,2,3")
private String gpuIds;
@Schema(description = "Master Port", example = "1122")
private Integer masterPort;
// Augmentation
@Schema(description = "Rotation 확률", example = "0.5")
private Double rotProb;
@Schema(description = "Flip 확률", example = "0.5")
private Double flipProb;
@Schema(description = "Rotation 각도", example = "-20,20")
private String rotDegree;
@Schema(description = "Exchange 확률", example = "0.5")
private Double exchangeProb;
@Schema(description = "Brightness Delta", example = "10")
private Integer brightnessDelta;
@Schema(description = "Contrast Range", example = "0.8,1.2")
private String contrastRange;
@Schema(description = "Saturation Range", example = "0.8,1.2")
private String saturationRange;
@Schema(description = "Hue Delta", example = "10")
private Integer hueDelta;
// Legacy
private Double dropoutRatio;
private Integer cnnFilterCnt;
// Common
@Schema(description = "메모", example = "안녕하세요 캠코담당자 입니다. 하이퍼파라미터 신규등록합니다")
private String memo;
@JsonFormatDttm private ZonedDateTime createdDttm;
}
@Schema(name = "DatasetInfo", description = "데이터셋 정보")
@Getter
@Setter
@NoArgsConstructor
@AllArgsConstructor
@Builder
public static class DatasetInfo {
private Long id;
private String title;
private String groupTitle;
private Long totalItems;
private String totalSize;
private Map<String, Integer> classCounts;
private String memo;
@JsonFormatDttm private ZonedDateTime createdDttm;
}
@Schema(name = "TrainStartReq", description = "학습 시작 요청")
@Getter
@Setter
@NoArgsConstructor
@AllArgsConstructor
public static class TrainStartReq {
@NotBlank(message = "하이퍼파라미터 버전은 필수입니다")
@Schema(example = "V3.99.251221.120518")
private String hyperVer;
@NotEmpty(message = "데이터셋은 최소 1개 이상 선택해야 합니다")
private List<Long> datasetIds;
@NotNull(message = "에폭 수는 필수입니다")
@jakarta.validation.constraints.Min(value = 1, message = "에폭 수는 최소 1 이상이어야 합니다")
@jakarta.validation.constraints.Max(value = 200, message = "에폭 수는 최대 200까지 설정 가능합니다")
@Schema(example = "200")
private Integer epoch;
@Schema(example = "7:2:1", description = "데이터 분할 비율 (Training:Validation:Test)")
private String datasetRatio;
}
@Schema(name = "TrainStartRes", description = "학습 시작 응답")
@Getter
@Setter
@NoArgsConstructor
@AllArgsConstructor
@Builder
public static class TrainStartRes {
private String uuid;
private String status;
}
@Schema(name = "ResumeInfo", description = "학습 재시작 정보")
@Getter
@Setter
@NoArgsConstructor
@AllArgsConstructor
@Builder
public static class ResumeInfo {
private Boolean canResume;
private Integer lastEpoch;
private Integer totalEpoch;
private String checkpointPath;
@JsonFormatDttm private ZonedDateTime failedAt;
}
@Schema(name = "ResumeRequest", description = "학습 재시작 요청")
@Getter
@Setter
@NoArgsConstructor
@AllArgsConstructor
public static class ResumeRequest {
@NotNull(message = "재시작 Epoch는 필수입니다")
private Integer resumeFromEpoch;
private Integer newTotalEpoch;
}
@Schema(name = "ResumeResponse", description = "학습 재시작 응답")
@Getter
@Setter
@NoArgsConstructor
@AllArgsConstructor
@Builder
public static class ResumeResponse {
private String uuid;
private String status;
private Integer resumedFromEpoch;
}
@Schema(name = "BestEpochRequest", description = "Best Epoch 설정 요청")
@Getter
@Setter
@NoArgsConstructor
@AllArgsConstructor
public static class BestEpochRequest {
@NotNull(message = "Best Epoch는 필수입니다")
private Integer bestEpoch;
private String reason;
}
@Schema(name = "BestEpochResponse", description = "Best Epoch 설정 응답")
@Getter
@Setter
@NoArgsConstructor
@AllArgsConstructor
@Builder
public static class BestEpochResponse {
private String uuid;
private Integer bestEpoch;
private Integer confirmedBestEpoch;
private Integer previousBestEpoch;
}
@Schema(name = "EpochMetric", description = "Epoch별 성능 지표")
@Getter
@Setter
@NoArgsConstructor
@AllArgsConstructor
@Builder
public static class EpochMetric {
private Integer epoch;
private Double mIoU;
private Double mFscore;
private Double loss;
private Boolean isBest;
}
}

View File

@@ -0,0 +1,248 @@
package com.kamco.cd.training.model.dto;
import com.fasterxml.jackson.annotation.JsonInclude;
import com.kamco.cd.training.common.enums.LearnDataType;
import com.kamco.cd.training.common.enums.TrainStatusType;
import com.kamco.cd.training.common.enums.TrainType;
import com.kamco.cd.training.common.utils.enums.Enums;
import com.kamco.cd.training.common.utils.interfaces.JsonFormatDttm;
import com.kamco.cd.training.dataset.dto.DatasetDto.SelectDataSet;
import io.swagger.v3.oas.annotations.media.Schema;
import java.time.Duration;
import java.time.ZonedDateTime;
import java.util.List;
import java.util.UUID;
import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Getter;
import lombok.NoArgsConstructor;
import lombok.Setter;
public class ModelTrainDetailDto {
@Schema(name = "모델학습관리 목록", description = "모델학습관리 목록")
@Getter
@Setter
@NoArgsConstructor
@AllArgsConstructor
@Builder
public static class DetailSummary {
private Long modelId;
private UUID uuid;
private String modelNo;
private String modelVer;
@JsonFormatDttm private ZonedDateTime step1StrtDttm;
@JsonFormatDttm private ZonedDateTime step2EndDttm;
private String statusCd;
private String trainType;
public String getStatusName() {
if (this.statusCd == null || this.statusCd.isBlank()) return null;
try {
return TrainStatusType.valueOf(this.statusCd).getText(); // 또는 getName()
} catch (IllegalArgumentException e) {
return this.statusCd; // 매핑 못하면 코드 그대로 반환(원하면 null 처리)
}
}
public String getTrainTypeName() {
if (this.trainType == null || this.trainType.isBlank()) return null;
try {
return TrainType.valueOf(this.trainType).getText(); // 또는 getName()
} catch (IllegalArgumentException e) {
return this.trainType; // 매핑 못하면 코드 그대로 반환(원하면 null 처리)
}
}
private String formatDuration(ZonedDateTime start, ZonedDateTime end) {
if (end == null) {
end = ZonedDateTime.now();
}
if (start == null) {
return null;
}
long totalSeconds = Math.abs(Duration.between(start, end).getSeconds());
long hours = totalSeconds / 3600;
long minutes = (totalSeconds % 3600) / 60;
long seconds = totalSeconds % 60;
return String.format("%d시간 %d분 %d초", hours, minutes, seconds);
}
public String getStepAllDuration() {
return formatDuration(this.step1StrtDttm, this.step2EndDttm);
}
}
@Schema(name = "모델학습관리 목록", description = "모델학습관리 목록")
@Getter
@Setter
@NoArgsConstructor
@AllArgsConstructor
@Builder
public static class HyperSummary {
private UUID uuid;
private Long hyperParamId;
private String hyperVer;
private String backbone;
private String inputSize;
private String cropSize;
private Integer batchSize;
}
@Schema(name = "모델학습관리 전이 하이파라미터", description = "모델학습관리 전이 하이파라미터")
@Getter
@Setter
@NoArgsConstructor
@AllArgsConstructor
@Builder
public static class TransferHyperSummary {
private UUID uuid;
private Long hyperParamId;
private String hyperVer;
private String backbone;
private String inputSize;
private String cropSize;
private Integer batchSize;
private UUID beforeUuid;
private Long beforeHyperParamId;
private String beforeHyperVer;
private String beforeBackbone;
private String beforeInputSize;
private String beforeCropSize;
private Integer beforeBatchSize;
}
@Schema(name = "선택한 데이터셋 목록", description = "선택한 데이터셋 목록")
@Getter
@Setter
@NoArgsConstructor
public static class MappingDataset {
private Long modelId;
private Long datasetId;
private String dataType;
private Integer compareYyyy;
private Integer targetYyyy;
private Long roundNo;
private String dataTypeName;
@JsonInclude(JsonInclude.Include.NON_NULL)
private Long buildingCnt;
@JsonInclude(JsonInclude.Include.NON_NULL)
private Long containerCnt;
@JsonInclude(JsonInclude.Include.NON_NULL)
private Long wasteCnt;
@JsonInclude(JsonInclude.Include.NON_NULL)
private Long landCoverCnt;
public MappingDataset(
Long modelId,
Long datasetId,
String dataType,
Integer compareYyyy,
Integer targetYyyy,
Long roundNo,
Long buildingCnt,
Long containerCnt,
Long wasteCnt,
Long landCoverCnt) {
this.modelId = modelId;
this.datasetId = datasetId;
this.dataType = dataType;
this.compareYyyy = compareYyyy;
this.targetYyyy = targetYyyy;
this.roundNo = roundNo;
this.buildingCnt = buildingCnt;
this.containerCnt = containerCnt;
this.wasteCnt = wasteCnt;
this.landCoverCnt = landCoverCnt;
this.dataTypeName = getDataTypeName(this.dataType);
}
public String getDataTypeName(String groupTitleCd) {
LearnDataType type = Enums.fromId(LearnDataType.class, groupTitleCd);
return type == null ? null : type.getText();
}
}
@Getter
@Setter
@NoArgsConstructor
@AllArgsConstructor
public static class TransferDetailDto {
private ModelConfigDto.Basic etcConfig;
private TransferHyperSummary modelTrainHyper;
private List<SelectDataSet> modelTrainDataset;
}
@Getter
@Setter
@NoArgsConstructor
@AllArgsConstructor
public static class ModelTrainMetrics {
private Integer epoch;
private Long iteration;
private Double loss;
private Double lr;
private Float durationTime;
}
@Getter
@Setter
@NoArgsConstructor
@AllArgsConstructor
public static class ModelValidationMetrics {
private Integer epoch;
private Float aAcc;
private Float mFscore;
private Float mPrecision;
private Float mRecall;
private Float mIou;
private Float mAcc;
private Float changedFscore;
private Float changedPrecision;
private Float changedRecall;
private Float unchangedFscore;
private Float unchangedPrecision;
private Float unchangedRecall;
}
@Getter
@Setter
@NoArgsConstructor
@AllArgsConstructor
public static class ModelTestMetrics {
private String model;
private Long tp;
private Long fp;
private Long fn;
private Float precision;
private Float recall;
private Float f1Score;
private Float accuracy;
private Float iou;
private Long detectionCount;
private Long gtCount;
}
@Getter
@Setter
@NoArgsConstructor
@AllArgsConstructor
public static class ModelBestEpoch {
private Integer epoch;
private Double loss;
private Float f1Score;
private Float precision;
private Float recall;
private Float iou;
private Float accuracy;
}
}

View File

@@ -0,0 +1,212 @@
package com.kamco.cd.training.model.dto;
import com.kamco.cd.training.common.dto.HyperParam;
import com.kamco.cd.training.common.enums.TrainStatusType;
import com.kamco.cd.training.common.enums.TrainType;
import com.kamco.cd.training.common.utils.interfaces.JsonFormatDttm;
import io.swagger.v3.oas.annotations.media.Schema;
import jakarta.validation.constraints.NotNull;
import java.time.Duration;
import java.time.ZonedDateTime;
import java.util.List;
import java.util.UUID;
import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Getter;
import lombok.NoArgsConstructor;
import lombok.Setter;
import org.springframework.data.domain.PageRequest;
import org.springframework.data.domain.Pageable;
public class ModelTrainMngDto {
@Schema(name = "모델학습관리 목록", description = "모델학습관리 목록")
@Getter
@Setter
@NoArgsConstructor
@AllArgsConstructor
@Builder
public static class Basic {
private Long id;
private UUID uuid;
private String modelVer;
@JsonFormatDttm private ZonedDateTime startDttm;
@JsonFormatDttm private ZonedDateTime step1StrtDttm;
@JsonFormatDttm private ZonedDateTime step1EndDttm;
@JsonFormatDttm private ZonedDateTime step2StrtDttm;
@JsonFormatDttm private ZonedDateTime step2EndDttm;
private String step1Status;
private String step2Status;
private String statusCd;
private String trainType;
private String modelNo;
private Long currentAttemptId;
public String getStatusName() {
if (this.statusCd == null || this.statusCd.isBlank()) return null;
try {
return TrainStatusType.valueOf(this.statusCd).getText(); // 또는 getName()
} catch (IllegalArgumentException e) {
return this.statusCd; // 매핑 못하면 코드 그대로 반환(원하면 null 처리)
}
}
public String getStep1StatusName() {
if (this.step1Status == null || this.step1Status.isBlank()) return null;
try {
return TrainStatusType.valueOf(this.step1Status).getText(); // 또는 getName()
} catch (IllegalArgumentException e) {
return this.step1Status; // 매핑 못하면 코드 그대로 반환(원하면 null 처리)
}
}
public String getStep2StatusNAme() {
if (this.step2Status == null || this.step2Status.isBlank()) return null;
try {
return TrainStatusType.valueOf(this.step2Status).getText(); // 또는 getName()
} catch (IllegalArgumentException e) {
return this.step2Status; // 매핑 못하면 코드 그대로 반환(원하면 null 처리)
}
}
public String getTrainTypeName() {
if (this.trainType == null || this.trainType.isBlank()) return null;
try {
return TrainType.valueOf(this.trainType).getText(); // 또는 getName()
} catch (IllegalArgumentException e) {
return this.trainType; // 매핑 못하면 코드 그대로 반환(원하면 null 처리)
}
}
private String formatDuration(ZonedDateTime start, ZonedDateTime end) {
if (start == null || end == null) {
return null;
}
long totalSeconds = Math.abs(Duration.between(start, end).getSeconds());
long hours = totalSeconds / 3600;
long minutes = (totalSeconds % 3600) / 60;
long seconds = totalSeconds % 60;
return String.format("%d시간 %d분 %d초", hours, minutes, seconds);
}
public String getStep1Duration() {
return formatDuration(this.step1StrtDttm, this.step1EndDttm);
}
public String getStep2Duration() {
return formatDuration(this.step2StrtDttm, this.step2EndDttm);
}
}
@Schema(name = "searchReq", description = "모델학습 관리 목록조회 파라미터")
@Getter
@Setter
@NoArgsConstructor
@AllArgsConstructor
public static class SearchReq {
private String status;
private String modelNo;
// 페이징 파라미터
private int page = 0;
private int size = 20;
public Pageable toPageable() {
return PageRequest.of(page, size);
}
}
@Schema(name = "addReq", description = "모델학습 관리 등록 파라미터")
@Getter
@Setter
@NoArgsConstructor
@AllArgsConstructor
public static class AddReq {
@NotNull
@Schema(description = "모델 종류 G1, G2, G3", example = "G1")
private String modelNo;
@NotNull
@Schema(description = "모델학습 실행 여부", example = "false")
private Boolean isStart;
@NotNull
@Schema(description = "학습타입 GENERAL(일반), TRANSFER(전이)", example = "GENERAL")
private String trainType;
@Schema(description = "전이학습일때 선택한 모델 id")
private Long beforeModelId;
@NotNull
@Schema(
description = "하이퍼 파라미터 선택 타입 OPTIMIZED(최적화 파라미터),EXISTING(기존 파라미터),NEW(신규 파라미터)",
example = "EXISTING")
private String hyperParamType;
@Schema(description = "하이퍼파라미터 uuid", example = "57fc9170-64c1-4128-aa7b-0657f08d6d10")
private UUID hyperUuid;
HyperParam hyperParam;
TrainingDataset trainingDataset;
ModelConfig modelConfig;
}
@Schema(name = "addReq", description = "모델학습 관리 등록 파라미터")
@Getter
@Setter
@NoArgsConstructor
@AllArgsConstructor
public static class UpdateReq {
private String requestPath;
private String responsePath;
}
@Getter
@Setter
public static class TrainingDataset {
Summary summary;
List<Long> datasetList;
}
@Getter
@Setter
public static class Summary {
@Schema(description = "건물", example = "0")
private Long buildingCnt;
@Schema(description = "컨테이너", example = "0")
private Long containerCnt;
@Schema(description = "폐기물", example = "0")
private Long wasteCnt;
@Schema(
description = "도로, 비닐하우스, 밭, 과수원, 초지, 숲, 물, 모재/자갈, 토분(무덤), 일반토지, 태양광, 기타",
example = "0")
private Long LandCoverCnt;
}
@Getter
@Setter
public static class ModelConfig {
@Schema(description = "에폭 횟수", example = "0")
private Integer epochCnt;
@Schema(description = "학습데이터셋 비율 Training", example = "0")
private Float trainingCnt;
@Schema(description = "학습데이터셋 비율 Validation", example = "0")
private Float validationCnt;
@Schema(description = "학습데이터셋 비율 Test", example = "0")
private Float testCnt;
@Schema(description = "메모", example = "메모 입니다.")
private String memo;
}
}

View File

@@ -1,61 +0,0 @@
package com.kamco.cd.training.model.dto;
import com.kamco.cd.training.common.utils.interfaces.JsonFormatDttm;
import io.swagger.v3.oas.annotations.media.Schema;
import java.time.ZonedDateTime;
import lombok.Getter;
public class ModelVerDto {
@Schema(name = "modelVer Basic", description = "모델버전 엔티티 기본 정보")
@Getter
public static class Basic {
private final Long id;
private final Long modelUid;
private final String modelCate;
private final String modelVer;
private final String usedState;
private final String modelState;
private final Double qualityProb;
private final String deployState;
private final String modelPath;
@JsonFormatDttm private final ZonedDateTime createdDttm;
private final Long createdUid;
@JsonFormatDttm private final ZonedDateTime updatedDttm;
private final Long updatedUid;
public Basic(
Long id,
Long modelUid,
String modelCate,
String modelVer,
String usedState,
String modelState,
Double qualityProb,
String deployState,
String modelPath,
ZonedDateTime createdDttm,
Long createdUid,
ZonedDateTime updatedDttm,
Long updatedUid) {
this.id = id;
this.modelUid = modelUid;
this.modelCate = modelCate;
this.modelVer = modelVer;
this.usedState = usedState;
this.modelState = modelState;
this.qualityProb = qualityProb;
this.deployState = deployState;
this.modelPath = modelPath;
this.createdDttm = createdDttm;
this.createdUid = createdUid;
this.updatedDttm = updatedDttm;
this.updatedUid = updatedUid;
}
}
}

View File

@@ -1,50 +0,0 @@
package com.kamco.cd.training.model.service;
import com.kamco.cd.training.model.dto.ModelMngDto;
import com.kamco.cd.training.model.dto.ModelMngDto.Basic;
import com.kamco.cd.training.model.dto.ModelMngDto.SearchReq;
import com.kamco.cd.training.postgres.core.ModelMngCoreService;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.data.domain.Page;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
@Service
@RequiredArgsConstructor
@Transactional(readOnly = true)
@Slf4j
public class ModelMngService {
private final ModelMngCoreService modelMngCoreService;
/**
* 모델 목록 조회
*
* @param searchReq 검색 조건
* @return 페이징 처리된 모델 목록
*/
public Page<Basic> findByModels(SearchReq searchReq) {
return modelMngCoreService.findByModels(searchReq);
}
/**
* 모델 상세 조회
*
* @param modelUid 모델 UID
* @return 모델 상세 정보
*/
public ModelMngDto.Detail getModelDetail(Long modelUid) {
return modelMngCoreService.getModelDetail(modelUid);
}
/**
* 모델 상세 조회 (UUID 기반)
*
* @param uuid 모델 UUID
* @return 모델 상세 정보
*/
public ModelMngDto.Detail getModelDetailByUuid(String uuid) {
return modelMngCoreService.getModelDetailByUuid(uuid);
}
}

View File

@@ -0,0 +1,119 @@
package com.kamco.cd.training.model.service;
import com.kamco.cd.training.dataset.dto.DatasetDto.DatasetReq;
import com.kamco.cd.training.dataset.dto.DatasetDto.SelectDataSet;
import com.kamco.cd.training.model.dto.ModelConfigDto;
import com.kamco.cd.training.model.dto.ModelTrainDetailDto.DetailSummary;
import com.kamco.cd.training.model.dto.ModelTrainDetailDto.HyperSummary;
import com.kamco.cd.training.model.dto.ModelTrainDetailDto.MappingDataset;
import com.kamco.cd.training.model.dto.ModelTrainDetailDto.ModelBestEpoch;
import com.kamco.cd.training.model.dto.ModelTrainDetailDto.ModelTestMetrics;
import com.kamco.cd.training.model.dto.ModelTrainDetailDto.ModelTrainMetrics;
import com.kamco.cd.training.model.dto.ModelTrainDetailDto.ModelValidationMetrics;
import com.kamco.cd.training.model.dto.ModelTrainDetailDto.TransferDetailDto;
import com.kamco.cd.training.model.dto.ModelTrainDetailDto.TransferHyperSummary;
import com.kamco.cd.training.model.dto.ModelTrainMngDto.Basic;
import com.kamco.cd.training.postgres.core.ModelTrainDetailCoreService;
import com.kamco.cd.training.postgres.core.ModelTrainMngCoreService;
import java.util.ArrayList;
import java.util.List;
import java.util.UUID;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
@Service
@RequiredArgsConstructor
@Transactional(readOnly = true)
@Slf4j
public class ModelTrainDetailService {
private final ModelTrainDetailCoreService modelTrainDetailCoreService;
private final ModelTrainMngCoreService mngCoreService;
/**
* 모델 상세정보 요약
*
* @param uuid
* @return
*/
public DetailSummary getModelDetailSummary(UUID uuid) {
return modelTrainDetailCoreService.getModelDetailSummary(uuid);
}
/**
* 모델 하이퍼파라미터 요약
*
* @param uuid
* @return
*/
public HyperSummary getByModelHyperParamSummary(UUID uuid) {
return modelTrainDetailCoreService.getByModelHyperParamSummary(uuid);
}
public List<MappingDataset> getByModelMappingDataset(UUID uuid) {
return modelTrainDetailCoreService.getByModelMappingDataset(uuid);
}
public Basic findByModelByUUID(UUID uuid) {
return modelTrainDetailCoreService.findByModelByUUID(uuid);
}
/**
* 전이학습 모델선택 정보
*
* @param uuid
* @return
*/
public TransferDetailDto getTransferDetail(UUID uuid) {
Basic modelInfo = modelTrainDetailCoreService.findByModelByUUID(uuid);
// config 정보 조회
ModelConfigDto.Basic configInfo = mngCoreService.findModelConfigByModelId(uuid);
// 하이파라미터 정보 조회
TransferHyperSummary hyperSummary = modelTrainDetailCoreService.getTransferHyperSummary(uuid);
List<SelectDataSet> dataSets = new ArrayList<>();
DatasetReq datasetReq = new DatasetReq();
List<Long> datasetIds = new ArrayList<>();
List<MappingDataset> mappingDatasets =
modelTrainDetailCoreService.getByModelMappingDataset(uuid);
for (MappingDataset mappingDataset : mappingDatasets) {
datasetIds.add(mappingDataset.getDatasetId());
}
datasetReq.setIds(datasetIds);
datasetReq.setModelNo(modelInfo.getModelNo());
if (modelInfo.getModelNo().equals("G1")) {
dataSets = mngCoreService.getDatasetSelectG1List(datasetReq);
} else {
dataSets = mngCoreService.getDatasetSelectG2G3List(datasetReq);
}
TransferDetailDto transferDetailDto = new TransferDetailDto();
transferDetailDto.setEtcConfig(configInfo);
transferDetailDto.setModelTrainHyper(hyperSummary);
transferDetailDto.setModelTrainDataset(dataSets);
return transferDetailDto;
}
public List<ModelTrainMetrics> getModelTrainMetricResult(UUID uuid) {
return modelTrainDetailCoreService.getModelTrainMetricResult(uuid);
}
public List<ModelValidationMetrics> getModelValidationMetricResult(UUID uuid) {
return modelTrainDetailCoreService.getModelValidationMetricResult(uuid);
}
public List<ModelTestMetrics> getModelTestMetricResult(UUID uuid) {
return modelTrainDetailCoreService.getModelTestMetricResult(uuid);
}
public ModelBestEpoch getModelTrainBestEpoch(UUID uuid) {
return modelTrainDetailCoreService.getModelTrainBestEpoch(uuid);
}
}

View File

@@ -0,0 +1,143 @@
package com.kamco.cd.training.model.service;
import com.kamco.cd.training.common.dto.HyperParam;
import com.kamco.cd.training.common.enums.HyperParamSelectType;
import com.kamco.cd.training.common.enums.TrainType;
import com.kamco.cd.training.common.exception.CustomApiException;
import com.kamco.cd.training.dataset.dto.DatasetDto.DatasetReq;
import com.kamco.cd.training.dataset.dto.DatasetDto.SelectDataSet;
import com.kamco.cd.training.hyperparam.dto.HyperParamDto;
import com.kamco.cd.training.model.dto.ModelConfigDto;
import com.kamco.cd.training.model.dto.ModelTrainMngDto;
import com.kamco.cd.training.model.dto.ModelTrainMngDto.SearchReq;
import com.kamco.cd.training.postgres.core.HyperParamCoreService;
import com.kamco.cd.training.postgres.core.ModelTrainMngCoreService;
import java.io.IOException;
import java.nio.file.Path;
import java.util.List;
import java.util.UUID;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.data.domain.Page;
import org.springframework.http.HttpStatus;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
@Service
@RequiredArgsConstructor
@Transactional(readOnly = true)
@Slf4j
public class ModelTrainMngService {
private final ModelTrainMngCoreService modelTrainMngCoreService;
private final HyperParamCoreService hyperParamCoreService;
private final TmpDatasetService tmpDatasetService;
/**
* 모델학습 조회
*
* @param searchReq 검색 조건
* @return 페이징 처리된 모델 목록
*/
public Page<ModelTrainMngDto.Basic> getModelList(SearchReq searchReq) {
return modelTrainMngCoreService.findByModelList(searchReq);
}
/**
* 모델학습 삭제
*
* @param uuid
*/
@Transactional
public void deleteModelTrain(UUID uuid) {
modelTrainMngCoreService.deleteModel(uuid);
}
/**
* 모델학습 등록
*
* @param req
* @return
*/
@Transactional
public UUID createModelTrain(ModelTrainMngDto.AddReq req) {
HyperParam hyperParam = req.getHyperParam();
HyperParamDto.Basic hyper = new HyperParamDto.Basic();
// 전이 학습은 모델 선택 필수
if (TrainType.TRANSFER.getId().equals(req.getTrainType())) {
if (req.getBeforeModelId() == null) {
throw new CustomApiException("BAD_REQUEST", HttpStatus.BAD_REQUEST, "모델을 선택해 주세요.");
}
}
// 하이파라미터 신규저장
if (HyperParamSelectType.NEW.getId().equals(req.getHyperParamType())) {
// 하이퍼파라미터 등록
hyper = hyperParamCoreService.createHyperParam(hyperParam);
req.setHyperUuid(hyper.getUuid());
}
// 모델학습 테이블 저장
ModelTrainMngDto.Basic modelDto = modelTrainMngCoreService.saveModel(req);
Long modelId = modelDto.getId();
UUID modelUuid = modelDto.getUuid();
// 모델학습 데이터셋 저장
modelTrainMngCoreService.saveModelDataset(modelId, req);
// 모델 데이터셋 mapping 저장
modelTrainMngCoreService.saveModelDatasetMap(
modelId, req.getTrainingDataset().getDatasetList());
// 모델 config 저장
modelTrainMngCoreService.saveModelConfig(modelId, req.getModelConfig());
UUID tmpUuid = UUID.randomUUID();
String raw = tmpUuid.toString().toUpperCase().replace("-", "");
List<String> uids =
modelTrainMngCoreService.findDatasetUid(req.getTrainingDataset().getDatasetList());
try {
// 데이터셋 심볼링크 생성
Path path = tmpDatasetService.buildTmpDatasetSymlink(raw, uids);
ModelTrainMngDto.UpdateReq updateReq = new ModelTrainMngDto.UpdateReq();
updateReq.setRequestPath(path.toString());
modelTrainMngCoreService.updateModelMaster(modelId, updateReq);
} catch (IOException e) {
throw new RuntimeException(e);
}
return modelUuid;
}
/**
* 모델학습 모델학습 uuid config정보 조회
*
* @param uuid 모델학습 uuid
* @return
*/
public ModelConfigDto.Basic getModelConfigByModelId(UUID uuid) {
return modelTrainMngCoreService.findModelConfigByModelId(uuid);
}
/**
* 모델별 데이터셋 목록 조회
*
* @param req
* @return
*/
public List<SelectDataSet> getDatasetSelectList(DatasetReq req) {
if (req.getModelNo().equals("G1")) {
return modelTrainMngCoreService.getDatasetSelectG1List(req);
} else {
return modelTrainMngCoreService.getDatasetSelectG2G3List(req);
}
}
public Long findModelStep1InProgressCnt() {
return modelTrainMngCoreService.findModelStep1InProgressCnt();
}
}

View File

@@ -1,353 +0,0 @@
package com.kamco.cd.training.model.service;
import com.kamco.cd.training.common.exception.BadRequestException;
import com.kamco.cd.training.common.exception.NotFoundException;
import com.kamco.cd.training.model.dto.ModelMngDto;
import com.kamco.cd.training.postgres.core.DatasetCoreService;
import com.kamco.cd.training.postgres.core.HyperParamCoreService;
import com.kamco.cd.training.postgres.core.ModelMngCoreService;
import com.kamco.cd.training.postgres.core.SystemMetricsCoreService;
import com.kamco.cd.training.postgres.entity.ModelTrainMasterEntity;
import java.util.List;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
@Service
@RequiredArgsConstructor
@Transactional(readOnly = true)
@Slf4j
public class ModelTrainService {
private final ModelMngCoreService modelMngCoreService;
private final HyperParamCoreService hyperParamCoreService;
private final DatasetCoreService datasetCoreService;
private final SystemMetricsCoreService systemMetricsCoreService;
/**
* 학습 모델 목록 조회
*
* @return 학습 모델 목록
*/
public List<ModelMngDto.TrainListRes> getTrainModelList() {
return modelMngCoreService.findAllTrainModels();
}
/**
* 학습 설정 통합 조회
*
* @return 학습 설정 폼 데이터
*/
public ModelMngDto.FormConfigRes getFormConfig() {
// 1. 현재 실행 중인 모델 확인
String runningModelUuid = modelMngCoreService.findRunningModelUuid();
boolean isTrainAvailable = (runningModelUuid == null);
// 2. 저장공간 체크 (10GB 미만 시 학습 불가)
if (isTrainAvailable) {
isTrainAvailable = systemMetricsCoreService.isStorageAvailableForTraining();
long availableMB = systemMetricsCoreService.getAvailableStorageMB();
log.info("저장공간 체크 완료: {}MB 사용 가능, 학습 가능 여부: {}", availableMB, isTrainAvailable);
}
// 3. 하이퍼파라미터 목록
List<ModelMngDto.HyperParamInfo> hyperParams =
null; // hyperParamCoreService.findAllActiveHyperParams();
// 4. 데이터셋 목록
List<ModelMngDto.DatasetInfo> datasets = datasetCoreService.findAllActiveDatasetsForTraining();
return ModelMngDto.FormConfigRes.builder()
.isTrainAvailable(isTrainAvailable)
.hyperParams(hyperParams)
.datasets(datasets)
.runningModelUuid(runningModelUuid)
.build();
}
/**
* 학습 시작
*
* @param trainReq 학습 시작 요청
* @return 학습 시작 응답
*/
@Transactional
public ModelMngDto.TrainStartRes startTraining(ModelMngDto.TrainStartReq trainReq) {
// 1. 동시 실행 방지 검증
String runningModelUuid = modelMngCoreService.findRunningModelUuid();
if (runningModelUuid != null) {
throw new BadRequestException(
"이미 실행 중인 학습이 있습니다. 학습은 한 번에 한 개만 실행할 수 있습니다. (실행 중인 모델: " + runningModelUuid + ")");
}
// 2. 저장공간 체크 (10GB 미만 시 학습 불가)
if (!systemMetricsCoreService.isStorageAvailableForTraining()) {
long availableMB = systemMetricsCoreService.getAvailableStorageMB();
long requiredMB = 10 * 1024; // 10GB
throw new BadRequestException(
String.format(
"저장공간이 부족하여 학습을 시작할 수 없습니다. (필요: %dMB, 사용 가능: %dMB)", requiredMB, availableMB));
}
// 3. 데이터셋 상태 검증 (COMPLETED 상태만 학습 가능)
validateDatasetStatus(trainReq.getDatasetIds());
// 4. 데이터 분할 비율 검증 (예: "7:2:1" 형식)
if (trainReq.getDatasetRatio() != null && !trainReq.getDatasetRatio().isEmpty()) {
validateDatasetRatio(trainReq.getDatasetRatio());
}
// 5. 학습 마스터 생성
ModelTrainMasterEntity entity = modelMngCoreService.createTrainMaster(trainReq);
// 5. 데이터셋 매핑 생성
modelMngCoreService.createDatasetMappings(entity.getId(), trainReq.getDatasetIds());
// 6. 실제 UUID 사용
String uuid = entity.getUuid().toString();
log.info(
"학습 시작: uuid={}, hyperVer={}, epoch={}, datasets={}",
uuid,
trainReq.getHyperVer(),
trainReq.getEpoch(),
trainReq.getDatasetIds());
// TODO: 비동기 GPU 학습 프로세스 트리거 로직 추가
return ModelMngDto.TrainStartRes.builder().uuid(uuid).status(entity.getStatusCd()).build();
}
/**
* 데이터셋 상태 검증
*
* @param datasetIds 데이터셋 ID 목록
*/
private void validateDatasetStatus(List<Long> datasetIds) {
for (Long datasetId : datasetIds) {
try {
var dataset = datasetCoreService.getOneById(datasetId);
// COMPLETED 상태가 아닌 데이터셋이 포함되어 있으면 예외 발생
if (dataset.getStatus() == null || !"COMPLETED".equals(dataset.getStatus())) {
throw new BadRequestException(
String.format(
"학습에 사용할 수 없는 데이터셋입니다. (ID: %d, 상태: %s). COMPLETED 상태의 데이터셋만 선택 가능합니다.",
datasetId, dataset.getStatus() != null ? dataset.getStatus() : "NULL"));
}
log.debug("데이터셋 상태 검증 통과: ID={}, Status={}", datasetId, dataset.getStatus());
} catch (NotFoundException e) {
throw new BadRequestException("존재하지 않는 데이터셋입니다. ID: " + datasetId);
}
}
log.info("모든 데이터셋 상태 검증 완료: {} 개", datasetIds.size());
}
/**
* 데이터 분할 비율 검증
*
* @param datasetRatio 데이터셋 비율 (예: "7:2:1")
*/
private void validateDatasetRatio(String datasetRatio) {
try {
String[] parts = datasetRatio.split(":");
if (parts.length != 3) {
throw new BadRequestException("데이터 분할 비율은 'Training:Validation:Test' 형식이어야 합니다 (예: 7:2:1)");
}
int train = Integer.parseInt(parts[0].trim());
int validation = Integer.parseInt(parts[1].trim());
int test = Integer.parseInt(parts[2].trim());
int sum = train + validation + test;
if (sum != 10) {
throw new BadRequestException(
String.format("데이터 분할 비율의 합계는 10이어야 합니다. (현재 합계: %d, 입력값: %s)", sum, datasetRatio));
}
if (train <= 0 || validation < 0 || test < 0) {
throw new BadRequestException("데이터 분할 비율은 모두 0 이상이어야 합니다 (Training은 1 이상)");
}
log.info(
"데이터 분할 비율 검증 완료: Training={}0%, Validation={}0%, Test={}0%", train, validation, test);
} catch (NumberFormatException e) {
throw new BadRequestException("데이터 분할 비율은 숫자로만 구성되어야 합니다: " + datasetRatio);
}
}
/**
* 학습 모델 삭제
*
* @param uuid 모델 UUID
*/
@Transactional
public void deleteTrainModel(String uuid) {
modelMngCoreService.deleteByUuid(uuid);
log.info("학습 모델 삭제 완료: uuid={}", uuid);
}
// ==================== Resume Training (학습 재시작) ====================
/**
* 학습 재시작 정보 조회
*
* @param uuid 모델 UUID
* @return 재시작 정보
*/
public ModelMngDto.ResumeInfo getResumeInfo(String uuid) {
ModelTrainMasterEntity entity = modelMngCoreService.findByUuid(uuid);
return ModelMngDto.ResumeInfo.builder()
.canResume(entity.getCanResume() != null && entity.getCanResume())
.lastEpoch(entity.getLastCheckpointEpoch())
.totalEpoch(entity.getEpochCnt())
.checkpointPath(entity.getCheckpointPath())
// .failedAt(
// entity.getStopDttm() != null
// ? entity.getStopDttm().atZone(java.time.ZoneId.systemDefault())
// : null)
.build();
}
/**
* 학습 재시작
*
* @param uuid 모델 UUID
* @param resumeReq 재시작 요청
* @return 재시작 응답
*/
@Transactional
public ModelMngDto.ResumeResponse resumeTraining(
String uuid, ModelMngDto.ResumeRequest resumeReq) {
ModelTrainMasterEntity entity = modelMngCoreService.findByUuid(uuid);
// 재시작 가능 여부 검증
if (entity.getCanResume() == null || !entity.getCanResume()) {
throw new IllegalStateException("학습 재시작이 불가능한 모델입니다: " + uuid);
}
if (entity.getLastCheckpointEpoch() == null) {
throw new IllegalStateException("Checkpoint가 존재하지 않습니다: " + uuid);
}
// 상태 업데이트
entity.setStatusCd("RUNNING");
entity.setProgressRate(0);
// 총 Epoch 수 변경 (선택사항)
if (resumeReq.getNewTotalEpoch() != null) {
entity.setEpochCnt(resumeReq.getNewTotalEpoch());
}
log.info(
"학습 재시작: uuid={}, resumeFromEpoch={}, totalEpoch={}",
uuid,
resumeReq.getResumeFromEpoch(),
entity.getEpochCnt());
// TODO: 비동기 GPU 학습 재시작 프로세스 트리거 로직 추가
// - Checkpoint 파일 로드
// - 지정된 Epoch부터 학습 재개
return ModelMngDto.ResumeResponse.builder()
.uuid(uuid)
.status(entity.getStatusCd())
.resumedFromEpoch(resumeReq.getResumeFromEpoch())
.build();
}
// ==================== Best Epoch Setting (Best Epoch 설정) ====================
/**
* Best Epoch 설정
*
* @param uuid 모델 UUID
* @param bestEpochReq Best Epoch 요청
* @return Best Epoch 응답
*/
@Transactional
public ModelMngDto.BestEpochResponse setBestEpoch(
String uuid, ModelMngDto.BestEpochRequest bestEpochReq) {
ModelTrainMasterEntity entity = modelMngCoreService.findByUuid(uuid);
// 1차 학습 완료 상태 검증
if (!"STEP1_COMPLETED".equals(entity.getStatusCd())
&& !"STEP1".equals(entity.getProcessStep())) {
log.warn(
"Best Epoch 설정 시도: 현재 상태={}, processStep={}",
entity.getStatusCd(),
entity.getProcessStep());
}
Integer previousBestEpoch = entity.getConfirmedBestEpoch();
// 사용자가 확정한 Best Epoch 설정
entity.setConfirmedBestEpoch(bestEpochReq.getBestEpoch());
// 2차 학습(Test) 단계로 상태 전이
entity.setProcessStep("STEP2");
entity.setStatusCd("STEP2_RUNNING");
entity.setProgressRate(0);
entity.setUpdatedDttm(java.time.ZonedDateTime.now());
log.info(
"Best Epoch 설정 및 2차 학습 시작: uuid={}, newBestEpoch={}, previousBestEpoch={}, reason={}, newStatus={}",
uuid,
bestEpochReq.getBestEpoch(),
previousBestEpoch,
bestEpochReq.getReason(),
entity.getStatusCd());
// TODO: 비동기 GPU 2차 학습(Test) 프로세스 트리거 로직 추가
// - Best Epoch 모델 로드
// - Test 데이터셋으로 성능 평가 실행
// - 완료 시 STEP2_COMPLETED 상태로 전환
return ModelMngDto.BestEpochResponse.builder()
.uuid(uuid)
.bestEpoch(entity.getBestEpoch()) // 자동 선택된 값
.confirmedBestEpoch(entity.getConfirmedBestEpoch()) // 사용자 확정 값
.previousBestEpoch(previousBestEpoch)
.build();
}
/**
* Epoch별 성능 지표 조회
*
* @param uuid 모델 UUID
* @return Epoch별 성능 지표 목록
*/
public List<ModelMngDto.EpochMetric> getEpochMetrics(String uuid) {
ModelTrainMasterEntity entity = modelMngCoreService.findByUuid(uuid);
// TODO: 실제 학습 로그 파일이나 DB에서 Epoch별 성능 지표 조회
// 현재는 샘플 데이터 반환
List<ModelMngDto.EpochMetric> metrics = new java.util.ArrayList<>();
if (entity.getEpochCnt() != null && entity.getBestEpoch() != null) {
// 샘플 데이터 생성 (실제로는 학습 로그 파일 파싱 또는 별도 테이블 조회)
for (int i = 1; i <= Math.min(entity.getEpochCnt(), 10); i++) {
int epoch = entity.getBestEpoch() - 5 + i;
if (epoch <= 0 || epoch > entity.getEpochCnt()) {
continue;
}
metrics.add(
ModelMngDto.EpochMetric.builder()
.epoch(epoch)
.mIoU(0.80 + (Math.random() * 0.15)) // 샘플 데이터
.mFscore(0.85 + (Math.random() * 0.10)) // 샘플 데이터
.loss(0.3 - (Math.random() * 0.15)) // 샘플 데이터
.isBest(entity.getBestEpoch() != null && epoch == entity.getBestEpoch())
.build());
}
}
log.info("Epoch별 성능 지표 조회: uuid={}, metricsCount={}", uuid, metrics.size());
return metrics;
}
}

View File

@@ -0,0 +1,66 @@
package com.kamco.cd.training.model.service;
import java.io.IOException;
import java.nio.file.*;
import java.util.List;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
@Slf4j
@Service
@RequiredArgsConstructor
public class TmpDatasetService {
@Value("${train.docker.requestDir}")
private String requestDir;
@Transactional(readOnly = true)
public Path buildTmpDatasetSymlink(String uid, List<String> datasetUids) throws IOException {
// 환경에 맞게 yml로 빼는 걸 추천
Path BASE = Paths.get(requestDir);
Path tmp = BASE.resolve("tmp").resolve(uid);
// mkdir -p "$TMP"/train/{input1,input2,label} ...
for (String type : List.of("train", "val")) {
for (String part : List.of("input1", "input2", "label")) {
Files.createDirectories(tmp.resolve(type).resolve(part));
}
}
for (String id : datasetUids) {
Path srcRoot = BASE.resolve(id);
for (String type : List.of("train", "val")) {
for (String part : List.of("input1", "input2", "label")) {
Path srcDir = srcRoot.resolve(type).resolve(part);
// zsh NULL_GLOB: 폴더가 없으면 그냥 continue
if (!Files.isDirectory(srcDir)) continue;
try (DirectoryStream<Path> stream = Files.newDirectoryStream(srcDir)) {
for (Path f : stream) {
if (!Files.isRegularFile(f)) continue;
String dstName = id + "__" + f.getFileName();
Path dst = tmp.resolve(type).resolve(part).resolve(dstName);
// 이미 있으면 스킵(원하면 덮어쓰기 로직으로 바꿀 수 있음)
if (Files.exists(dst)) continue;
// ln -s "$f" "$dst" 와 동일
Files.createSymbolicLink(dst, f.toAbsolutePath());
}
}
}
}
}
log.info("tmp dataset created: {}", tmp);
return tmp;
}
}

View File

@@ -1,17 +1,23 @@
package com.kamco.cd.training.postgres.core;
import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.kamco.cd.training.common.enums.LearnDataRegister;
import com.kamco.cd.training.common.enums.LearnDataType;
import com.kamco.cd.training.common.exception.NotFoundException;
import com.kamco.cd.training.common.service.BaseCoreService;
import com.kamco.cd.training.dataset.dto.DatasetDto;
import com.kamco.cd.training.model.dto.ModelMngDto;
import com.kamco.cd.training.dataset.dto.DatasetDto.DatasetMngRegDto;
import com.kamco.cd.training.dataset.dto.DatasetObjDto.Basic;
import com.kamco.cd.training.dataset.dto.DatasetObjDto.DatasetClass;
import com.kamco.cd.training.dataset.dto.DatasetObjDto.DatasetObjRegDto;
import com.kamco.cd.training.dataset.dto.DatasetObjDto.SearchReq;
import com.kamco.cd.training.postgres.entity.DatasetEntity;
import com.kamco.cd.training.postgres.entity.DatasetObjEntity;
import com.kamco.cd.training.postgres.repository.dataset.DatasetObjRepository;
import com.kamco.cd.training.postgres.repository.dataset.DatasetRepository;
import jakarta.transaction.Transactional;
import java.time.ZonedDateTime;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.UUID;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
@@ -25,6 +31,7 @@ import org.springframework.stereotype.Service;
public class DatasetCoreService
implements BaseCoreService<DatasetDto.Basic, Long, DatasetDto.SearchReq> {
private final DatasetRepository datasetRepository;
private final DatasetObjRepository datasetObjRepository;
private final ObjectMapper objectMapper;
/**
@@ -99,25 +106,26 @@ public class DatasetCoreService
// 먼저 id1 필드를 임시값(0)으로 설정하여 저장
DatasetEntity entity = new DatasetEntity();
entity.setTitle(registerReq.getTitle());
entity.setYear(registerReq.getYear());
entity.setGroupTitle("PRODUCTION");
entity.setDataYear(registerReq.getYear());
// entity.setYear(registerReq.getYear());
entity.setDataType(LearnDataType.PRODUCTION.getId());
entity.setCompareYyyy(registerReq.getCompareYear());
entity.setTargetYyyy(registerReq.getTargetYyyy());
entity.setRoundNo(registerReq.getRoundNo() != null ? registerReq.getRoundNo() : 1L);
entity.setMemo(registerReq.getMemo());
entity.setStatus("READY");
entity.setStatus(LearnDataRegister.READY.getId());
entity.setDataType("CREATE");
entity.setTotalItems(0L);
entity.setTotalSize(0L);
entity.setItemCount(0L);
entity.setDeleted(false);
entity.setCreatedDttm(ZonedDateTime.now());
entity.setId1(0L); // 임시값
// entity.setId1(0L); // 임시값
DatasetEntity savedEntity = datasetRepository.save(entity);
// 저장 후 id1을 dataset_uid와 동일하게 업데이트
savedEntity.setId1(savedEntity.getId());
savedEntity = datasetRepository.save(savedEntity);
// savedEntity.setId1(savedEntity.getId());
// savedEntity = datasetRepository.save(savedEntity);
return savedEntity.toDto();
}
@@ -178,99 +186,64 @@ public class DatasetCoreService
summaryReq.getDatasetIds().size(), totalMapSheets, totalFileSize, averageMapSheets);
}
/**
* 활성 데이터셋 전체 조회 (학습 관리용)
*
* @return 데이터셋 정보 목록
*/
public List<ModelMngDto.DatasetInfo> findAllActiveDatasetsForTraining() {
List<DatasetEntity> entities = datasetRepository.findByDeletedOrderByCreatedDttmDesc(false);
public Page<Basic> searchDatasetObjectList(SearchReq searchReq) {
Page<DatasetObjEntity> entityPage = datasetObjRepository.searchDatasetObjectList(searchReq);
return entityPage.map(DatasetObjEntity::toDto);
}
return entities.stream()
.map(
entity -> {
// totalSize를 읽기 쉬운 형식으로 변환
String totalSizeStr = formatFileSize(entity.getTotalSize());
// classCounts JSON 파싱
Map<String, Integer> classCounts = entity.getClassCounts();
return ModelMngDto.DatasetInfo.builder()
.id(entity.getId())
.title(entity.getTitle())
.groupTitle(entity.getGroupTitle())
.totalItems(entity.getTotalItems())
.totalSize(totalSizeStr)
.classCounts(classCounts)
.memo(entity.getMemo())
.createdDttm(entity.getCreatedDttm())
.build();
})
.toList();
public UUID deleteDatasetObjByUuid(UUID uuid) {
DatasetObjEntity entity =
datasetObjRepository
.findByUuid(uuid)
.orElseThrow(() -> new NotFoundException("데이터셋 obj를 찾을 수 없습니다. ID: " + uuid));
entity.setDeleted(true);
datasetObjRepository.save(entity);
return entity.getUuid();
}
/**
* JSON 문자열을 Map으로 파싱
* 데이터셋 object class 조회
*
* @param jsonStr JSON 문자열
* @return 클래스별 카운트 맵
* @param uuid dataset uuid
* @param type compare, target
* @return
*/
private Map<String, Integer> parseClassCounts(String jsonStr) {
if (jsonStr == null || jsonStr.trim().isEmpty()) {
return new HashMap<>();
}
try {
return objectMapper.readValue(jsonStr, new TypeReference<Map<String, Integer>>() {});
} catch (Exception e) {
log.warn("클래스 통계 JSON 파싱 실패: {}", jsonStr, e);
return new HashMap<>();
}
public List<DatasetClass> findDatasetObjClassByUuid(UUID uuid, String type) {
return datasetObjRepository.findDatasetObjClassByUuid(uuid, type);
}
/**
* 데이터셋의 클래스 통계 계산 및 저장
*
* @param datasetId 데이터셋 ID
* @param classCounts 클래스별 카운트
*/
public void updateClassCounts(Long datasetId, Map<String, Integer> classCounts) {
public Long getDatasetMaxStage(int compareYyyy, int targetYyyy) {
return datasetRepository.getDatasetMaxStage(compareYyyy, targetYyyy);
}
@Transactional
public Long insertDatasetMngData(DatasetMngRegDto mngRegDto) {
return datasetRepository.insertDatasetMngData(mngRegDto);
}
@Transactional
public void insertDatasetObj(DatasetObjRegDto objRegDto) {
datasetObjRepository.insertDatasetObj(objRegDto);
}
public String getFilePathByUUIDPathType(UUID uuid, String pathType) {
return datasetObjRepository.getFilePathByUUIDPathType(uuid, pathType);
}
public void insertDatasetTestObj(DatasetObjRegDto objRegDto) {
datasetObjRepository.insertDatasetTestObj(objRegDto);
}
public void updateDatasetUploadStatus(Long datasetUid) {
DatasetEntity entity =
datasetRepository
.findById(datasetId)
.orElseThrow(() -> new NotFoundException("데이터셋을 찾을 수 없습니다. ID: " + datasetId));
.findById(datasetUid)
.orElseThrow(() -> new NotFoundException("데이터셋을 찾을 수 없습니다. ID: " + datasetUid));
try {
entity.setClassCounts(classCounts);
datasetRepository.save(entity);
log.info("데이터셋 클래스 통계 업데이트 완료: datasetId={}, classes={}", datasetId, classCounts.keySet());
} catch (Exception e) {
log.error("클래스 통계 JSON 변환 실패: datasetId={}", datasetId, e);
}
entity.setStatus(LearnDataRegister.COMPLETED.getId());
}
/**
* 파일 크기를 읽기 쉬운 형식으로 변환
*
* @param size 바이트 단위 크기
* @return 형식화된 문자열 (예: "1.5GB")
*/
private String formatFileSize(Long size) {
if (size == null || size == 0) {
return "0 GB";
}
double gb = size / (1024.0 * 1024.0 * 1024.0);
if (gb >= 1.0) {
return String.format("%.2f GB", gb);
}
double mb = size / (1024.0 * 1024.0);
if (mb >= 1.0) {
return String.format("%.2f MB", mb);
}
double kb = size / 1024.0;
return String.format("%.2f KB", kb);
public void insertDatasetValObj(DatasetObjRegDto objRegDto) {
datasetObjRepository.insertDatasetValObj(objRegDto);
}
}

View File

@@ -1,8 +1,12 @@
package com.kamco.cd.training.postgres.core;
import com.kamco.cd.training.common.dto.HyperParam;
import com.kamco.cd.training.common.enums.ModelType;
import com.kamco.cd.training.common.exception.CustomApiException;
import com.kamco.cd.training.common.utils.UserUtil;
import com.kamco.cd.training.hyperparam.dto.HyperParamDto;
import com.kamco.cd.training.hyperparam.dto.HyperParamDto.Basic;
import com.kamco.cd.training.hyperparam.dto.HyperParamDto.SearchReq;
import com.kamco.cd.training.postgres.entity.ModelHyperParamEntity;
import com.kamco.cd.training.postgres.repository.hyperparam.HyperParamRepository;
import java.time.ZonedDateTime;
@@ -15,17 +19,18 @@ import org.springframework.stereotype.Service;
@Service
@RequiredArgsConstructor
public class HyperParamCoreService {
private final HyperParamRepository hyperParamRepository;
private final UserUtil userUtil;
/**
* 하이퍼파라미터 등록
*
* @param createReq 등록 요청
* @param createReq ModelTrainMngDto.HyperParamDto
* @return 등록된 버전명
*/
public String createHyperParam(HyperParamDto.HyperParamCreateReq createReq) {
String firstVersion = getFirstHyperParamVersion();
public Basic createHyperParam(HyperParam createReq) {
String firstVersion = getFirstHyperParamVersion(createReq.getModel());
ModelHyperParamEntity entity = new ModelHyperParamEntity();
entity.setHyperVer(firstVersion);
@@ -36,22 +41,28 @@ public class HyperParamCoreService {
entity.setCreatedUid(userUtil.getId());
ModelHyperParamEntity resultEntity = hyperParamRepository.save(entity);
return resultEntity.getHyperVer();
Basic basic = new Basic();
basic.setUuid(resultEntity.getUuid());
basic.setHyperVer(resultEntity.getHyperVer());
return basic;
}
/**
* 하이퍼파라미터 수정
*
* @param uuid uuid
* @param uuid uuid
* @param createReq 등록 요청
* @return ver
*/
public String updateHyperParam(UUID uuid, HyperParamDto.HyperParamCreateReq createReq) {
public String updateHyperParam(UUID uuid, HyperParam createReq) {
ModelHyperParamEntity entity =
hyperParamRepository
.findHyperParamByUuid(uuid)
.orElseThrow(() -> new CustomApiException("NOT_FOUND_DATA", HttpStatus.NOT_FOUND));
hyperParamRepository
.findHyperParamByUuid(uuid)
.orElseThrow(() -> new CustomApiException("NOT_FOUND_DATA", HttpStatus.NOT_FOUND));
if (entity.getIsDefault()) {
throw new CustomApiException("UNPROCESSABLE_ENTITY_UPDATE", HttpStatus.UNPROCESSABLE_ENTITY);
}
applyHyperParam(entity, createReq);
// user
@@ -61,48 +72,6 @@ public class HyperParamCoreService {
return entity.getHyperVer();
}
private void applyHyperParam(
ModelHyperParamEntity entity, HyperParamDto.HyperParamCreateReq createReq) {
// Important
entity.setBackbone(createReq.getBackbone());
entity.setInputSize(createReq.getInputSize());
entity.setCropSize(createReq.getCropSize());
entity.setBatchSize(createReq.getBatchSize());
// Data
entity.setTrainNumWorkers(createReq.getTrainNumWorkers());
entity.setValNumWorkers(createReq.getValNumWorkers());
entity.setTestNumWorkers(createReq.getTestNumWorkers());
entity.setTrainShuffle(createReq.getTrainShuffle());
entity.setTrainPersistent(createReq.getTrainPersistent());
entity.setValPersistent(createReq.getValPersistent());
// Model Architecture
entity.setDropPathRate(createReq.getDropPathRate());
entity.setFrozenStages(createReq.getFrozenStages());
entity.setNeckPolicy(createReq.getNeckPolicy());
entity.setClassWeight(createReq.getClassWeight());
entity.setDecoderChannels(createReq.getDecoderChannels());
// Loss & Optimization
entity.setLearningRate(createReq.getLearningRate());
entity.setWeightDecay(createReq.getWeightDecay());
entity.setLayerDecayRate(createReq.getLayerDecayRate());
entity.setDdpFindUnusedParams(createReq.getDdpFindUnusedParams());
entity.setIgnoreIndex(createReq.getIgnoreIndex());
entity.setNumLayers(createReq.getNumLayers());
// Evaluation
entity.setMetrics(createReq.getMetrics());
entity.setSaveBest(createReq.getSaveBest());
entity.setSaveBestRule(createReq.getSaveBestRule());
entity.setValInterval(createReq.getValInterval());
entity.setLogInterval(createReq.getLogInterval());
entity.setVisInterval(createReq.getVisInterval());
// memo
entity.setMemo(createReq.getMemo());
}
/**
* 하이퍼파라미터 삭제
@@ -111,15 +80,20 @@ public class HyperParamCoreService {
*/
public void deleteHyperParam(UUID uuid) {
ModelHyperParamEntity entity =
hyperParamRepository
.findHyperParamByUuid(uuid)
.orElseThrow(() -> new CustomApiException("NOT_FOUND_DATA", HttpStatus.NOT_FOUND));
hyperParamRepository
.findHyperParamByUuid(uuid)
.orElseThrow(() -> new CustomApiException("NOT_FOUND_DATA", HttpStatus.NOT_FOUND));
if (entity.getHyperVer().equals("HPs_0001")) {
throw new CustomApiException("CONFLICT", HttpStatus.CONFLICT, "HPs_0001 버전은 삭제할수 없습니다.");
// if (entity.getHyperVer().equals("HPs_0001")) {
// throw new CustomApiException("UNPROCESSABLE_ENTITY", HttpStatus.UNPROCESSABLE_ENTITY);
// }
//디폴트면 삭제불가
if (entity.getIsDefault()) {
throw new CustomApiException("UNPROCESSABLE_ENTITY", HttpStatus.UNPROCESSABLE_ENTITY);
}
entity.setDelYn("Y");
entity.setDelYn(true);
entity.setUpdatedUid(userUtil.getId());
entity.setUpdatedDttm(ZonedDateTime.now());
}
@@ -129,9 +103,11 @@ public class HyperParamCoreService {
*
* @return
*/
public HyperParamDto.Basic getInitHyperParam() {
ModelHyperParamEntity entity = new ModelHyperParamEntity();
entity.setUuid(null);
public HyperParamDto.Basic getInitHyperParam(ModelType model) {
ModelHyperParamEntity entity =
hyperParamRepository
.getHyperparamByType(model)
.orElseThrow(() -> new CustomApiException("NOT_FOUND_DATA", HttpStatus.NOT_FOUND));
return entity.toDto();
}
@@ -142,38 +118,101 @@ public class HyperParamCoreService {
*/
public HyperParamDto.Basic getHyperParam(UUID uuid) {
ModelHyperParamEntity entity =
hyperParamRepository
.findHyperParamByUuid(uuid)
.orElseThrow(() -> new CustomApiException("NOT_FOUND_DATA", HttpStatus.NOT_FOUND));
hyperParamRepository
.findHyperParamByUuid(uuid)
.orElseThrow(() -> new CustomApiException("NOT_FOUND_DATA", HttpStatus.NOT_FOUND));
return entity.toDto();
}
/**
* 하이퍼파라미터 목록 조회
*
* @param model
* @param req
* @return
*/
public Page<HyperParamDto.List> findByHyperVerList(HyperParamDto.SearchReq req) {
return hyperParamRepository.findByHyperVerList(req);
public Page<HyperParamDto.List> findByHyperVerList(ModelType model, SearchReq req) {
return hyperParamRepository.findByHyperVerList(model, req);
}
/**
* 하이퍼파라미터 버전 조회
*
* @param model 모델 타입
* @return ver
*/
public String getFirstHyperParamVersion() {
public String getFirstHyperParamVersion(ModelType model) {
return hyperParamRepository
.findHyperParamVer()
.map(ModelHyperParamEntity::getHyperVer)
.map(this::increase)
.orElse("HPs_0001");
.findHyperParamVerByModelType(model)
.map(ModelHyperParamEntity::getHyperVer)
.map(ver -> increase(ver, model))
.orElse(model.name() + "_000001");
}
private String increase(String hyperVer) {
String prefix = "HPs_";
/**
* 하이퍼 파라미터의 버전을 증가시킨다.
*
* @param hyperVer 현재 버전
* @param modelType 모델 타입
* @return 증가된 버전
*/
private String increase(String hyperVer, ModelType modelType) {
String prefix = modelType.name() + "_";
int num = Integer.parseInt(hyperVer.substring(prefix.length()));
return prefix + String.format("%04d", num + 1);
return prefix + String.format("%06d", num + 1);
}
private void applyHyperParam(ModelHyperParamEntity entity, HyperParam src) {
ModelType model = src.getModel();
// 하드코딩 모델별로 다른경우 250212 bbn 하드코딩
if (model == ModelType.G3) {
entity.setCropSize("512,512");
} else {
entity.setCropSize("256,256");
}
// entity.setCropSize(src.getCropSize());
// Important
entity.setModelType(model); // 20250212 modeltype추가
entity.setBackbone(src.getBackbone());
entity.setInputSize(src.getInputSize());
entity.setBatchSize(src.getBatchSize());
// Data
entity.setTrainNumWorkers(src.getTrainNumWorkers());
entity.setValNumWorkers(src.getValNumWorkers());
entity.setTestNumWorkers(src.getTestNumWorkers());
entity.setTrainShuffle(src.getTrainShuffle());
entity.setTrainPersistent(src.getTrainPersistent());
entity.setValPersistent(src.getValPersistent());
// Model Architecture
entity.setDropPathRate(src.getDropPathRate());
entity.setFrozenStages(src.getFrozenStages());
entity.setNeckPolicy(src.getNeckPolicy());
entity.setClassWeight(src.getClassWeight());
entity.setDecoderChannels(src.getDecoderChannels());
// Loss & Optimization
entity.setLearningRate(src.getLearningRate());
entity.setWeightDecay(src.getWeightDecay());
entity.setLayerDecayRate(src.getLayerDecayRate());
entity.setDdpFindUnusedParams(src.getDdpFindUnusedParams());
entity.setIgnoreIndex(src.getIgnoreIndex());
entity.setNumLayers(src.getNumLayers());
// Evaluation
entity.setMetrics(src.getMetrics());
entity.setSaveBest(src.getSaveBest());
entity.setSaveBestRule(src.getSaveBestRule());
entity.setValInterval(src.getValInterval());
entity.setLogInterval(src.getLogInterval());
entity.setVisInterval(src.getVisInterval());
// memo
entity.setMemo(src.getMemo());
}
}

View File

@@ -1,197 +0,0 @@
package com.kamco.cd.training.postgres.core;
import com.kamco.cd.training.common.exception.BadRequestException;
import com.kamco.cd.training.common.exception.NotFoundException;
import com.kamco.cd.training.model.dto.ModelMngDto;
import com.kamco.cd.training.model.dto.ModelMngDto.Basic;
import com.kamco.cd.training.postgres.entity.ModelDatasetMappEntity;
import com.kamco.cd.training.postgres.entity.ModelTrainMasterEntity;
import com.kamco.cd.training.postgres.repository.model.ModelDatasetMappRepository;
import com.kamco.cd.training.postgres.repository.model.ModelMngRepository;
import java.time.ZonedDateTime;
import java.util.List;
import lombok.RequiredArgsConstructor;
import org.springframework.data.domain.Page;
import org.springframework.stereotype.Service;
@Service
@RequiredArgsConstructor
public class ModelMngCoreService {
private final ModelMngRepository modelMngRepository;
private final ModelDatasetMappRepository modelDatasetMappRepository;
/**
* 모델 목록 조회
*
* @param searchReq 검색 조건
* @return 페이징 처리된 모델 목록
*/
public Page<Basic> findByModels(ModelMngDto.SearchReq searchReq) {
Page<ModelTrainMasterEntity> entityPage = modelMngRepository.findByModels(searchReq);
return entityPage.map(ModelTrainMasterEntity::toDto);
}
/**
* 모델 상세 조회
*
* @param modelUid 모델 UID
* @return 모델 상세 정보
*/
public ModelMngDto.Detail getModelDetail(Long modelUid) {
ModelTrainMasterEntity entity =
modelMngRepository
.findById(modelUid)
.orElseThrow(() -> new NotFoundException("모델을 찾을 수 없습니다. ID: " + modelUid));
if (Boolean.TRUE.equals(entity.getDelYn())) {
throw new NotFoundException("삭제된 모델입니다. ID: " + modelUid);
}
return ModelMngDto.Detail.builder()
.uuid(entity.getUuid().toString())
.modelVer(entity.getModelVer())
.epochVer(entity.getEpochVer())
.processStep(entity.getProcessStep())
.trainStartDttm(entity.getTrainStartDttm())
.epochCnt(entity.getEpochCnt())
.datasetRatio(entity.getDatasetRatio())
.bestEpoch(entity.getBestEpoch())
.confirmedBestEpoch(entity.getConfirmedBestEpoch())
.step1EndDttm(entity.getStep1EndDttm())
.step1Duration(entity.getStep1Duration())
.step2EndDttm(entity.getStep2EndDttm())
.step2Duration(entity.getStep2Duration())
.progressRate(entity.getProgressRate())
.createdDttm(entity.getCreatedDttm())
.updatedDttm(entity.getUpdatedDttm())
.modelPath(entity.getModelPath())
.errorMsg(entity.getErrorMsg())
.build();
}
/**
* 모델 상세 조회 (UUID 기반)
*
* @param uuid 모델 UUID
* @return 모델 상세 정보
*/
public ModelMngDto.Detail getModelDetailByUuid(String uuid) {
ModelTrainMasterEntity entity = findByUuid(uuid);
return getModelDetail(entity.getId());
}
/**
* 학습 모델 전체 목록 조회 (삭제되지 않은 것만)
*
* @return 학습 모델 목록
*/
public List<ModelMngDto.TrainListRes> findAllTrainModels() {
List<ModelTrainMasterEntity> entities =
modelMngRepository.findByDelYnOrderByCreatedDttmDesc(false);
return entities.stream()
.map(
entity ->
ModelMngDto.TrainListRes.builder()
.uuid(entity.getUuid().toString())
.modelVer(entity.getModelVer())
.processStep(entity.getProcessStep())
.trainStartDttm(entity.getTrainStartDttm())
.progressRate(entity.getProgressRate())
.epochCnt(entity.getEpochCnt())
.step1EndDttm(entity.getStep1EndDttm())
.step1Duration(entity.getStep1Duration())
.step2EndDttm(entity.getStep2EndDttm())
.step2Duration(entity.getStep2Duration())
.createdDttm(entity.getCreatedDttm())
.errorMsg(entity.getErrorMsg())
.canResume(entity.getCanResume())
.lastCheckpointEpoch(entity.getLastCheckpointEpoch())
.build())
.toList();
}
/**
* 현재 실행 중인 모델 확인
*
* @return 실행 중인 모델 UUID (없으면 null)
*/
public String findRunningModelUuid() {
return modelMngRepository
.findFirstByStatusCdAndDelYn("RUNNING", false)
.map(entity -> entity.getUuid().toString())
.orElse(null);
}
/**
* 학습 마스터 생성
*
* @param trainReq 학습 시작 요청
* @return 생성된 모델 Entity
*/
public ModelTrainMasterEntity createTrainMaster(ModelMngDto.TrainStartReq trainReq) {
ModelTrainMasterEntity entity = new ModelTrainMasterEntity();
entity.setModelVer(trainReq.getHyperVer());
entity.setEpochVer(String.valueOf(trainReq.getEpoch()));
entity.setProcessStep("STEP1");
entity.setTrainStartDttm(ZonedDateTime.now());
entity.setEpochCnt(trainReq.getEpoch());
entity.setDatasetRatio(trainReq.getDatasetRatio());
entity.setDelYn(false);
entity.setCreatedDttm(ZonedDateTime.now());
entity.setProgressRate(0);
return modelMngRepository.save(entity);
}
/**
* 데이터셋 매핑 생성
*
* @param modelUid 모델 UID
* @param datasetIds 데이터셋 ID 목록
*/
public void createDatasetMappings(Long modelUid, List<Long> datasetIds) {
for (Long datasetId : datasetIds) {
ModelDatasetMappEntity mapping = new ModelDatasetMappEntity();
mapping.setModelUid(modelUid);
mapping.setDatasetUid(datasetId);
mapping.setDatasetType("TRAIN");
modelDatasetMappRepository.save(mapping);
}
}
/**
* UUID로 모델 조회
*
* @param uuid UUID
* @return 모델 Entity
*/
public ModelTrainMasterEntity findByUuid(String uuid) {
try {
java.util.UUID uuidObj = java.util.UUID.fromString(uuid);
return modelMngRepository
.findByUuid(uuidObj)
.orElseThrow(() -> new NotFoundException("모델을 찾을 수 없습니다. UUID: " + uuid));
} catch (IllegalArgumentException e) {
throw new BadRequestException("잘못된 UUID 형식입니다: " + uuid);
}
}
/**
* 모델 삭제 (논리 삭제)
*
* @param uuid UUID
*/
public void deleteByUuid(String uuid) {
ModelTrainMasterEntity entity = findByUuid(uuid);
// 진행 중인 모델은 삭제 불가
// if ("RUNNING".equals(entity.getStatusCd())) {
// throw new BadRequestException("진행 중인 모델은 삭제할 수 없습니다.");
// }
entity.setDelYn(true);
entity.setUpdatedDttm(ZonedDateTime.now());
modelMngRepository.save(entity);
}
}

View File

@@ -0,0 +1,29 @@
package com.kamco.cd.training.postgres.core;
import com.kamco.cd.training.postgres.repository.train.ModelTestMetricsJobRepository;
import com.kamco.cd.training.train.dto.ModelTrainMetricsDto.ResponsePathDto;
import java.util.List;
import lombok.RequiredArgsConstructor;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
@Service
@RequiredArgsConstructor
public class ModelTestMetricsJobCoreService {
private final ModelTestMetricsJobRepository modelTestMetricsJobRepository;
@Transactional
public void updateModelMetricsTrainSaveYn(Long modelId, String stepNo) {
modelTestMetricsJobRepository.updateModelMetricsTrainSaveYn(modelId, stepNo);
}
// Test 로직 시작
public List<ResponsePathDto> getTestMetricSaveNotYetModelIds() {
return modelTestMetricsJobRepository.getTestMetricSaveNotYetModelIds();
}
public void insertModelMetricsTest(List<Object[]> batchArgs) {
modelTestMetricsJobRepository.insertModelMetricsTest(batchArgs);
}
}

View File

@@ -0,0 +1,100 @@
package com.kamco.cd.training.postgres.core;
import com.kamco.cd.training.common.exception.BadRequestException;
import com.kamco.cd.training.common.exception.NotFoundException;
import com.kamco.cd.training.common.utils.UserUtil;
import com.kamco.cd.training.model.dto.ModelConfigDto;
import com.kamco.cd.training.model.dto.ModelTrainDetailDto.DetailSummary;
import com.kamco.cd.training.model.dto.ModelTrainDetailDto.HyperSummary;
import com.kamco.cd.training.model.dto.ModelTrainDetailDto.MappingDataset;
import com.kamco.cd.training.model.dto.ModelTrainDetailDto.ModelBestEpoch;
import com.kamco.cd.training.model.dto.ModelTrainDetailDto.ModelTestMetrics;
import com.kamco.cd.training.model.dto.ModelTrainDetailDto.ModelTrainMetrics;
import com.kamco.cd.training.model.dto.ModelTrainDetailDto.ModelValidationMetrics;
import com.kamco.cd.training.model.dto.ModelTrainDetailDto.TransferHyperSummary;
import com.kamco.cd.training.model.dto.ModelTrainMngDto.Basic;
import com.kamco.cd.training.postgres.entity.ModelMasterEntity;
import com.kamco.cd.training.postgres.repository.model.ModelConfigRepository;
import com.kamco.cd.training.postgres.repository.model.ModelDetailRepository;
import java.util.List;
import java.util.UUID;
import lombok.RequiredArgsConstructor;
import org.springframework.stereotype.Service;
@Service
@RequiredArgsConstructor
public class ModelTrainDetailCoreService {
private final ModelDetailRepository modelDetailRepository;
private final ModelConfigRepository modelConfigRepository;
private final UserUtil userUtil;
/**
* UUID로 모델 조회
*
* @param uuid UUID
* @return 모델 Entity
*/
public ModelMasterEntity findByUuid(String uuid) {
try {
UUID uuidObj = UUID.fromString(uuid);
return modelDetailRepository
.findByUuid(uuidObj)
.orElseThrow(() -> new NotFoundException("모델을 찾을 수 없습니다. UUID: " + uuid));
} catch (IllegalArgumentException e) {
throw new BadRequestException("잘못된 UUID 형식입니다: " + uuid);
}
}
/**
* 상세정보 페이지 > 요약정보
*
* @param uuid
* @return
*/
public DetailSummary getModelDetailSummary(UUID uuid) {
return modelDetailRepository.getModelDetailSummary(uuid);
}
public HyperSummary getByModelHyperParamSummary(UUID uuid) {
return modelDetailRepository.getByModelHyperParamSummary(uuid);
}
public TransferHyperSummary getTransferHyperSummary(UUID uuid) {
return modelDetailRepository.getByModelTransferHyperParamSummary(uuid);
}
public List<MappingDataset> getByModelMappingDataset(UUID uuid) {
return modelDetailRepository.getByModelMappingDataset(uuid);
}
public Basic findByModelByUUID(UUID uuid) {
ModelMasterEntity entity = modelDetailRepository.findByModelByUUID(uuid);
return entity.toDto();
}
/**
* 모델 학습별 config 정보 조회
*
* @param modelId
* @return
*/
public ModelConfigDto.Basic findModelConfig(Long modelId) {
return modelConfigRepository.findModelConfigByModelId(modelId).orElse(null);
}
public List<ModelTrainMetrics> getModelTrainMetricResult(UUID uuid) {
return modelDetailRepository.getModelTrainMetricResult(uuid);
}
public List<ModelValidationMetrics> getModelValidationMetricResult(UUID uuid) {
return modelDetailRepository.getModelValidationMetricResult(uuid);
}
public List<ModelTestMetrics> getModelTestMetricResult(UUID uuid) {
return modelDetailRepository.getModelTestMetricResult(uuid);
}
public ModelBestEpoch getModelTrainBestEpoch(UUID uuid) {
return modelDetailRepository.getModelTrainBestEpoch(uuid);
}
}

View File

@@ -0,0 +1,107 @@
package com.kamco.cd.training.postgres.core;
import com.kamco.cd.training.postgres.entity.ModelTrainJobEntity;
import com.kamco.cd.training.postgres.repository.train.ModelTrainJobRepository;
import com.kamco.cd.training.train.dto.ModelTrainJobDto;
import java.time.ZonedDateTime;
import java.util.Map;
import java.util.Optional;
import lombok.RequiredArgsConstructor;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
@Service
@RequiredArgsConstructor
@Transactional(readOnly = true)
public class ModelTrainJobCoreService {
private final ModelTrainJobRepository modelTrainJobRepository;
public int findMaxAttemptNo(Long modelId) {
return modelTrainJobRepository.findMaxAttemptNo(modelId);
}
public Optional<ModelTrainJobDto> findLatestByModelId(Long modelId) {
return modelTrainJobRepository.findLatestByModelId(modelId).map(ModelTrainJobEntity::toDto);
}
public Optional<ModelTrainJobDto> findById(Long jobId) {
return modelTrainJobRepository.findById(jobId).map(ModelTrainJobEntity::toDto);
}
/** QUEUED Job 생성 */
@Transactional
public Long createQueuedJob(
Long modelId, int attemptNo, Map<String, Object> paramsJson, ZonedDateTime queuedDttm) {
ModelTrainJobEntity job = new ModelTrainJobEntity();
job.setModelId(modelId);
job.setAttemptNo(attemptNo);
job.setStatusCd("QUEUED");
job.setParamsJson(paramsJson);
job.setQueuedDttm(queuedDttm != null ? queuedDttm : ZonedDateTime.now());
modelTrainJobRepository.save(job);
return job.getId();
}
/** 실행 시작 처리 */
@Transactional
public void markRunning(
Long jobId, String containerName, String logPath, String lockedBy, Integer totalEpoch) {
ModelTrainJobEntity job =
modelTrainJobRepository
.findById(jobId)
.orElseThrow(() -> new IllegalArgumentException("Job not found: " + jobId));
job.setStatusCd("RUNNING");
job.setContainerName(containerName);
job.setLogPath(logPath);
job.setStartedDttm(ZonedDateTime.now());
job.setLockedDttm(ZonedDateTime.now());
job.setLockedBy(lockedBy);
if (totalEpoch != null) {
job.setTotalEpoch(totalEpoch);
}
}
/** 성공 처리 */
@Transactional
public void markSuccess(Long jobId, int exitCode) {
ModelTrainJobEntity job =
modelTrainJobRepository
.findById(jobId)
.orElseThrow(() -> new IllegalArgumentException("Job not found: " + jobId));
job.setStatusCd("SUCCESS");
job.setExitCode(exitCode);
job.setFinishedDttm(ZonedDateTime.now());
}
/** 실패 처리 */
@Transactional
public void markFailed(Long jobId, Integer exitCode, String errorMessage) {
ModelTrainJobEntity job =
modelTrainJobRepository
.findById(jobId)
.orElseThrow(() -> new IllegalArgumentException("Job not found: " + jobId));
job.setStatusCd("FAILED");
job.setExitCode(exitCode);
job.setErrorMessage(errorMessage);
job.setFinishedDttm(ZonedDateTime.now());
}
/** 취소 처리 */
@Transactional
public void markCanceled(Long jobId) {
ModelTrainJobEntity job =
modelTrainJobRepository
.findById(jobId)
.orElseThrow(() -> new IllegalArgumentException("Job not found: " + jobId));
job.setStatusCd("STOPPED");
job.setFinishedDttm(ZonedDateTime.now());
}
}

View File

@@ -0,0 +1,32 @@
package com.kamco.cd.training.postgres.core;
import com.kamco.cd.training.postgres.repository.train.ModelTrainMetricsJobRepository;
import com.kamco.cd.training.train.dto.ModelTrainMetricsDto.ResponsePathDto;
import java.util.List;
import lombok.RequiredArgsConstructor;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
@Service
@RequiredArgsConstructor
public class ModelTrainMetricsJobCoreService {
private final ModelTrainMetricsJobRepository modelTrainMetricsJobRepository;
public List<ResponsePathDto> getTrainMetricSaveNotYetModelIds() {
return modelTrainMetricsJobRepository.getTrainMetricSaveNotYetModelIds();
}
public void insertModelMetricsTrain(List<Object[]> batchArgs) {
modelTrainMetricsJobRepository.insertModelMetricsTrain(batchArgs);
}
@Transactional
public void updateModelMetricsTrainSaveYn(Long modelId, String stepNo) {
modelTrainMetricsJobRepository.updateModelMetricsTrainSaveYn(modelId, stepNo);
}
public void insertModelMetricsValidation(List<Object[]> batchArgs) {
modelTrainMetricsJobRepository.insertModelMetricsValidation(batchArgs);
}
}

View File

@@ -0,0 +1,534 @@
package com.kamco.cd.training.postgres.core;
import com.kamco.cd.training.common.enums.HyperParamSelectType;
import com.kamco.cd.training.common.enums.ModelType;
import com.kamco.cd.training.common.enums.TrainStatusType;
import com.kamco.cd.training.common.exception.BadRequestException;
import com.kamco.cd.training.common.exception.CustomApiException;
import com.kamco.cd.training.common.utils.UserUtil;
import com.kamco.cd.training.dataset.dto.DatasetDto.DatasetReq;
import com.kamco.cd.training.dataset.dto.DatasetDto.SelectDataSet;
import com.kamco.cd.training.model.dto.ModelConfigDto;
import com.kamco.cd.training.model.dto.ModelTrainMngDto;
import com.kamco.cd.training.model.dto.ModelTrainMngDto.Basic;
import com.kamco.cd.training.model.dto.ModelTrainMngDto.TrainingDataset;
import com.kamco.cd.training.postgres.entity.ModelConfigEntity;
import com.kamco.cd.training.postgres.entity.ModelDatasetEntity;
import com.kamco.cd.training.postgres.entity.ModelDatasetMappEntity;
import com.kamco.cd.training.postgres.entity.ModelHyperParamEntity;
import com.kamco.cd.training.postgres.entity.ModelMasterEntity;
import com.kamco.cd.training.postgres.repository.dataset.DatasetRepository;
import com.kamco.cd.training.postgres.repository.hyperparam.HyperParamRepository;
import com.kamco.cd.training.postgres.repository.model.ModelConfigRepository;
import com.kamco.cd.training.postgres.repository.model.ModelDatasetMappRepository;
import com.kamco.cd.training.postgres.repository.model.ModelDatasetRepository;
import com.kamco.cd.training.postgres.repository.model.ModelMngRepository;
import com.kamco.cd.training.train.dto.TrainRunRequest;
import java.time.ZonedDateTime;
import java.util.ArrayList;
import java.util.List;
import java.util.UUID;
import lombok.RequiredArgsConstructor;
import org.springframework.data.domain.Page;
import org.springframework.http.HttpStatus;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Propagation;
import org.springframework.transaction.annotation.Transactional;
@Service
@RequiredArgsConstructor
public class ModelTrainMngCoreService {
private final ModelMngRepository modelMngRepository;
private final ModelDatasetRepository modelDatasetRepository;
private final ModelDatasetMappRepository modelDatasetMapRepository;
private final ModelConfigRepository modelConfigRepository;
private final HyperParamRepository hyperParamRepository;
private final DatasetRepository datasetRepository;
private final UserUtil userUtil;
/**
* 모델 목록 조회
*
* @param searchReq 검색 조건
* @return 페이징 처리된 모델 목록
*/
public Page<Basic> findByModelList(ModelTrainMngDto.SearchReq searchReq) {
Page<ModelMasterEntity> entityPage = modelMngRepository.findByModels(searchReq);
return entityPage.map(ModelMasterEntity::toDto);
}
/**
* 학습모델 삭제
*
* @param uuid
*/
public void deleteModel(UUID uuid) {
ModelMasterEntity entity =
modelMngRepository
.findByUuid(uuid)
.orElseThrow(() -> new CustomApiException("NOT_FOUND_DATA", HttpStatus.NOT_FOUND));
entity.setDelYn(true);
entity.setUpdatedDttm(ZonedDateTime.now());
entity.setUpdatedUid(userUtil.getId());
}
/**
* 모델학습 저장
*
* @param addReq
* @return
*/
public ModelTrainMngDto.Basic saveModel(ModelTrainMngDto.AddReq addReq) {
ModelMasterEntity entity = new ModelMasterEntity();
ModelHyperParamEntity hyperParamEntity = new ModelHyperParamEntity();
// 최적화 파라미터는 모델 type의 디폴트사용
if (HyperParamSelectType.OPTIMIZED.getId().equals(addReq.getHyperParamType())) {
ModelType modelType = ModelType.getValueData(addReq.getModelNo());
hyperParamEntity = hyperParamRepository.getHyperparamByType(modelType).orElse(null);
// hyperParamEntity = hyperParamRepository.findByHyperVer("HPs_0001").orElse(null);
} else {
hyperParamEntity =
hyperParamRepository.findHyperParamByUuid(addReq.getHyperUuid()).orElse(null);
}
if (hyperParamEntity == null || hyperParamEntity.getHyperVer() == null) {
throw new CustomApiException("NOT_FOUND_DATA", HttpStatus.NOT_FOUND);
}
String modelVer =
String.join(
".", addReq.getModelNo(), hyperParamEntity.getHyperVer(), entity.getUuid().toString());
entity.setModelVer(modelVer);
entity.setHyperParamId(hyperParamEntity.getId());
entity.setModelNo(addReq.getModelNo());
entity.setTrainType(addReq.getTrainType()); // 일반, 전이
entity.setBeforeModelId(addReq.getBeforeModelId());
if (addReq.getIsStart()) {
entity.setModelStep((short) 1);
entity.setStatusCd(TrainStatusType.IN_PROGRESS.getId());
entity.setStrtDttm(ZonedDateTime.now());
entity.setStep1StrtDttm(ZonedDateTime.now());
entity.setStep1State(TrainStatusType.IN_PROGRESS.getId());
} else {
entity.setStatusCd(TrainStatusType.READY.getId());
entity.setStep1State(TrainStatusType.READY.getId());
}
entity.setCreatedUid(userUtil.getId());
ModelMasterEntity resultEntity = modelMngRepository.save(entity);
ModelTrainMngDto.Basic result = new ModelTrainMngDto.Basic();
result.setId(resultEntity.getId());
result.setUuid(resultEntity.getUuid());
return result;
}
/**
* data set 저장
*
* @param modelId 저장한 모델 학습 id
* @param addReq 요청 파라미터
*/
public void saveModelDataset(Long modelId, ModelTrainMngDto.AddReq addReq) {
TrainingDataset dataset = addReq.getTrainingDataset();
ModelMasterEntity modelMasterEntity = new ModelMasterEntity();
ModelDatasetEntity datasetEntity = new ModelDatasetEntity();
modelMasterEntity.setId(modelId);
datasetEntity.setModel(modelMasterEntity);
if (addReq.getModelNo().equals(ModelType.G1.getId())) {
datasetEntity.setBuildingCnt(dataset.getSummary().getBuildingCnt());
datasetEntity.setContainerCnt(dataset.getSummary().getContainerCnt());
} else if (addReq.getModelNo().equals(ModelType.G2.getId())) {
datasetEntity.setWasteCnt(dataset.getSummary().getWasteCnt());
} else if (addReq.getModelNo().equals(ModelType.G3.getId())) {
datasetEntity.setLandCoverCnt(dataset.getSummary().getLandCoverCnt());
}
datasetEntity.setCreatedUid(userUtil.getId());
// data set 저장
modelDatasetRepository.save(datasetEntity);
}
/**
* 학습모델 수정
*
* @param modelId
* @param req
*/
public void updateModelMaster(Long modelId, ModelTrainMngDto.UpdateReq req) {
ModelMasterEntity entity =
modelMngRepository
.findById(modelId)
.orElseThrow(() -> new CustomApiException("NOT_FOUND_DATA", HttpStatus.NOT_FOUND));
if (req.getRequestPath() != null && !req.getRequestPath().isEmpty()) {
entity.setRequestPath(req.getRequestPath());
}
if (req.getResponsePath() != null && !req.getResponsePath().isEmpty()) {
entity.setRequestPath(req.getResponsePath());
}
}
/**
* 모델 데이터셋 mapping 테이블 저장
*
* @param modelId 모델학습 id
* @param datasetList 선택한 data set
*/
public void saveModelDatasetMap(Long modelId, List<Long> datasetList) {
for (Long datasetId : datasetList) {
ModelDatasetMappEntity mapEntity = new ModelDatasetMappEntity();
mapEntity.setModelUid(modelId);
mapEntity.setDatasetUid(datasetId);
modelDatasetMapRepository.save(mapEntity);
}
}
/**
* 모델학습 config 저장
*
* @param modelId 모델학습 id
* @param req 요청 파라미터
* @return
*/
public void saveModelConfig(Long modelId, ModelTrainMngDto.ModelConfig req) {
ModelMasterEntity modelMasterEntity = new ModelMasterEntity();
ModelConfigEntity entity = new ModelConfigEntity();
modelMasterEntity.setId(modelId);
entity.setModel(modelMasterEntity);
entity.setEpochCount(req.getEpochCnt());
entity.setTrainPercent(req.getTrainingCnt());
entity.setValidationPercent(req.getValidationCnt());
entity.setTestPercent(req.getTestCnt());
entity.setMemo(req.getMemo());
modelConfigRepository.save(entity);
}
/**
* 데이터셋 매핑 생성
*
* @param modelUid 모델 UID
* @param datasetIds 데이터셋 ID 목록
*/
public void createDatasetMappings(Long modelUid, List<Long> datasetIds) {
for (Long datasetId : datasetIds) {
ModelDatasetMappEntity mapping = new ModelDatasetMappEntity();
mapping.setModelUid(modelUid);
mapping.setDatasetUid(datasetId);
mapping.setDatasetType("TRAIN");
modelDatasetMapRepository.save(mapping);
}
}
/**
* UUID로 모델 조회
*
* @param uuid UUID
* @return 모델 Entity
*/
public ModelMasterEntity findByUuid(UUID uuid) {
try {
return modelMngRepository
.findByUuid(uuid)
.orElseThrow(() -> new CustomApiException("NOT_FOUND_DATA", HttpStatus.NOT_FOUND));
} catch (IllegalArgumentException e) {
throw new BadRequestException("잘못된 UUID 형식입니다: " + uuid);
}
}
/**
* uuid로 model id 조회
*
* @param uuid
* @return
*/
public Long findModelIdByUuid(UUID uuid) {
ModelMasterEntity entity =
modelMngRepository
.findByUuid(uuid)
.orElseThrow(() -> new CustomApiException("NOT_FOUND_DATA", HttpStatus.NOT_FOUND));
return entity.getId();
}
/**
* 모델학습 아이디로 config정보 조회
*
* @param uuid
* @return
*/
public ModelConfigDto.Basic findModelConfigByModelId(UUID uuid) {
ModelMasterEntity modelEntity = findByUuid(uuid);
return modelConfigRepository
.findModelConfigByModelId(modelEntity.getId())
.orElseThrow(() -> new CustomApiException("NOT_FOUND_DATA", HttpStatus.NOT_FOUND));
}
/**
* 데이터셋 G1 목록
*
* @param req
* @return
*/
public List<SelectDataSet> getDatasetSelectG1List(DatasetReq req) {
return datasetRepository.getDatasetSelectG1List(req);
}
/**
* 데이터셋 G2, G3 목록
*
* @param req
* @return
*/
public List<SelectDataSet> getDatasetSelectG2G3List(DatasetReq req) {
return datasetRepository.getDatasetSelectG2G3List(req);
}
/**
* 모델관리 조회
*
* @param id
* @return
*/
public ModelTrainMngDto.Basic findModelById(Long id) {
ModelMasterEntity entity =
modelMngRepository
.findById(id)
.orElseThrow(() -> new IllegalArgumentException("Model not found: " + id));
return entity.toDto();
}
/**
* 마스터를 IN_PROGRESS로 전환하고, 현재 실행 jobId를 연결 - UI/중단/상태조회 모두 currentAttemptId를 기준으로 동작
*/
@Transactional
public void markInProgress(Long modelId, Long jobId) {
ModelMasterEntity master =
modelMngRepository
.findById(modelId)
.orElseThrow(() -> new IllegalArgumentException("Model not found: " + modelId));
master.setStatusCd(TrainStatusType.IN_PROGRESS.getId());
master.setCurrentAttemptId(jobId);
// 필요하면 시작시간도 여기서 찍어줌
}
/**
* 마지막 에러 메시지 초기화 - 재시작/새 실행 때 이전 에러 흔적 제거
*/
@Transactional
public void clearLastError(Long modelId) {
ModelMasterEntity master =
modelMngRepository
.findById(modelId)
.orElseThrow(() -> new IllegalArgumentException("Model not found: " + modelId));
master.setLastError(null);
}
/**
* 중단 처리(옵션) - cancel에서 쓰려고 하면 같이 구현
*/
@Transactional
public void markStopped(Long modelId) {
ModelMasterEntity master =
modelMngRepository
.findById(modelId)
.orElseThrow(() -> new IllegalArgumentException("Model not found: " + modelId));
master.setStatusCd(TrainStatusType.STOPPED.getId());
}
/**
* 완료 처리(옵션) - Worker가 성공 시 호출
*/
@Transactional
public void markCompleted(Long modelId) {
ModelMasterEntity master =
modelMngRepository
.findById(modelId)
.orElseThrow(() -> new IllegalArgumentException("Model not found: " + modelId));
master.setStatusCd(TrainStatusType.COMPLETED.getId());
}
/**
* step 1오류 처리(옵션) - Worker가 실패 시 호출
*/
@Transactional
public void markError(Long modelId, String errorMessage) {
ModelMasterEntity master =
modelMngRepository
.findById(modelId)
.orElseThrow(() -> new IllegalArgumentException("Model not found: " + modelId));
master.setStatusCd(TrainStatusType.ERROR.getId());
master.setStep1State(TrainStatusType.ERROR.getId());
master.setLastError(errorMessage);
master.setUpdatedUid(userUtil.getId());
master.setUpdatedDttm(ZonedDateTime.now());
}
/**
* step 2오류 처리(옵션) - Worker가 실패 시 호출
*/
@Transactional
public void markStep2Error(Long modelId, String errorMessage) {
ModelMasterEntity master =
modelMngRepository
.findById(modelId)
.orElseThrow(() -> new IllegalArgumentException("Model not found: " + modelId));
master.setStatusCd(TrainStatusType.ERROR.getId());
master.setStep2State(TrainStatusType.ERROR.getId());
master.setLastError(errorMessage);
master.setUpdatedUid(userUtil.getId());
master.setUpdatedDttm(ZonedDateTime.now());
}
@Transactional
public void markSuccess(Long modelId) {
ModelMasterEntity master =
modelMngRepository
.findById(modelId)
.orElseThrow(() -> new IllegalArgumentException("Model not found: " + modelId));
// 모델 상태 완료 처리
master.setStatusCd(TrainStatusType.COMPLETED.getId());
// (선택) 마지막 에러 메시지 비우기
master.setLastError(null);
}
/**
* 학습 실행에 필요한 파라미터 조회
*
* @param modelId
* @return
*/
public TrainRunRequest findTrainRunRequest(Long modelId) {
return modelMngRepository.findTrainRunRequest(modelId);
}
/**
* step1 진행중 처리
*
* @param modelId
* @param jobId
*/
@Transactional(propagation = Propagation.REQUIRES_NEW)
public void markStep1InProgress(Long modelId, Long jobId) {
ModelMasterEntity entity =
modelMngRepository
.findById(modelId)
.orElseThrow(() -> new IllegalArgumentException("Model not found: " + modelId));
entity.setStatusCd(TrainStatusType.IN_PROGRESS.getId());
entity.setStep1StrtDttm(ZonedDateTime.now());
entity.setStep1State(TrainStatusType.IN_PROGRESS.getId());
entity.setCurrentAttemptId(jobId);
entity.setUpdatedDttm(ZonedDateTime.now());
entity.setUpdatedUid(userUtil.getId());
}
/**
* step2 진행중 처리
*
* @param modelId
*/
@Transactional(propagation = Propagation.REQUIRES_NEW)
public void markStep2InProgress(Long modelId, Long jobId) {
ModelMasterEntity entity =
modelMngRepository
.findById(modelId)
.orElseThrow(() -> new IllegalArgumentException("Model not found: " + modelId));
entity.setStatusCd(TrainStatusType.IN_PROGRESS.getId());
entity.setStep2StrtDttm(ZonedDateTime.now());
entity.setStep2State(TrainStatusType.IN_PROGRESS.getId());
entity.setCurrentAttemptId(jobId);
entity.setUpdatedDttm(ZonedDateTime.now());
entity.setUpdatedUid(userUtil.getId());
}
/**
* step1 완료처리
*
* @param modelId
*/
@Transactional(propagation = Propagation.REQUIRES_NEW)
public void markStep1Success(Long modelId) {
ModelMasterEntity entity =
modelMngRepository
.findById(modelId)
.orElseThrow(() -> new IllegalArgumentException("Model not found: " + modelId));
entity.setStatusCd(TrainStatusType.COMPLETED.getId());
entity.setStep1State(TrainStatusType.COMPLETED.getId());
entity.setStep1EndDttm(ZonedDateTime.now());
entity.setUpdatedDttm(ZonedDateTime.now());
entity.setUpdatedUid(userUtil.getId());
}
/**
* step2 완료처리
*
* @param modelId
*/
@Transactional(propagation = Propagation.REQUIRES_NEW)
public void markStep2Success(Long modelId) {
ModelMasterEntity entity =
modelMngRepository
.findById(modelId)
.orElseThrow(() -> new IllegalArgumentException("Model not found: " + modelId));
entity.setStatusCd(TrainStatusType.COMPLETED.getId());
entity.setStep2State(TrainStatusType.COMPLETED.getId());
entity.setStep2EndDttm(ZonedDateTime.now());
entity.setUpdatedDttm(ZonedDateTime.now());
entity.setUpdatedUid(userUtil.getId());
}
public void updateModelMasterBestEpoch(Long modelId, int epoch) {
ModelMasterEntity entity =
modelMngRepository
.findById(modelId)
.orElseThrow(() -> new IllegalArgumentException("Model not found: " + modelId));
entity.setBestEpoch(epoch);
}
/**
* 데이터셋 uid 조회
*
* @param datasetIds
* @return
*/
public List<String> findDatasetUid(List<Long> datasetIds) {
return datasetRepository.findDatasetUid(datasetIds);
}
public List<Long> findModelDatasetMapp(Long modelId) {
List<Long> datasetUids = new ArrayList<>();
List<ModelDatasetMappEntity> entities = modelDatasetMapRepository.findByModelUid(modelId);
for (ModelDatasetMappEntity entity : entities) {
datasetUids.add(entity.getDatasetUid());
}
return datasetUids;
}
public Long findModelStep1InProgressCnt() {
return modelMngRepository.findModelStep1InProgressCnt();
}
}

View File

@@ -2,6 +2,7 @@ package com.kamco.cd.training.postgres.core;
import com.kamco.cd.training.postgres.repository.upload.UploadSessionRepository;
import com.kamco.cd.training.upload.dto.UploadDto;
import java.util.UUID;
import lombok.RequiredArgsConstructor;
import org.springframework.stereotype.Service;
@@ -39,8 +40,8 @@ public class UploadSessionCoreService {
uploadSessionRepository.insertUploadSession(addReq);
}
public UploadDto.uploadDto findByDatasetUid(Long datasetId, String uploadDivi) {
return uploadSessionRepository.findByDatasetUid(datasetId, uploadDivi);
public UploadDto.uploadDto findByDatasetUid(String uploadDivi, UUID uuid) {
return uploadSessionRepository.findByDatasetUid(uploadDivi, uuid);
}
public UploadDto.uploadDto findByUuid(String uuid) {

View File

@@ -30,19 +30,14 @@ public class DatasetEntity {
@Column(name = "dataset_uid", nullable = false)
private Long id;
@Column(name = "uuid", nullable = false)
private UUID uuid;
@Column(name = "compare_yyyy", nullable = false)
private Integer compareYyyy;
@Size(max = 200)
@Column(name = "group_title", length = 200)
private String groupTitle;
@Size(max = 10)
@Column(name = "data_year", length = 10)
private String dataYear;
@Column(name = "target_yyyy", nullable = false)
private Integer targetYyyy;
@Size(max = 50)
@ColumnDefault("'CREATE'")
@ColumnDefault("'PRODUCTION'")
@Column(name = "data_type", length = 50)
private String dataType;
@@ -77,8 +72,9 @@ public class DatasetEntity {
@Column(name = "mod_dttm")
private Instant modDttm;
@Column(name = "id")
private Long id1;
@NotNull
@Column(name = "id", nullable = false)
private Long LegacyId;
@Size(max = 50)
@Column(name = "created_by", length = 50)
@@ -87,7 +83,7 @@ public class DatasetEntity {
@NotNull
@ColumnDefault("now()")
@Column(name = "created_dttm", nullable = false)
private ZonedDateTime createdDttm;
private ZonedDateTime createdDttm = ZonedDateTime.now();
@NotNull
@ColumnDefault("false")
@@ -108,25 +104,40 @@ public class DatasetEntity {
@Column(name = "updated_dttm")
private ZonedDateTime updatedDttm;
@Size(max = 4)
@Column(name = "year", length = 4)
private String year;
@NotNull
@ColumnDefault("uuid_generate_v4()")
@Column(name = "uuid", nullable = false)
private UUID uuid = UUID.randomUUID();
@Column(name = "class_counts", columnDefinition = "jsonb")
@ColumnDefault("0")
@Column(name = "total_object_count")
private Long totalObjectCount;
@Size(max = 1000)
@Column(name = "dataset_path", length = 1000)
private String datasetPath;
@Column(name = "class_counts")
@JdbcTypeCode(SqlTypes.JSON)
private Map<String, Integer> classCounts;
@Size(max = 32)
@Column(name = "uid")
private String uid;
public DatasetDto.Basic toDto() {
return new DatasetDto.Basic(
this.id,
this.uuid,
this.groupTitle,
this.title,
this.roundNo,
this.compareYyyy,
this.targetYyyy,
this.totalSize,
this.memo,
this.createdDttm,
this.status,
this.deleted);
this.deleted,
this.dataType);
}
}

View File

@@ -0,0 +1,118 @@
package com.kamco.cd.training.postgres.entity;
import com.kamco.cd.training.dataset.dto.DatasetObjDto;
import com.kamco.cd.training.dataset.dto.DatasetObjDto.Basic;
import jakarta.persistence.Column;
import jakarta.persistence.Entity;
import jakarta.persistence.GeneratedValue;
import jakarta.persistence.GenerationType;
import jakarta.persistence.Id;
import jakarta.persistence.Table;
import jakarta.validation.constraints.NotNull;
import jakarta.validation.constraints.Size;
import java.time.ZonedDateTime;
import java.util.UUID;
import lombok.Getter;
import lombok.Setter;
import org.hibernate.annotations.ColumnDefault;
import org.hibernate.annotations.JdbcTypeCode;
import org.hibernate.type.SqlTypes;
import org.locationtech.jts.geom.Geometry;
@Getter
@Setter
@Entity
@Table(name = "tb_dataset_obj")
public class DatasetObjEntity {
@Id
@GeneratedValue(strategy = GenerationType.IDENTITY)
@Column(name = "obj_id", nullable = false)
private Long objId;
@NotNull
@Column(name = "dataset_uid", nullable = false)
private Long datasetUid;
@Column(name = "target_yyyy")
private Integer targetYyyy;
@Size(max = 255)
@Column(name = "target_class_cd")
private String targetClassCd;
@Column(name = "compare_yyyy")
private Integer compareYyyy;
@Size(max = 255)
@Column(name = "compare_class_cd")
private String compareClassCd;
@Size(max = 255)
@Column(name = "target_path")
private String targetPath;
@Size(max = 255)
@Column(name = "compare_path")
private String comparePath;
@Size(max = 255)
@Column(name = "label_path")
private String labelPath;
@Size(max = 255)
@Column(name = "geojson_path")
private String geojsonPath;
@Size(max = 255)
@Column(name = "map_sheet_num")
private String mapSheetNum;
@ColumnDefault("now()")
@Column(name = "created_dttm")
private ZonedDateTime createdDttm;
@Column(name = "created_uid")
private Long createdUid;
@ColumnDefault("false")
@Column(name = "deleted")
private Boolean deleted;
@Column(name = "uuid")
private UUID uuid;
@Size(max = 32)
@Column(name = "uid")
private String uid;
@JdbcTypeCode(SqlTypes.JSON)
@Column(name = "geo_jsonb", columnDefinition = "jsonb")
private String geoJsonb;
@Column(name = "file_name")
private String fileName;
@Column(name = "geom", columnDefinition = "geometry")
private Geometry geom;
public Basic toDto() {
return new DatasetObjDto.Basic(
this.objId,
this.datasetUid,
this.targetYyyy,
this.targetClassCd,
this.compareYyyy,
this.compareClassCd,
this.targetPath,
this.comparePath,
this.labelPath,
this.geojsonPath,
this.mapSheetNum,
this.createdDttm,
this.createdUid,
this.deleted,
this.uuid,
this.geoJsonb);
}
}

View File

@@ -0,0 +1,117 @@
package com.kamco.cd.training.postgres.entity;
import com.kamco.cd.training.dataset.dto.DatasetObjDto.Basic;
import jakarta.persistence.Column;
import jakarta.persistence.Entity;
import jakarta.persistence.GeneratedValue;
import jakarta.persistence.GenerationType;
import jakarta.persistence.Id;
import jakarta.persistence.Table;
import jakarta.validation.constraints.NotNull;
import jakarta.validation.constraints.Size;
import java.time.ZonedDateTime;
import java.util.UUID;
import lombok.Getter;
import lombok.Setter;
import org.hibernate.annotations.ColumnDefault;
import org.hibernate.annotations.JdbcTypeCode;
import org.hibernate.type.SqlTypes;
import org.locationtech.jts.geom.Geometry;
@Getter
@Setter
@Entity
@Table(name = "tb_dataset_test_obj")
public class DatasetTestObjEntity {
@Id
@GeneratedValue(strategy = GenerationType.IDENTITY)
@Column(name = "obj_id", nullable = false)
private Long objId;
@NotNull
@Column(name = "dataset_uid", nullable = false)
private Long datasetUid;
@Column(name = "target_yyyy")
private Integer targetYyyy;
@Size(max = 255)
@Column(name = "target_class_cd")
private String targetClassCd;
@Column(name = "compare_yyyy")
private Integer compareYyyy;
@Size(max = 255)
@Column(name = "compare_class_cd")
private String compareClassCd;
@Size(max = 255)
@Column(name = "target_path")
private String targetPath;
@Size(max = 255)
@Column(name = "compare_path")
private String comparePath;
@Size(max = 255)
@Column(name = "label_path")
private String labelPath;
@Size(max = 255)
@Column(name = "geojson_path")
private String geojsonPath;
@Size(max = 255)
@Column(name = "map_sheet_num")
private String mapSheetNum;
@ColumnDefault("now()")
@Column(name = "created_dttm")
private ZonedDateTime createdDttm;
@Column(name = "created_uid")
private Long createdUid;
@ColumnDefault("false")
@Column(name = "deleted")
private Boolean deleted;
@Column(name = "uuid")
private UUID uuid;
@Size(max = 32)
@Column(name = "uid")
private String uid;
@JdbcTypeCode(SqlTypes.JSON)
@Column(name = "geo_jsonb", columnDefinition = "jsonb")
private String geoJsonb;
@Column(name = "file_name")
private String fileName;
@Column(name = "geom", columnDefinition = "geometry")
private Geometry geom;
public Basic toDto() {
return new Basic(
this.objId,
this.datasetUid,
this.targetYyyy,
this.targetClassCd,
this.compareYyyy,
this.compareClassCd,
this.targetPath,
this.comparePath,
this.labelPath,
this.geojsonPath,
this.mapSheetNum,
this.createdDttm,
this.createdUid,
this.deleted,
this.uuid,
this.geoJsonb);
}
}

View File

@@ -0,0 +1,117 @@
package com.kamco.cd.training.postgres.entity;
import com.kamco.cd.training.dataset.dto.DatasetObjDto.Basic;
import jakarta.persistence.Column;
import jakarta.persistence.Entity;
import jakarta.persistence.GeneratedValue;
import jakarta.persistence.GenerationType;
import jakarta.persistence.Id;
import jakarta.persistence.Table;
import jakarta.validation.constraints.NotNull;
import jakarta.validation.constraints.Size;
import java.time.ZonedDateTime;
import java.util.UUID;
import lombok.Getter;
import lombok.Setter;
import org.hibernate.annotations.ColumnDefault;
import org.hibernate.annotations.JdbcTypeCode;
import org.hibernate.type.SqlTypes;
import org.locationtech.jts.geom.Geometry;
@Getter
@Setter
@Entity
@Table(name = "tb_dataset_val_obj")
public class DatasetValObjEntity {
@Id
@GeneratedValue(strategy = GenerationType.IDENTITY)
@Column(name = "obj_id", nullable = false)
private Long objId;
@NotNull
@Column(name = "dataset_uid", nullable = false)
private Long datasetUid;
@Column(name = "target_yyyy")
private Integer targetYyyy;
@Size(max = 255)
@Column(name = "target_class_cd")
private String targetClassCd;
@Column(name = "compare_yyyy")
private Integer compareYyyy;
@Size(max = 255)
@Column(name = "compare_class_cd")
private String compareClassCd;
@Size(max = 255)
@Column(name = "target_path")
private String targetPath;
@Size(max = 255)
@Column(name = "compare_path")
private String comparePath;
@Size(max = 255)
@Column(name = "label_path")
private String labelPath;
@Size(max = 255)
@Column(name = "geojson_path")
private String geojsonPath;
@Size(max = 255)
@Column(name = "map_sheet_num")
private String mapSheetNum;
@ColumnDefault("now()")
@Column(name = "created_dttm")
private ZonedDateTime createdDttm;
@Column(name = "created_uid")
private Long createdUid;
@ColumnDefault("false")
@Column(name = "deleted")
private Boolean deleted;
@Column(name = "uuid")
private UUID uuid;
@Size(max = 32)
@Column(name = "uid")
private String uid;
@JdbcTypeCode(SqlTypes.JSON)
@Column(name = "geo_jsonb", columnDefinition = "jsonb")
private String geoJsonb;
@Column(name = "file_name")
private String fileName;
@Column(name = "geom", columnDefinition = "geometry")
private Geometry geom;
public Basic toDto() {
return new Basic(
this.objId,
this.datasetUid,
this.targetYyyy,
this.targetClassCd,
this.compareYyyy,
this.compareClassCd,
this.targetPath,
this.comparePath,
this.labelPath,
this.geojsonPath,
this.mapSheetNum,
this.createdDttm,
this.createdUid,
this.deleted,
this.uuid,
this.geoJsonb);
}
}

View File

@@ -0,0 +1,62 @@
package com.kamco.cd.training.postgres.entity;
import jakarta.persistence.Column;
import jakarta.persistence.Entity;
import jakarta.persistence.FetchType;
import jakarta.persistence.GeneratedValue;
import jakarta.persistence.GenerationType;
import jakarta.persistence.Id;
import jakarta.persistence.JoinColumn;
import jakarta.persistence.ManyToOne;
import jakarta.persistence.Table;
import jakarta.validation.constraints.NotNull;
import java.time.ZonedDateTime;
import lombok.Getter;
import lombok.Setter;
import org.hibernate.annotations.ColumnDefault;
@Getter
@Setter
@Entity
@Table(name = "tb_model_config")
public class ModelConfigEntity {
@Id
@GeneratedValue(strategy = GenerationType.IDENTITY)
@Column(name = "config_id", nullable = false)
private Long id;
@NotNull
@ManyToOne(fetch = FetchType.LAZY, optional = false)
@JoinColumn(name = "model_id", nullable = false)
private ModelMasterEntity model;
@Column(name = "epoch_count")
private Integer epochCount;
@Column(name = "train_percent")
private Float trainPercent;
@Column(name = "validation_percent")
private Float validationPercent;
@Column(name = "test_percent")
private Float testPercent;
@Column(name = "memo")
private String memo;
@ColumnDefault("now()")
@Column(name = "created_dttm")
private ZonedDateTime createdDttm;
@Column(name = "created_uid")
private Long createdUid;
@ColumnDefault("now()")
@Column(name = "updated_dttm")
private ZonedDateTime updatedDttm;
@Column(name = "updated_uid")
private Long updatedUid;
}

View File

@@ -0,0 +1,52 @@
package com.kamco.cd.training.postgres.entity;
import jakarta.persistence.Column;
import jakarta.persistence.Entity;
import jakarta.persistence.FetchType;
import jakarta.persistence.GeneratedValue;
import jakarta.persistence.GenerationType;
import jakarta.persistence.Id;
import jakarta.persistence.JoinColumn;
import jakarta.persistence.ManyToOne;
import jakarta.persistence.Table;
import jakarta.validation.constraints.NotNull;
import java.time.ZonedDateTime;
import lombok.Getter;
import lombok.Setter;
import org.hibernate.annotations.ColumnDefault;
@Getter
@Setter
@Entity
@Table(name = "tb_model_dataset")
public class ModelDatasetEntity {
@Id
@GeneratedValue(strategy = GenerationType.IDENTITY)
@Column(name = "id", nullable = false)
private Long id;
@NotNull
@ManyToOne(fetch = FetchType.LAZY, optional = false)
@JoinColumn(name = "model_id", nullable = false)
private ModelMasterEntity model;
@Column(name = "building_cnt")
private Long buildingCnt;
@Column(name = "container_cnt")
private Long containerCnt;
@Column(name = "waste_cnt")
private Long wasteCnt;
@Column(name = "land_cover_cnt")
private Long landCoverCnt;
@ColumnDefault("now()")
@Column(name = "created_dttm")
private ZonedDateTime createdDttm = ZonedDateTime.now();
@Column(name = "created_uid")
private Long createdUid;
}

View File

@@ -0,0 +1,74 @@
package com.kamco.cd.training.postgres.entity;
import jakarta.persistence.Column;
import jakarta.persistence.Entity;
import jakarta.persistence.FetchType;
import jakarta.persistence.GeneratedValue;
import jakarta.persistence.GenerationType;
import jakarta.persistence.Id;
import jakarta.persistence.JoinColumn;
import jakarta.persistence.ManyToOne;
import jakarta.persistence.Table;
import jakarta.validation.constraints.NotNull;
import jakarta.validation.constraints.Size;
import java.time.ZonedDateTime;
import lombok.Getter;
import lombok.Setter;
import org.hibernate.annotations.ColumnDefault;
@Getter
@Setter
@Entity
@Table(name = "tb_model_history")
public class ModelHistoryEntity {
@Id
@GeneratedValue(strategy = GenerationType.IDENTITY)
@Column(name = "hst_id", nullable = false)
private Integer id;
@NotNull
@ManyToOne(fetch = FetchType.LAZY, optional = false)
@JoinColumn(name = "model_id", nullable = false)
private ModelMasterEntity model;
@Column(name = "model_no")
private Short modelNo;
@Size(max = 50)
@Column(name = "model_ver", length = 50)
private String modelVer;
@Column(name = "model_step")
private Short modelStep;
@Column(name = "step1_strt_dttm")
private ZonedDateTime step1StrtDttm;
@Column(name = "step1_end_dttm")
private ZonedDateTime step1EndDttm;
@Size(max = 20)
@Column(name = "step1_state", length = 20)
private String step1State;
@Column(name = "step2_strt_dttm")
private ZonedDateTime step2StrtDttm;
@Column(name = "step2_end_dttm")
private ZonedDateTime step2EndDttm;
@Size(max = 20)
@Column(name = "step2_state", length = 20)
private String step2State;
@Column(name = "error_msg", length = Integer.MAX_VALUE)
private String errorMsg;
@ColumnDefault("now()")
@Column(name = "created_dttm")
private ZonedDateTime createdDttm;
@Column(name = "created_uid")
private Long createdUid;
}

View File

@@ -1,12 +1,11 @@
package com.kamco.cd.training.postgres.entity;
import com.kamco.cd.training.common.enums.ModelType;
import com.kamco.cd.training.hyperparam.dto.HyperParamDto;
import jakarta.persistence.*;
import jakarta.validation.constraints.NotNull;
import jakarta.validation.constraints.Size;
import java.time.ZonedDateTime;
import java.util.LinkedHashSet;
import java.util.Set;
import java.util.UUID;
import lombok.Getter;
import lombok.Setter;
@@ -280,9 +279,9 @@ public class ModelHyperParamEntity {
private String memo;
@NotNull
@ColumnDefault("'N'")
@ColumnDefault("false")
@Column(name = "del_yn", nullable = false, length = 1)
private String delYn = "N";
private Boolean delYn = false;
@NotNull
@ColumnDefault("CURRENT_TIMESTAMP")
@@ -313,13 +312,20 @@ public class ModelHyperParamEntity {
@Column(name = "m3_use_cnt")
private Long m3UseCnt = 0L;
@OneToMany(mappedBy = "hyperParams", fetch = FetchType.LAZY)
private Set<ModelTrainMasterEntity> trainMasters = new LinkedHashSet<>();
@Column(name = "model_type")
@Enumerated(EnumType.STRING)
private ModelType modelType;
@Column(name = "default_param")
private Boolean isDefault = false;
public HyperParamDto.Basic toDto() {
return new HyperParamDto.Basic(
this.modelType,
this.uuid,
this.hyperVer,
this.createdDttm,
// -------------------------
// Important
// -------------------------
@@ -389,6 +395,8 @@ public class ModelHyperParamEntity {
// -------------------------
this.gpuCnt,
this.gpuIds,
this.masterPort);
this.masterPort
, this.isDefault
);
}
}

View File

@@ -0,0 +1,132 @@
package com.kamco.cd.training.postgres.entity;
import com.kamco.cd.training.model.dto.ModelTrainMngDto;
import jakarta.persistence.Column;
import jakarta.persistence.Entity;
import jakarta.persistence.GeneratedValue;
import jakarta.persistence.GenerationType;
import jakarta.persistence.Id;
import jakarta.persistence.Table;
import jakarta.validation.constraints.Size;
import java.time.ZonedDateTime;
import java.util.UUID;
import lombok.Getter;
import lombok.Setter;
import org.hibernate.annotations.ColumnDefault;
@Getter
@Setter
@Entity
@Table(name = "tb_model_master")
public class ModelMasterEntity {
@Id
@GeneratedValue(strategy = GenerationType.IDENTITY)
@Column(name = "model_id", nullable = false)
private Long id;
@Column(name = "hyper_param_id")
private Long hyperParamId;
@Size(max = 10)
@Column(name = "model_no", length = 10)
private String modelNo;
@Size(max = 200)
@Column(name = "model_ver", length = 200)
private String modelVer;
@Column(name = "model_step")
private Short modelStep;
@Size(max = 20)
@Column(name = "status_cd", length = 20)
private String statusCd;
@Column(name = "strt_dttm")
private ZonedDateTime strtDttm;
@Column(name = "step1_strt_dttm")
private ZonedDateTime step1StrtDttm;
@Column(name = "step1_end_dttm")
private ZonedDateTime step1EndDttm;
@Size(max = 20)
@Column(name = "step1_state", length = 20)
private String step1State;
@Column(name = "step2_strt_dttm")
private ZonedDateTime step2StrtDttm;
@Column(name = "step2_end_dttm")
private ZonedDateTime step2EndDttm;
@Size(max = 20)
@Column(name = "step2_state", length = 20)
private String step2State;
@Column(name = "del_yn")
private Boolean delYn = false;
@ColumnDefault("now()")
@Column(name = "created_dttm")
private ZonedDateTime createdDttm = ZonedDateTime.now();
@Column(name = "created_uid")
private Long createdUid;
@Column(name = "updated_dttm")
private ZonedDateTime updatedDttm;
@Column(name = "updated_uid")
private Long updatedUid;
@Column(name = "uuid")
private UUID uuid = UUID.randomUUID();
@Column(name = "train_type")
private String trainType;
@Column(name = "before_model_id")
private Long beforeModelId;
@Column(name = "step1_metric_save_yn")
private Boolean step1MetricSaveYn;
@Column(name = "step2_metric_save_yn")
private Boolean step2MetricSaveYn;
@Column(name = "current_attempt_id")
private Long currentAttemptId;
@Column(name = "last_error")
private String lastError;
@Column(name = "best_epoch")
private Integer bestEpoch;
@Column(name = "request_path")
private String requestPath;
@Column(name = "response_path")
private String responsePath;
public ModelTrainMngDto.Basic toDto() {
return new ModelTrainMngDto.Basic(
this.id,
this.uuid,
this.modelVer,
this.strtDttm,
this.step1StrtDttm,
this.step1EndDttm,
this.step2StrtDttm,
this.step2EndDttm,
this.step1State,
this.step2State,
this.statusCd,
this.trainType,
this.modelNo,
this.currentAttemptId);
}
}

View File

@@ -0,0 +1,75 @@
package com.kamco.cd.training.postgres.entity;
import jakarta.persistence.Column;
import jakarta.persistence.Entity;
import jakarta.persistence.FetchType;
import jakarta.persistence.GeneratedValue;
import jakarta.persistence.GenerationType;
import jakarta.persistence.Id;
import jakarta.persistence.JoinColumn;
import jakarta.persistence.ManyToOne;
import jakarta.persistence.Table;
import jakarta.validation.constraints.NotNull;
import jakarta.validation.constraints.Size;
import java.time.ZonedDateTime;
import lombok.Getter;
import lombok.Setter;
import org.hibernate.annotations.ColumnDefault;
@Getter
@Setter
@Entity
@Table(name = "tb_model_metrics_test")
public class ModelMetricsTestEntity {
@Id
@GeneratedValue(strategy = GenerationType.IDENTITY)
@Column(name = "mtrc_id", nullable = false)
private Long id;
@NotNull
@ManyToOne(fetch = FetchType.LAZY, optional = false)
@JoinColumn(name = "model_id", nullable = false)
private ModelMasterEntity model;
@Size(max = 50)
@Column(name = "model", length = 50)
private String model1;
@Column(name = "tp")
private Long tp;
@Column(name = "fp")
private Long fp;
@Column(name = "fn")
private Long fn;
@Column(name = "precisions")
private Float precisions;
@Column(name = "recall")
private Float recall;
@Column(name = "f1_score")
private Float f1Score;
@Column(name = "accuracy")
private Float accuracy;
@Column(name = "iou")
private Float iou;
@Column(name = "detection_count")
private Long detectionCount;
@Column(name = "gt_count")
private Long gtCount;
@ColumnDefault("now()")
@Column(name = "created_dttm")
private ZonedDateTime createdDttm;
@Column(name = "created_uid")
private Long createdUid;
}

View File

@@ -0,0 +1,55 @@
package com.kamco.cd.training.postgres.entity;
import jakarta.persistence.Column;
import jakarta.persistence.Entity;
import jakarta.persistence.FetchType;
import jakarta.persistence.GeneratedValue;
import jakarta.persistence.GenerationType;
import jakarta.persistence.Id;
import jakarta.persistence.JoinColumn;
import jakarta.persistence.ManyToOne;
import jakarta.persistence.Table;
import jakarta.validation.constraints.NotNull;
import java.time.ZonedDateTime;
import lombok.Getter;
import lombok.Setter;
import org.hibernate.annotations.ColumnDefault;
@Getter
@Setter
@Entity
@Table(name = "tb_model_metrics_train")
public class ModelMetricsTrainEntity {
@Id
@GeneratedValue(strategy = GenerationType.IDENTITY)
@Column(name = "mtrc_id", nullable = false)
private Long id;
@NotNull
@ManyToOne(fetch = FetchType.LAZY, optional = false)
@JoinColumn(name = "model_id", nullable = false)
private ModelMasterEntity model;
@Column(name = "epoch")
private Integer epoch;
@Column(name = "iteration")
private Long iteration;
@Column(name = "loss")
private Double loss;
@Column(name = "lr")
private Double lr;
@Column(name = "duration_time")
private Float durationTime;
@ColumnDefault("now()")
@Column(name = "created_dttm")
private ZonedDateTime createdDttm;
@Column(name = "created_uid")
private Long createdUid;
}

View File

@@ -0,0 +1,79 @@
package com.kamco.cd.training.postgres.entity;
import jakarta.persistence.Column;
import jakarta.persistence.Entity;
import jakarta.persistence.FetchType;
import jakarta.persistence.GeneratedValue;
import jakarta.persistence.GenerationType;
import jakarta.persistence.Id;
import jakarta.persistence.JoinColumn;
import jakarta.persistence.ManyToOne;
import jakarta.persistence.Table;
import jakarta.validation.constraints.NotNull;
import java.time.ZonedDateTime;
import lombok.Getter;
import lombok.Setter;
import org.hibernate.annotations.ColumnDefault;
@Getter
@Setter
@Entity
@Table(name = "tb_model_metrics_validation")
public class ModelMetricsValidationEntity {
@Id
@GeneratedValue(strategy = GenerationType.IDENTITY)
@Column(name = "mtrc_id", nullable = false)
private Long id;
@NotNull
@ManyToOne(fetch = FetchType.LAZY, optional = false)
@JoinColumn(name = "model_id", nullable = false)
private ModelMasterEntity model;
@Column(name = "epoch")
private Integer epoch;
@Column(name = "a_acc")
private Float aAcc;
@Column(name = "m_fscore")
private Float mFscore;
@Column(name = "m_precision")
private Float mPrecision;
@Column(name = "m_recall")
private Float mRecall;
@Column(name = "m_iou")
private Float mIou;
@Column(name = "m_acc")
private Float mAcc;
@Column(name = "changed_fscore")
private Float changedFscore;
@Column(name = "changed_precision")
private Float changedPrecision;
@Column(name = "changed_recall")
private Float changedRecall;
@Column(name = "unchanged_fscore")
private Float unchangedFscore;
@Column(name = "unchanged_precision")
private Float unchangedPrecision;
@Column(name = "unchanged_recall")
private Float unchangedRecall;
@ColumnDefault("now()")
@Column(name = "created_dttm")
private ZonedDateTime createdDttm;
@Column(name = "created_uid")
private Long createdUid;
}

View File

@@ -0,0 +1,103 @@
package com.kamco.cd.training.postgres.entity;
import com.kamco.cd.training.train.dto.ModelTrainJobDto;
import jakarta.persistence.Column;
import jakarta.persistence.Entity;
import jakarta.persistence.GeneratedValue;
import jakarta.persistence.GenerationType;
import jakarta.persistence.Id;
import jakarta.persistence.Table;
import jakarta.validation.constraints.NotNull;
import jakarta.validation.constraints.Size;
import java.time.ZonedDateTime;
import java.util.Map;
import lombok.Getter;
import lombok.Setter;
import org.hibernate.annotations.ColumnDefault;
import org.hibernate.annotations.JdbcTypeCode;
import org.hibernate.type.SqlTypes;
@Getter
@Setter
@Entity
@Table(name = "tb_model_train_job")
public class ModelTrainJobEntity {
@Id
@GeneratedValue(strategy = GenerationType.IDENTITY)
@Column(name = "id", nullable = false)
private Long id;
@NotNull
@Column(name = "model_id", nullable = false)
private Long modelId;
@NotNull
@Column(name = "attempt_no", nullable = false)
private Integer attemptNo;
@Size(max = 30)
@NotNull
@Column(name = "status_cd", nullable = false, length = 30)
private String statusCd;
@NotNull
@Column(name = "params_json", nullable = false)
@JdbcTypeCode(SqlTypes.JSON)
private Map<String, Object> paramsJson;
@Size(max = 200)
@Column(name = "container_name", length = 200)
private String containerName;
@Size(max = 500)
@Column(name = "log_path", length = 500)
private String logPath;
@Column(name = "exit_code")
private Integer exitCode;
@Size(max = 2000)
@Column(name = "error_message", length = 2000)
private String errorMessage;
@ColumnDefault("now()")
@Column(name = "queued_dttm")
private ZonedDateTime queuedDttm;
@Column(name = "started_dttm")
private ZonedDateTime startedDttm;
@Column(name = "finished_dttm")
private ZonedDateTime finishedDttm;
@Column(name = "locked_dttm")
private ZonedDateTime lockedDttm;
@Size(max = 100)
@Column(name = "locked_by", length = 100)
private String lockedBy;
@Column(name = "total_epoch")
private Integer totalEpoch;
@Column(name = "current_epoch")
private Integer currentEpoch;
public ModelTrainJobDto toDto() {
return new ModelTrainJobDto(
this.id,
this.modelId,
this.attemptNo,
this.statusCd,
this.exitCode,
this.errorMessage,
this.containerName,
this.paramsJson,
this.queuedDttm,
this.startedDttm,
this.finishedDttm,
this.totalEpoch,
this.currentEpoch);
}
}

View File

@@ -1,191 +0,0 @@
package com.kamco.cd.training.postgres.entity;
import com.kamco.cd.training.model.dto.ModelMngDto;
import jakarta.persistence.Column;
import jakarta.persistence.Entity;
import jakarta.persistence.FetchType;
import jakarta.persistence.GeneratedValue;
import jakarta.persistence.GenerationType;
import jakarta.persistence.Id;
import jakarta.persistence.JoinColumn;
import jakarta.persistence.ManyToOne;
import jakarta.persistence.Table;
import jakarta.validation.constraints.NotNull;
import jakarta.validation.constraints.Size;
import java.time.ZonedDateTime;
import java.util.UUID;
import lombok.Getter;
import lombok.Setter;
import org.hibernate.annotations.ColumnDefault;
import org.hibernate.annotations.OnDelete;
import org.hibernate.annotations.OnDeleteAction;
import org.hibernate.annotations.UuidGenerator;
@Getter
@Setter
@Entity
@Table(name = "tb_model_train_master")
public class ModelTrainMasterEntity {
@Id
@GeneratedValue(strategy = GenerationType.IDENTITY)
@Column(name = "model_uid", nullable = false)
private Long id;
@Size(max = 50)
@NotNull
@Column(name = "model_ver", nullable = false, length = 50)
private String modelVer;
@Size(max = 50)
@Column(name = "hyper_ver", length = 50)
private String hyperVer;
/** tb_model_train_master.hyper_param_id -> tb_model_hyper_params.hyper_param_id */
@NotNull
@Column(name = "hyper_param_id", nullable = false)
private Long hyperParamId;
/** - hyperParamId 컬럼으로 저장/수정하고 - 객체로 조회할 때만 hyperParams 사용 */
@ManyToOne(fetch = FetchType.LAZY)
@JoinColumn(
name = "hyper_param_id",
referencedColumnName = "hyper_param_id",
insertable = false,
updatable = false)
private ModelHyperParamEntity hyperParams;
@Size(max = 50)
@Column(name = "epoch_ver", length = 50)
private String epochVer;
@Size(max = 50)
@Column(name = "process_step", length = 50)
private String processStep;
@Column(name = "train_start_dttm")
private ZonedDateTime trainStartDttm;
@Column(name = "epoch_cnt")
private Integer epochCnt;
@Size(max = 50)
@Column(name = "dataset_ratio", length = 50)
private String datasetRatio;
@Column(name = "best_epoch")
private Integer bestEpoch;
@Column(name = "step1_end_dttm")
private ZonedDateTime step1EndDttm;
@Column(name = "step1_duration")
private ZonedDateTime step1Duration;
@Column(name = "step2_end_dttm")
private ZonedDateTime step2EndDttm;
@Column(name = "step2_duration")
private ZonedDateTime step2Duration;
@NotNull
@ColumnDefault("false")
@Column(name = "del_yn", nullable = false)
private Boolean delYn = false;
@Column(name = "created_uid")
private Long createdUid;
@Column(name = "updated_uid")
private Long updatedUid;
@ColumnDefault("now()")
@Column(name = "created_dttm")
private ZonedDateTime createdDttm;
@Column(name = "updated_dttm")
private ZonedDateTime updatedDttm;
@ColumnDefault("0")
@Column(name = "progress_rate")
private Integer progressRate;
@Column(name = "stop_dttm")
private ZonedDateTime stopDttm;
@Column(name = "confirmed_best_epoch")
private Integer confirmedBestEpoch;
@Size(max = 255)
@Column(name = "model_path")
private String modelPath;
@Column(name = "error_msg")
private String errorMsg;
@Column(name = "step2_start_dttm")
private ZonedDateTime step2StartDttm;
@Size(max = 1000)
@Column(name = "train_log_path", length = 1000)
private String trainLogPath;
@Column(name = "memo")
private String memo;
/** 기존 자기참조(base_model_uid)는 유지 */
@ManyToOne(fetch = FetchType.LAZY)
@OnDelete(action = OnDeleteAction.SET_NULL)
@JoinColumn(name = "base_model_uid")
private ModelTrainMasterEntity baseModel;
@Size(max = 1000)
@Column(name = "pretrained_model_path", length = 1000)
private String pretrainedModelPath;
@Column(name = "last_checkpoint_epoch")
private Integer lastCheckpointEpoch;
@Size(max = 500)
@Column(name = "checkpoint_path", length = 500)
private String checkpointPath;
@Column(name = "can_resume")
private Boolean canResume;
@NotNull
@UuidGenerator
@Column(name = "uuid", nullable = false, updatable = false)
private UUID uuid = UUID.randomUUID();
@Size(max = 10)
@Column(name = "step1_status", length = 10)
private String step1Status;
@Size(max = 10)
@Column(name = "step2_status", length = 10)
private String step2Status;
@Size(max = 10)
@Column(name = "transfer_status", length = 10)
private String transferStatus;
@Size(max = 10)
@Column(name = "status_cd", length = 10)
private String statusCd;
public ModelMngDto.Basic toDto() {
return new ModelMngDto.Basic(
this.id,
this.uuid,
this.modelVer,
this.trainStartDttm,
this.step1EndDttm,
this.step1Duration,
this.step2EndDttm,
this.step2Duration,
this.step1Status,
this.step2Status,
this.transferStatus);
}
}

View File

@@ -0,0 +1,7 @@
package com.kamco.cd.training.postgres.repository.dataset;
import com.kamco.cd.training.postgres.entity.DatasetObjEntity;
import org.springframework.data.jpa.repository.JpaRepository;
public interface DatasetObjRepository
extends JpaRepository<DatasetObjEntity, Long>, DatasetObjRepositoryCustom {}

View File

@@ -0,0 +1,27 @@
package com.kamco.cd.training.postgres.repository.dataset;
import com.kamco.cd.training.dataset.dto.DatasetObjDto.DatasetClass;
import com.kamco.cd.training.dataset.dto.DatasetObjDto.DatasetObjRegDto;
import com.kamco.cd.training.dataset.dto.DatasetObjDto.SearchReq;
import com.kamco.cd.training.postgres.entity.DatasetObjEntity;
import java.util.List;
import java.util.Optional;
import java.util.UUID;
import org.springframework.data.domain.Page;
public interface DatasetObjRepositoryCustom {
Optional<DatasetObjEntity> findByUuid(UUID id);
Page<DatasetObjEntity> searchDatasetObjectList(SearchReq searchReq);
List<DatasetClass> findDatasetObjClassByUuid(UUID uuid, String type);
void insertDatasetObj(DatasetObjRegDto objRegDto);
String getFilePathByUUIDPathType(UUID uuid, String pathType);
void insertDatasetTestObj(DatasetObjRegDto objRegDto);
void insertDatasetValObj(DatasetObjRegDto objRegDto);
}

View File

@@ -0,0 +1,279 @@
package com.kamco.cd.training.postgres.repository.dataset;
import static com.kamco.cd.training.postgres.entity.QDatasetEntity.datasetEntity;
import static com.kamco.cd.training.postgres.entity.QDatasetObjEntity.datasetObjEntity;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.kamco.cd.training.dataset.dto.DatasetObjDto.DatasetClass;
import com.kamco.cd.training.dataset.dto.DatasetObjDto.DatasetObjRegDto;
import com.kamco.cd.training.dataset.dto.DatasetObjDto.SearchReq;
import com.kamco.cd.training.postgres.entity.DatasetEntity;
import com.kamco.cd.training.postgres.entity.DatasetObjEntity;
import com.kamco.cd.training.postgres.entity.QDatasetEntity;
import com.querydsl.core.BooleanBuilder;
import com.querydsl.core.types.Expression;
import com.querydsl.core.types.Projections;
import com.querydsl.core.types.dsl.Expressions;
import com.querydsl.core.types.dsl.StringPath;
import com.querydsl.jpa.impl.JPAQueryFactory;
import jakarta.persistence.EntityManager;
import jakarta.persistence.EntityNotFoundException;
import jakarta.persistence.PersistenceContext;
import java.util.List;
import java.util.Objects;
import java.util.Optional;
import java.util.UUID;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import org.locationtech.jts.geom.Geometry;
import org.springframework.data.domain.Page;
import org.springframework.data.domain.PageImpl;
import org.springframework.data.domain.Pageable;
import org.springframework.stereotype.Repository;
@Slf4j
@Repository
@RequiredArgsConstructor
public class DatasetObjRepositoryImpl implements DatasetObjRepositoryCustom {
private final JPAQueryFactory queryFactory;
private final QDatasetEntity dataset = datasetEntity;
@PersistenceContext EntityManager em;
@Override
public Optional<DatasetObjEntity> findByUuid(UUID id) {
return Optional.ofNullable(
queryFactory
.select(datasetObjEntity)
.from(datasetObjEntity)
.where(datasetObjEntity.uuid.eq(id), datasetObjEntity.deleted.isFalse())
.fetchOne());
}
@Override
public void insertDatasetTestObj(DatasetObjRegDto objRegDto) {
ObjectMapper objectMapper = new ObjectMapper();
String json;
Geometry geometry;
String geometryJson;
try {
json = objectMapper.writeValueAsString(objRegDto.getGeojson());
geometryJson =
objectMapper.writeValueAsString(
objRegDto.getGeojson().path("features").get(0).path("geometry"));
} catch (JsonProcessingException e) {
throw new RuntimeException(e);
}
try {
em.createNativeQuery(
"""
insert into tb_dataset_test_obj
(dataset_uid, target_yyyy, target_class_cd,
compare_yyyy, compare_class_cd,
target_path, compare_path, label_path, geo_jsonb, map_sheet_num, file_name, geom, geojson_path)
values
(?, ?, ?, ?, ?, ?, ?, ?, cast(? as jsonb), ?, ?, ST_SetSRID(ST_GeomFromGeoJSON(?), 5186), ?)
""")
.setParameter(1, objRegDto.getDatasetUid())
.setParameter(2, objRegDto.getTargetYyyy())
.setParameter(3, objRegDto.getTargetClassCd())
.setParameter(4, objRegDto.getCompareYyyy())
.setParameter(5, objRegDto.getCompareClassCd())
.setParameter(6, objRegDto.getTargetPath())
.setParameter(7, objRegDto.getComparePath())
.setParameter(8, objRegDto.getLabelPath())
.setParameter(9, json)
.setParameter(10, objRegDto.getMapSheetNum())
.setParameter(11, objRegDto.getFileName())
.setParameter(12, geometryJson)
.setParameter(13, objRegDto.getGeojsonPath())
.executeUpdate();
} catch (Exception e) {
throw new RuntimeException(e);
}
}
@Override
public void insertDatasetValObj(DatasetObjRegDto objRegDto) {
ObjectMapper objectMapper = new ObjectMapper();
String json;
String geometryJson;
try {
json = objectMapper.writeValueAsString(objRegDto.getGeojson());
geometryJson =
objectMapper.writeValueAsString(
objRegDto.getGeojson().path("features").get(0).path("geometry"));
} catch (JsonProcessingException e) {
throw new RuntimeException(e);
}
try {
em.createNativeQuery(
"""
insert into tb_dataset_val_obj
(dataset_uid, target_yyyy, target_class_cd,
compare_yyyy, compare_class_cd,
target_path, compare_path, label_path, geo_jsonb, map_sheet_num, file_name, geom, geojson_path)
values
(?, ?, ?, ?, ?, ?, ?, ?, cast(? as jsonb), ?, ?, ST_SetSRID(ST_GeomFromGeoJSON(?), 5186), ?)
""")
.setParameter(1, objRegDto.getDatasetUid())
.setParameter(2, objRegDto.getTargetYyyy())
.setParameter(3, objRegDto.getTargetClassCd())
.setParameter(4, objRegDto.getCompareYyyy())
.setParameter(5, objRegDto.getCompareClassCd())
.setParameter(6, objRegDto.getTargetPath())
.setParameter(7, objRegDto.getComparePath())
.setParameter(8, objRegDto.getLabelPath())
.setParameter(9, json)
.setParameter(10, objRegDto.getMapSheetNum())
.setParameter(11, objRegDto.getFileName())
.setParameter(12, geometryJson)
.setParameter(13, objRegDto.getGeojsonPath())
.executeUpdate();
} catch (Exception e) {
throw new RuntimeException(e);
}
}
@Override
public Page<DatasetObjEntity> searchDatasetObjectList(SearchReq searchReq) {
Pageable pageable = searchReq.toPageable();
BooleanBuilder builder = new BooleanBuilder();
if (StringUtils.isNotBlank(searchReq.getCompareClassCd())) {
builder.and(datasetObjEntity.compareClassCd.eq(searchReq.getCompareClassCd()));
}
if (StringUtils.isNotBlank(searchReq.getTargetClassCd())) {
builder.and(datasetObjEntity.targetClassCd.eq(searchReq.getTargetClassCd()));
}
if (StringUtils.isNotBlank(searchReq.getMapSheetNum())) {
builder.and(datasetObjEntity.mapSheetNum.eq(searchReq.getMapSheetNum()));
}
DatasetEntity entity =
queryFactory
.selectFrom(datasetEntity)
.where(datasetEntity.uuid.eq(searchReq.getUuid()))
.fetchOne();
if (Objects.isNull(entity)) {
throw new EntityNotFoundException("DatasetEntity not found for uuid: " + searchReq.getUuid());
}
List<DatasetObjEntity> content =
queryFactory
.selectFrom(datasetObjEntity)
.where(
datasetObjEntity
.deleted
.isFalse()
.and(datasetObjEntity.datasetUid.eq(entity.getId()))
.and(builder))
.offset(pageable.getOffset())
.limit(pageable.getPageSize())
.fetch();
// Count 쿼리 별도 실행 (null safe handling)
long total =
Optional.ofNullable(
queryFactory
.select(datasetObjEntity.count())
.from(datasetObjEntity)
.where(
datasetObjEntity
.deleted
.isFalse()
.and(datasetObjEntity.datasetUid.eq(entity.getId()))
.and(builder))
.fetchOne())
.orElse(0L);
return new PageImpl<>(content, pageable, total);
}
@Override
public List<DatasetClass> findDatasetObjClassByUuid(UUID uuid, String type) {
StringPath classCd =
type.equals("compare") ? datasetObjEntity.compareClassCd : datasetObjEntity.targetClassCd;
BooleanBuilder builder = new BooleanBuilder();
builder.and(datasetEntity.uuid.eq(uuid));
builder.and(datasetEntity.deleted.isFalse());
builder.and(datasetObjEntity.deleted.isFalse());
return queryFactory
.select(Projections.constructor(DatasetClass.class, classCd))
.from(datasetEntity)
.join(datasetObjEntity)
.on(datasetEntity.id.eq(datasetObjEntity.datasetUid))
.where(builder)
.groupBy(classCd)
.fetch();
}
@Override
public void insertDatasetObj(DatasetObjRegDto objRegDto) {
ObjectMapper objectMapper = new ObjectMapper();
String json;
String geometryJson;
try {
json = objectMapper.writeValueAsString(objRegDto.getGeojson());
geometryJson =
objectMapper.writeValueAsString(
objRegDto.getGeojson().path("features").get(0).path("geometry"));
} catch (JsonProcessingException e) {
throw new RuntimeException(e);
}
try {
em.createNativeQuery(
"""
insert into tb_dataset_obj
(dataset_uid, target_yyyy, target_class_cd,
compare_yyyy, compare_class_cd,
target_path, compare_path, label_path, geo_jsonb, map_sheet_num, file_name, geom, geojson_path)
values
(?, ?, ?, ?, ?, ?, ?, ?, cast(? as jsonb), ?, ?, ST_SetSRID(ST_GeomFromGeoJSON(?), 5186), ?)
""")
.setParameter(1, objRegDto.getDatasetUid())
.setParameter(2, objRegDto.getTargetYyyy())
.setParameter(3, objRegDto.getTargetClassCd())
.setParameter(4, objRegDto.getCompareYyyy())
.setParameter(5, objRegDto.getCompareClassCd())
.setParameter(6, objRegDto.getTargetPath())
.setParameter(7, objRegDto.getComparePath())
.setParameter(8, objRegDto.getLabelPath())
.setParameter(9, json)
.setParameter(10, objRegDto.getMapSheetNum())
.setParameter(11, objRegDto.getFileName())
.setParameter(12, geometryJson)
.setParameter(13, objRegDto.getGeojsonPath())
.executeUpdate();
} catch (Exception e) {
throw new RuntimeException(e);
}
}
@Override
public String getFilePathByUUIDPathType(UUID uuid, String pathType) {
Expression<String> pathExpr =
switch (pathType) {
case "compare" -> datasetObjEntity.comparePath;
case "target" -> datasetObjEntity.targetPath;
case "label" -> datasetObjEntity.labelPath;
default -> Expressions.constant("");
};
return queryFactory
.select(pathExpr)
.from(datasetObjEntity)
.where(datasetObjEntity.uuid.eq(uuid))
.fetchOne();
}
}

View File

@@ -1,7 +1,11 @@
package com.kamco.cd.training.postgres.repository.dataset;
import com.kamco.cd.training.dataset.dto.DatasetDto;
import com.kamco.cd.training.dataset.dto.DatasetDto.DatasetMngRegDto;
import com.kamco.cd.training.dataset.dto.DatasetDto.DatasetReq;
import com.kamco.cd.training.dataset.dto.DatasetDto.SelectDataSet;
import com.kamco.cd.training.postgres.entity.DatasetEntity;
import java.util.List;
import java.util.Optional;
import java.util.UUID;
import org.springframework.data.domain.Page;
@@ -10,4 +14,14 @@ public interface DatasetRepositoryCustom {
Page<DatasetEntity> findDatasetList(DatasetDto.SearchReq searchReq);
Optional<DatasetEntity> findByUuid(UUID id);
List<SelectDataSet> getDatasetSelectG1List(DatasetReq req);
List<SelectDataSet> getDatasetSelectG2G3List(DatasetReq req);
Long getDatasetMaxStage(int compareYyyy, int targetYyyy);
Long insertDatasetMngData(DatasetMngRegDto mngRegDto);
List<String> findDatasetUid(List<Long> datasetIds);
}

View File

@@ -1,9 +1,19 @@
package com.kamco.cd.training.postgres.repository.dataset;
import com.kamco.cd.training.dataset.dto.DatasetDto;
import static com.kamco.cd.training.postgres.entity.QDatasetObjEntity.datasetObjEntity;
import com.kamco.cd.training.common.enums.ModelType;
import com.kamco.cd.training.dataset.dto.DatasetDto.DatasetMngRegDto;
import com.kamco.cd.training.dataset.dto.DatasetDto.DatasetReq;
import com.kamco.cd.training.dataset.dto.DatasetDto.SearchReq;
import com.kamco.cd.training.dataset.dto.DatasetDto.SelectDataSet;
import com.kamco.cd.training.postgres.entity.DatasetEntity;
import com.kamco.cd.training.postgres.entity.QDatasetEntity;
import com.querydsl.core.BooleanBuilder;
import com.querydsl.core.types.Projections;
import com.querydsl.core.types.dsl.CaseBuilder;
import com.querydsl.core.types.dsl.Expressions;
import com.querydsl.core.types.dsl.NumberExpression;
import com.querydsl.jpa.impl.JPAQueryFactory;
import java.util.List;
import java.util.Optional;
@@ -29,7 +39,7 @@ public class DatasetRepositoryImpl implements DatasetRepositoryCustom {
* @return 페이징 처리된 데이터셋 Entity 목록
*/
@Override
public Page<DatasetEntity> findDatasetList(DatasetDto.SearchReq searchReq) {
public Page<DatasetEntity> findDatasetList(SearchReq searchReq) {
Pageable pageable = searchReq.toPageable();
BooleanBuilder builder = new BooleanBuilder();
@@ -40,8 +50,8 @@ public class DatasetRepositoryImpl implements DatasetRepositoryCustom {
}
// 구분
if (StringUtils.isNotBlank(searchReq.getGroupTitle())) {
builder.and(dataset.groupTitle.eq(searchReq.getGroupTitle()));
if (StringUtils.isNotBlank(searchReq.getDataType())) {
builder.and(dataset.dataType.eq(searchReq.getDataType()));
}
// Entity 직접 조회 (Projections 사용 지양)
@@ -72,4 +82,169 @@ public class DatasetRepositoryImpl implements DatasetRepositoryCustom {
.where(dataset.uuid.eq(id), dataset.deleted.isFalse())
.fetchOne());
}
@Override
public List<SelectDataSet> getDatasetSelectG1List(DatasetReq req) {
BooleanBuilder builder = new BooleanBuilder();
if (StringUtils.isNotBlank(req.getDataType()) && !"CURRENT".equals(req.getDataType())) {
builder.and(dataset.dataType.eq(req.getDataType()));
}
if (StringUtils.isNotBlank(req.getDataType()) && !"CURRENT".equals(req.getDataType())) {
builder.and(dataset.dataType.eq(req.getDataType()));
}
if (req.getIds() != null) {
builder.and(dataset.id.in(req.getIds()));
}
return queryFactory
.select(
Projections.constructor(
SelectDataSet.class,
Expressions.constant(req.getModelNo()),
dataset.id,
dataset.uuid,
dataset.dataType,
dataset.title,
dataset.roundNo,
dataset.compareYyyy,
dataset.targetYyyy,
dataset.memo,
new CaseBuilder()
.when(datasetObjEntity.targetClassCd.eq("building"))
.then(1)
.otherwise(0)
.sum(),
new CaseBuilder()
.when(datasetObjEntity.targetClassCd.eq("container"))
.then(1)
.otherwise(0)
.sum()))
.from(dataset)
.leftJoin(datasetObjEntity)
.on(dataset.id.eq(datasetObjEntity.datasetUid))
.where(builder) // datasetObjEntity.targetClassCd.in("building", "container").and(
.groupBy(
dataset.id,
dataset.uuid,
dataset.dataType,
dataset.title,
dataset.roundNo,
dataset.memo)
.orderBy(dataset.createdDttm.desc())
.fetch();
}
@Override
public List<SelectDataSet> getDatasetSelectG2G3List(DatasetReq req) {
BooleanBuilder builder = new BooleanBuilder();
NumberExpression<Long> selectedCnt = null;
NumberExpression<Long> wasteCnt =
datasetObjEntity.targetClassCd.when("waste").then(1L).otherwise(0L).sum();
NumberExpression<Long> elseCnt =
new CaseBuilder()
.when(datasetObjEntity.targetClassCd.notIn("building", "container", "waste"))
.then(1L)
.otherwise(0L)
.sum();
if (StringUtils.isNotBlank(req.getModelNo())) {
if (req.getModelNo().equals(ModelType.G2.getId())) {
selectedCnt = wasteCnt;
} else {
selectedCnt = elseCnt;
}
}
if (StringUtils.isNotBlank(req.getDataType())) {
if (!"CURRENT".equals(req.getDataType())) {
builder.and(dataset.dataType.eq(req.getDataType()));
}
}
if (req.getIds() != null) {
builder.and(dataset.id.in(req.getIds()));
}
return queryFactory
.select(
Projections.constructor(
SelectDataSet.class,
Expressions.constant(req.getModelNo()),
dataset.id,
dataset.uuid,
dataset.dataType,
dataset.title,
dataset.roundNo,
dataset.compareYyyy,
dataset.targetYyyy,
dataset.memo,
selectedCnt.as("cnt")))
.from(dataset)
.leftJoin(datasetObjEntity)
.on(dataset.id.eq(datasetObjEntity.datasetUid))
.where(builder)
.groupBy(
dataset.id,
dataset.uuid,
dataset.dataType,
dataset.title,
dataset.roundNo,
dataset.memo)
.orderBy(dataset.createdDttm.desc())
.fetch();
}
@Override
public Long getDatasetMaxStage(int compareYyyy, int targetYyyy) {
return queryFactory
.select(dataset.roundNo.max().coalesce(0L))
.from(dataset)
.where(dataset.compareYyyy.eq(compareYyyy), dataset.targetYyyy.eq(targetYyyy))
.fetchOne();
}
@Override
public Long insertDatasetMngData(DatasetMngRegDto mngRegDto) {
queryFactory
.insert(dataset)
.columns(
dataset.uid,
dataset.dataType,
dataset.compareYyyy,
dataset.targetYyyy,
dataset.roundNo,
dataset.totalSize,
dataset.title,
dataset.memo,
dataset.datasetPath)
.values(
mngRegDto.getUid(),
mngRegDto.getDataType(),
mngRegDto.getCompareYyyy(),
mngRegDto.getTargetYyyy(),
mngRegDto.getRoundNo(),
mngRegDto.getTotalSize(),
mngRegDto.getTitle(),
mngRegDto.getMemo(),
mngRegDto.getDatasetPath())
.execute();
return queryFactory
.select(dataset.id)
.from(dataset)
.where(dataset.uid.eq(mngRegDto.getUid()))
.fetchOne();
}
@Override
public List<String> findDatasetUid(List<Long> datasetIds) {
return queryFactory.select(dataset.uid).from(dataset).where(dataset.id.in(datasetIds)).fetch();
}
}

View File

@@ -1,9 +1,12 @@
package com.kamco.cd.training.postgres.repository.hyperparam;
import com.kamco.cd.training.postgres.entity.ModelHyperParamEntity;
import java.util.Optional;
import org.springframework.data.jpa.repository.JpaRepository;
import org.springframework.stereotype.Repository;
@Repository
public interface HyperParamRepository
extends JpaRepository<ModelHyperParamEntity, Long>, HyperParamRepositoryCustom {}
extends JpaRepository<ModelHyperParamEntity, Long>, HyperParamRepositoryCustom {
Optional<ModelHyperParamEntity> findByHyperVer(String hyperVer);
}

View File

@@ -1,6 +1,8 @@
package com.kamco.cd.training.postgres.repository.hyperparam;
import com.kamco.cd.training.common.enums.ModelType;
import com.kamco.cd.training.hyperparam.dto.HyperParamDto;
import com.kamco.cd.training.hyperparam.dto.HyperParamDto.SearchReq;
import com.kamco.cd.training.postgres.entity.ModelHyperParamEntity;
import java.util.Optional;
import java.util.UUID;
@@ -13,9 +15,22 @@ public interface HyperParamRepositoryCustom {
*
* @return
*/
@Deprecated
Optional<ModelHyperParamEntity> findHyperParamVer();
/**
* 모델 타입별 마지막 버전 조회
*
* @param modelType 모델 타입
* @return
*/
Optional<ModelHyperParamEntity> findHyperParamVerByModelType(ModelType modelType);
Optional<ModelHyperParamEntity> findHyperParamByHyperVer(String hyperVer);
Optional<ModelHyperParamEntity> findHyperParamByUuid(UUID uuid);
Page<HyperParamDto.List> findByHyperVerList(HyperParamDto.SearchReq req);
Page<HyperParamDto.List> findByHyperVerList(ModelType model, SearchReq req);
Optional<ModelHyperParamEntity> getHyperparamByType(ModelType modelType);
}

View File

@@ -2,8 +2,10 @@ package com.kamco.cd.training.postgres.repository.hyperparam;
import static com.kamco.cd.training.postgres.entity.QModelHyperParamEntity.modelHyperParamEntity;
import com.kamco.cd.training.common.enums.ModelType;
import com.kamco.cd.training.hyperparam.dto.HyperParamDto;
import com.kamco.cd.training.hyperparam.dto.HyperParamDto.HyperType;
import com.kamco.cd.training.hyperparam.dto.HyperParamDto.SearchReq;
import com.kamco.cd.training.postgres.entity.ModelHyperParamEntity;
import com.querydsl.core.BooleanBuilder;
import com.querydsl.core.types.Projections;
@@ -35,28 +37,62 @@ public class HyperParamRepositoryImpl implements HyperParamRepositoryCustom {
queryFactory
.select(modelHyperParamEntity)
.from(modelHyperParamEntity)
.where(modelHyperParamEntity.delYn.eq("N"))
.where(modelHyperParamEntity.delYn.isFalse())
.orderBy(modelHyperParamEntity.hyperVer.desc())
.limit(1)
.fetchOne());
}
@Override
public Optional<ModelHyperParamEntity> findHyperParamVerByModelType(ModelType modelType) {
return Optional.ofNullable(
queryFactory
.select(modelHyperParamEntity)
.from(modelHyperParamEntity)
.where(
modelHyperParamEntity
.delYn
.isFalse()
.and(modelHyperParamEntity.modelType.eq(modelType)))
.orderBy(modelHyperParamEntity.hyperVer.desc())
.limit(1)
.fetchOne());
}
@Override
public Optional<ModelHyperParamEntity> findHyperParamByHyperVer(String hyperVer) {
return Optional.ofNullable(
queryFactory
.select(modelHyperParamEntity)
.from(modelHyperParamEntity)
.where(
modelHyperParamEntity
.delYn
.isFalse()
.and(modelHyperParamEntity.hyperVer.eq(hyperVer)))
.limit(1)
.fetchOne());
}
@Override
public Optional<ModelHyperParamEntity> findHyperParamByUuid(UUID uuid) {
return Optional.ofNullable(
queryFactory
.select(modelHyperParamEntity)
.from(modelHyperParamEntity)
.where(modelHyperParamEntity.delYn.eq("N").and(modelHyperParamEntity.uuid.eq(uuid)))
.where(modelHyperParamEntity.delYn.isFalse().and(modelHyperParamEntity.uuid.eq(uuid)))
.fetchOne());
}
@Override
public Page<HyperParamDto.List> findByHyperVerList(HyperParamDto.SearchReq req) {
public Page<HyperParamDto.List> findByHyperVerList(ModelType model, SearchReq req) {
Pageable pageable = req.toPageable();
BooleanBuilder builder = new BooleanBuilder();
builder.and(modelHyperParamEntity.delYn.eq("N"));
builder.and(modelHyperParamEntity.modelType.eq(model));
builder.and(modelHyperParamEntity.delYn.isFalse());
if (req.getHyperVer() != null && !req.getHyperVer().isEmpty()) {
// 버전
@@ -145,4 +181,14 @@ public class HyperParamRepositoryImpl implements HyperParamRepositoryCustom {
return new PageImpl<>(content, pageable, totalCount);
}
@Override
public Optional<ModelHyperParamEntity> getHyperparamByType(ModelType modelType) {
return Optional.ofNullable(
queryFactory
.select(modelHyperParamEntity)
.from(modelHyperParamEntity)
.where(modelHyperParamEntity.delYn.isFalse().and(modelHyperParamEntity.modelType.eq(modelType)))
.fetchOne());
}
}

View File

@@ -0,0 +1,7 @@
package com.kamco.cd.training.postgres.repository.model;
import com.kamco.cd.training.postgres.entity.ModelConfigEntity;
import org.springframework.data.jpa.repository.JpaRepository;
public interface ModelConfigRepository
extends JpaRepository<ModelConfigEntity, Long>, ModelConfigRepositoryCustom {}

View File

@@ -0,0 +1,8 @@
package com.kamco.cd.training.postgres.repository.model;
import com.kamco.cd.training.model.dto.ModelConfigDto;
import java.util.Optional;
public interface ModelConfigRepositoryCustom {
Optional<ModelConfigDto.Basic> findModelConfigByModelId(Long modelId);
}

View File

@@ -0,0 +1,37 @@
package com.kamco.cd.training.postgres.repository.model;
import static com.kamco.cd.training.postgres.entity.QModelConfigEntity.modelConfigEntity;
import com.kamco.cd.training.model.dto.ModelConfigDto.Basic;
import com.querydsl.core.types.Projections;
import com.querydsl.jpa.impl.JPAQueryFactory;
import java.util.Optional;
import lombok.RequiredArgsConstructor;
import org.springframework.stereotype.Repository;
@Repository
@RequiredArgsConstructor
public class ModelConfigRepositoryImpl implements ModelConfigRepositoryCustom {
private final JPAQueryFactory queryFactory;
@Override
public Optional<Basic> findModelConfigByModelId(Long modelId) {
return Optional.ofNullable(
queryFactory
.select(
Projections.constructor(
Basic.class,
modelConfigEntity.id,
modelConfigEntity.model.id,
modelConfigEntity.epochCount,
modelConfigEntity.trainPercent,
modelConfigEntity.validationPercent,
modelConfigEntity.testPercent,
modelConfigEntity.memo))
.from(modelConfigEntity)
.where(modelConfigEntity.model.id.eq(modelId))
.fetchOne());
}
}

View File

@@ -6,4 +6,5 @@ import org.springframework.stereotype.Repository;
@Repository
public interface ModelDatasetMappRepository
extends JpaRepository<ModelDatasetMappEntity, ModelDatasetMappEntity.ModelDatasetMappId> {}
extends JpaRepository<ModelDatasetMappEntity, ModelDatasetMappEntity.ModelDatasetMappId>,
ModelDatasetMappRepositoryCustom {}

View File

@@ -0,0 +1,8 @@
package com.kamco.cd.training.postgres.repository.model;
import com.kamco.cd.training.postgres.entity.ModelDatasetMappEntity;
import java.util.List;
public interface ModelDatasetMappRepositoryCustom {
List<ModelDatasetMappEntity> findByModelUid(Long modelId);
}

View File

@@ -0,0 +1,25 @@
package com.kamco.cd.training.postgres.repository.model;
import static com.kamco.cd.training.postgres.entity.QModelDatasetMappEntity.modelDatasetMappEntity;
import com.kamco.cd.training.postgres.entity.ModelDatasetMappEntity;
import com.querydsl.jpa.impl.JPAQueryFactory;
import java.util.List;
import lombok.RequiredArgsConstructor;
import org.springframework.stereotype.Repository;
@Repository
@RequiredArgsConstructor
public class ModelDatasetMappRepositoryImpl implements ModelDatasetMappRepositoryCustom {
private final JPAQueryFactory queryFactory;
@Override
public List<ModelDatasetMappEntity> findByModelUid(Long modelId) {
queryFactory
.select(modelDatasetMappEntity)
.from(modelDatasetMappEntity)
.where(modelDatasetMappEntity.modelUid.eq(modelId));
return List.of();
}
}

View File

@@ -0,0 +1,7 @@
package com.kamco.cd.training.postgres.repository.model;
import com.kamco.cd.training.postgres.entity.ModelDatasetEntity;
import org.springframework.data.jpa.repository.JpaRepository;
public interface ModelDatasetRepository
extends JpaRepository<ModelDatasetEntity, Long>, ModelDatasetRepositoryCustom {}

View File

@@ -0,0 +1,3 @@
package com.kamco.cd.training.postgres.repository.model;
public interface ModelDatasetRepositoryCustom {}

View File

@@ -0,0 +1,7 @@
package com.kamco.cd.training.postgres.repository.model;
import com.kamco.cd.training.postgres.entity.ModelMasterEntity;
import org.springframework.data.jpa.repository.JpaRepository;
public interface ModelDetailRepository
extends JpaRepository<ModelMasterEntity, Long>, ModelDetailRepositoryCustom {}

View File

@@ -0,0 +1,37 @@
package com.kamco.cd.training.postgres.repository.model;
import com.kamco.cd.training.model.dto.ModelTrainDetailDto.DetailSummary;
import com.kamco.cd.training.model.dto.ModelTrainDetailDto.HyperSummary;
import com.kamco.cd.training.model.dto.ModelTrainDetailDto.MappingDataset;
import com.kamco.cd.training.model.dto.ModelTrainDetailDto.ModelBestEpoch;
import com.kamco.cd.training.model.dto.ModelTrainDetailDto.ModelTestMetrics;
import com.kamco.cd.training.model.dto.ModelTrainDetailDto.ModelTrainMetrics;
import com.kamco.cd.training.model.dto.ModelTrainDetailDto.ModelValidationMetrics;
import com.kamco.cd.training.model.dto.ModelTrainDetailDto.TransferHyperSummary;
import com.kamco.cd.training.postgres.entity.ModelMasterEntity;
import java.util.List;
import java.util.Optional;
import java.util.UUID;
public interface ModelDetailRepositoryCustom {
Optional<ModelMasterEntity> findByUuid(UUID uuid);
DetailSummary getModelDetailSummary(UUID uuid);
HyperSummary getByModelHyperParamSummary(UUID uuid);
TransferHyperSummary getByModelTransferHyperParamSummary(UUID uuid);
List<MappingDataset> getByModelMappingDataset(UUID uuid);
ModelMasterEntity findByModelByUUID(UUID uuid);
List<ModelTrainMetrics> getModelTrainMetricResult(UUID uuid);
List<ModelValidationMetrics> getModelValidationMetricResult(UUID uuid);
List<ModelTestMetrics> getModelTestMetricResult(UUID uuid);
ModelBestEpoch getModelTrainBestEpoch(UUID uuid);
}

View File

@@ -0,0 +1,272 @@
package com.kamco.cd.training.postgres.repository.model;
import static com.kamco.cd.training.postgres.entity.QDatasetEntity.datasetEntity;
import static com.kamco.cd.training.postgres.entity.QModelDatasetEntity.modelDatasetEntity;
import static com.kamco.cd.training.postgres.entity.QModelDatasetMappEntity.modelDatasetMappEntity;
import static com.kamco.cd.training.postgres.entity.QModelHyperParamEntity.modelHyperParamEntity;
import static com.kamco.cd.training.postgres.entity.QModelMasterEntity.modelMasterEntity;
import static com.kamco.cd.training.postgres.entity.QModelMetricsTestEntity.modelMetricsTestEntity;
import static com.kamco.cd.training.postgres.entity.QModelMetricsTrainEntity.modelMetricsTrainEntity;
import static com.kamco.cd.training.postgres.entity.QModelMetricsValidationEntity.modelMetricsValidationEntity;
import com.kamco.cd.training.model.dto.ModelTrainDetailDto.DetailSummary;
import com.kamco.cd.training.model.dto.ModelTrainDetailDto.HyperSummary;
import com.kamco.cd.training.model.dto.ModelTrainDetailDto.MappingDataset;
import com.kamco.cd.training.model.dto.ModelTrainDetailDto.ModelBestEpoch;
import com.kamco.cd.training.model.dto.ModelTrainDetailDto.ModelTestMetrics;
import com.kamco.cd.training.model.dto.ModelTrainDetailDto.ModelTrainMetrics;
import com.kamco.cd.training.model.dto.ModelTrainDetailDto.ModelValidationMetrics;
import com.kamco.cd.training.model.dto.ModelTrainDetailDto.TransferHyperSummary;
import com.kamco.cd.training.postgres.entity.ModelMasterEntity;
import com.kamco.cd.training.postgres.entity.QModelHyperParamEntity;
import com.kamco.cd.training.postgres.entity.QModelMasterEntity;
import com.querydsl.core.types.Projections;
import com.querydsl.jpa.JPAExpressions;
import com.querydsl.jpa.impl.JPAQueryFactory;
import java.util.List;
import java.util.Optional;
import java.util.UUID;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Repository;
@Slf4j
@Repository
@RequiredArgsConstructor
public class ModelDetailRepositoryImpl implements ModelDetailRepositoryCustom {
private final JPAQueryFactory queryFactory;
/**
* 모델 조회
*
* @param uuid
* @return
*/
@Override
public Optional<ModelMasterEntity> findByUuid(UUID uuid) {
return Optional.ofNullable(
queryFactory
.select(modelMasterEntity)
.from(modelMasterEntity)
.where(modelMasterEntity.uuid.eq(uuid))
.fetchOne());
}
@Override
public DetailSummary getModelDetailSummary(UUID uuid) {
return queryFactory
.select(
Projections.constructor(
DetailSummary.class,
modelMasterEntity.id,
modelMasterEntity.uuid,
modelMasterEntity.modelNo,
modelMasterEntity.modelVer,
modelMasterEntity.step1StrtDttm,
modelMasterEntity.step2EndDttm,
modelMasterEntity.statusCd,
modelMasterEntity.trainType))
.from(modelMasterEntity)
.where(modelMasterEntity.uuid.eq(uuid))
.fetchOne();
}
@Override
public HyperSummary getByModelHyperParamSummary(UUID uuid) {
return queryFactory
.select(
Projections.constructor(
HyperSummary.class,
modelHyperParamEntity.uuid,
modelHyperParamEntity.id,
modelHyperParamEntity.hyperVer,
modelHyperParamEntity.backbone,
modelHyperParamEntity.inputSize,
modelHyperParamEntity.cropSize,
modelHyperParamEntity.batchSize))
.from(modelHyperParamEntity)
.where(
modelHyperParamEntity.id.eq(
JPAExpressions.select(modelMasterEntity.hyperParamId)
.from(modelMasterEntity)
.where(modelMasterEntity.uuid.eq(uuid))))
.fetchOne();
}
@Override
public TransferHyperSummary getByModelTransferHyperParamSummary(UUID uuid) {
QModelMasterEntity subMaster = new QModelMasterEntity("subMaster");
QModelHyperParamEntity subHyper = new QModelHyperParamEntity("subHyper");
return queryFactory
.select(
Projections.constructor(
TransferHyperSummary.class,
modelHyperParamEntity.uuid,
modelHyperParamEntity.id,
modelHyperParamEntity.hyperVer,
modelHyperParamEntity.backbone,
modelHyperParamEntity.inputSize,
modelHyperParamEntity.cropSize,
modelHyperParamEntity.batchSize,
subHyper.uuid,
subHyper.id,
subHyper.hyperVer,
subHyper.backbone,
subHyper.inputSize,
subHyper.cropSize,
subHyper.batchSize))
.from(modelMasterEntity)
.innerJoin(modelHyperParamEntity)
.on(modelHyperParamEntity.id.eq(modelMasterEntity.hyperParamId))
.leftJoin(subMaster)
.on(subMaster.id.eq(modelMasterEntity.beforeModelId))
.leftJoin(subHyper)
.on(subHyper.id.eq(subMaster.hyperParamId))
.where(modelMasterEntity.uuid.eq(uuid))
.fetchOne();
}
@Override
public List<MappingDataset> getByModelMappingDataset(UUID uuid) {
return queryFactory
.select(
Projections.constructor(
MappingDataset.class,
modelMasterEntity.id,
datasetEntity.id,
datasetEntity.dataType,
datasetEntity.compareYyyy,
datasetEntity.targetYyyy,
datasetEntity.roundNo,
modelDatasetEntity.buildingCnt,
modelDatasetEntity.containerCnt,
modelDatasetEntity.wasteCnt,
modelDatasetEntity.landCoverCnt))
.from(modelMasterEntity)
.innerJoin(modelDatasetEntity)
.on(modelMasterEntity.id.eq(modelDatasetEntity.model.id))
.innerJoin(modelDatasetMappEntity)
.on(modelMasterEntity.id.eq(modelDatasetMappEntity.modelUid))
.innerJoin(datasetEntity)
.on(modelDatasetMappEntity.datasetUid.eq(datasetEntity.id))
.where(modelMasterEntity.uuid.eq(uuid))
.fetch();
}
@Override
public ModelMasterEntity findByModelByUUID(UUID uuid) {
return queryFactory
.selectFrom(modelMasterEntity)
.where(modelMasterEntity.uuid.eq(uuid))
.fetchOne();
}
@Override
public List<ModelTrainMetrics> getModelTrainMetricResult(UUID uuid) {
ModelMasterEntity modelMasterEntity = findByModelByUUID(uuid);
if (modelMasterEntity == null) {
return List.of();
}
return queryFactory
.select(
Projections.constructor(
ModelTrainMetrics.class,
modelMetricsTrainEntity.epoch,
modelMetricsTrainEntity.iteration,
modelMetricsTrainEntity.loss,
modelMetricsTrainEntity.lr,
modelMetricsTrainEntity.durationTime))
.from(modelMetricsTrainEntity)
.where(modelMetricsTrainEntity.model.id.eq(modelMasterEntity.getId()))
.fetch();
}
@Override
public List<ModelValidationMetrics> getModelValidationMetricResult(UUID uuid) {
ModelMasterEntity modelMasterEntity = findByModelByUUID(uuid);
if (modelMasterEntity == null) {
return List.of();
}
return queryFactory
.select(
Projections.constructor(
ModelValidationMetrics.class,
modelMetricsValidationEntity.epoch,
modelMetricsValidationEntity.aAcc,
modelMetricsValidationEntity.mFscore,
modelMetricsValidationEntity.mPrecision,
modelMetricsValidationEntity.mRecall,
modelMetricsValidationEntity.mIou,
modelMetricsValidationEntity.mAcc,
modelMetricsValidationEntity.changedFscore,
modelMetricsValidationEntity.changedPrecision,
modelMetricsValidationEntity.changedRecall,
modelMetricsValidationEntity.unchangedFscore,
modelMetricsValidationEntity.unchangedPrecision,
modelMetricsValidationEntity.unchangedRecall))
.from(modelMetricsValidationEntity)
.where(modelMetricsValidationEntity.model.id.eq(modelMasterEntity.getId()))
.fetch();
}
@Override
public List<ModelTestMetrics> getModelTestMetricResult(UUID uuid) {
ModelMasterEntity modelMasterEntity = findByModelByUUID(uuid);
if (modelMasterEntity == null) {
return List.of();
}
return queryFactory
.select(
Projections.constructor(
ModelTestMetrics.class,
modelMetricsTestEntity.model1,
modelMetricsTestEntity.tp,
modelMetricsTestEntity.fp,
modelMetricsTestEntity.fn,
modelMetricsTestEntity.precisions,
modelMetricsTestEntity.recall,
modelMetricsTestEntity.f1Score,
modelMetricsTestEntity.accuracy,
modelMetricsTestEntity.iou,
modelMetricsTestEntity.detectionCount,
modelMetricsTestEntity.gtCount))
.from(modelMetricsTestEntity)
.where(modelMetricsTestEntity.model.id.eq(modelMasterEntity.getId()))
.fetch();
}
@Override
public ModelBestEpoch getModelTrainBestEpoch(UUID uuid) {
ModelMasterEntity modelMasterEntity = findByModelByUUID(uuid);
if (modelMasterEntity == null) {
return null;
}
return queryFactory
.select(
Projections.constructor(
ModelBestEpoch.class,
modelMetricsTrainEntity.epoch,
modelMetricsTrainEntity.loss,
modelMetricsValidationEntity.mFscore,
modelMetricsValidationEntity.mPrecision,
modelMetricsValidationEntity.mRecall,
modelMetricsValidationEntity.mIou,
modelMetricsValidationEntity.mAcc))
.from(modelMetricsTrainEntity)
.leftJoin(modelMetricsValidationEntity)
.on(
modelMetricsTrainEntity.model.eq(modelMetricsValidationEntity.model),
modelMetricsTrainEntity.epoch.eq(modelMetricsValidationEntity.epoch))
.where(
modelMetricsTrainEntity.model.id.eq(modelMasterEntity.getId()),
modelMetricsTrainEntity.epoch.eq(modelMasterEntity.getBestEpoch()))
.fetchOne();
}
}

View File

@@ -1,17 +1,7 @@
package com.kamco.cd.training.postgres.repository.model;
import com.kamco.cd.training.postgres.entity.ModelTrainMasterEntity;
import java.util.List;
import java.util.Optional;
import java.util.UUID;
import com.kamco.cd.training.postgres.entity.ModelMasterEntity;
import org.springframework.data.jpa.repository.JpaRepository;
public interface ModelMngRepository
extends JpaRepository<ModelTrainMasterEntity, Long>, ModelMngRepositoryCustom {
List<ModelTrainMasterEntity> findByDelYnOrderByCreatedDttmDesc(Boolean delYn);
Optional<ModelTrainMasterEntity> findFirstByStatusCdAndDelYn(String statusCd, Boolean delYn);
Optional<ModelTrainMasterEntity> findByUuid(UUID uuid);
}
extends JpaRepository<ModelMasterEntity, Long>, ModelMngRepositoryCustom {}

View File

@@ -1,7 +1,10 @@
package com.kamco.cd.training.postgres.repository.model;
import com.kamco.cd.training.model.dto.ModelMngDto;
import com.kamco.cd.training.postgres.entity.ModelTrainMasterEntity;
import com.kamco.cd.training.model.dto.ModelTrainMngDto;
import com.kamco.cd.training.postgres.entity.ModelMasterEntity;
import com.kamco.cd.training.train.dto.TrainRunRequest;
import java.util.Optional;
import java.util.UUID;
import org.springframework.data.domain.Page;
public interface ModelMngRepositoryCustom {
@@ -12,5 +15,13 @@ public interface ModelMngRepositoryCustom {
* @param searchReq
* @return
*/
Page<ModelTrainMasterEntity> findByModels(ModelMngDto.SearchReq searchReq);
Page<ModelMasterEntity> findByModels(ModelTrainMngDto.SearchReq searchReq);
Optional<ModelMasterEntity> findByUuid(UUID uuid);
Optional<ModelMasterEntity> findFirstByStatusCdAndDelYn(String statusCd, Boolean delYn);
TrainRunRequest findTrainRunRequest(Long modelId);
Long findModelStep1InProgressCnt();
}

View File

@@ -1,12 +1,20 @@
package com.kamco.cd.training.postgres.repository.model;
import com.kamco.cd.training.model.dto.ModelMngDto;
import com.kamco.cd.training.postgres.entity.ModelTrainMasterEntity;
import com.kamco.cd.training.postgres.entity.QModelTrainMasterEntity;
import static com.kamco.cd.training.postgres.entity.QModelConfigEntity.modelConfigEntity;
import static com.kamco.cd.training.postgres.entity.QModelHyperParamEntity.modelHyperParamEntity;
import static com.kamco.cd.training.postgres.entity.QModelMasterEntity.modelMasterEntity;
import com.kamco.cd.training.common.enums.TrainStatusType;
import com.kamco.cd.training.model.dto.ModelTrainMngDto;
import com.kamco.cd.training.postgres.entity.ModelMasterEntity;
import com.kamco.cd.training.train.dto.TrainRunRequest;
import com.querydsl.core.BooleanBuilder;
import com.querydsl.core.types.Projections;
import com.querydsl.core.types.dsl.Expressions;
import com.querydsl.jpa.impl.JPAQueryFactory;
import java.util.List;
import java.util.Optional;
import java.util.UUID;
import lombok.RequiredArgsConstructor;
import org.springframework.data.domain.Page;
import org.springframework.data.domain.PageImpl;
@@ -18,43 +26,135 @@ import org.springframework.stereotype.Repository;
public class ModelMngRepositoryImpl implements ModelMngRepositoryCustom {
private final JPAQueryFactory queryFactory;
private final QModelTrainMasterEntity modelMng = QModelTrainMasterEntity.modelTrainMasterEntity;
/**
* 모델 목록 조회
*
* @param searchReq
* @param req
* @return
*/
@Override
public Page<ModelTrainMasterEntity> findByModels(ModelMngDto.SearchReq searchReq) {
Pageable pageable = searchReq.toPageable();
public Page<ModelMasterEntity> findByModels(ModelTrainMngDto.SearchReq req) {
Pageable pageable = req.toPageable();
BooleanBuilder builder = new BooleanBuilder();
//
// if (StringUtils.isNotBlank(searchReq.getStatus())) {
// builder.and(modelMng.statusCd.eq(searchReq.getStatus()));
// }
// Entity 직접 조회 (Projections 사용 지양)
List<ModelTrainMasterEntity> content =
if (req.getStatus() != null && !req.getStatus().isEmpty()) {
builder.and(modelMasterEntity.statusCd.eq(req.getStatus()));
}
if (req.getModelNo() != null && !req.getModelNo().isEmpty()) {
builder.and(modelMasterEntity.modelNo.eq(req.getModelNo()));
}
builder.and(modelMasterEntity.delYn.isFalse());
List<ModelMasterEntity> content =
queryFactory
.selectFrom(modelMng)
.where(builder.and(modelMng.delYn.isFalse()))
.selectFrom(modelMasterEntity)
.where(builder)
.offset(pageable.getOffset())
.limit(pageable.getPageSize())
.orderBy(modelMng.createdDttm.desc())
.orderBy(modelMasterEntity.createdDttm.desc())
.fetch();
// Count 쿼리 별도 실행 (null safe handling)
long total =
Optional.ofNullable(
queryFactory
.select(modelMng.count())
.from(modelMng)
.where(builder.and(modelMng.delYn.isFalse()))
.fetchOne())
.orElse(0L);
Long total =
queryFactory
.select(modelMasterEntity.count())
.from(modelMasterEntity)
.where(builder)
.fetchOne();
return new PageImpl<>(content, pageable, total);
long totalCount = (total != null) ? total : 0L;
return new PageImpl<>(content, pageable, totalCount);
}
/**
* 모델 조회
*
* @param uuid
* @return
*/
@Override
public Optional<ModelMasterEntity> findByUuid(UUID uuid) {
return Optional.ofNullable(
queryFactory
.select(modelMasterEntity)
.from(modelMasterEntity)
.where(modelMasterEntity.uuid.eq(uuid))
.fetchOne());
}
@Override
public Optional<ModelMasterEntity> findFirstByStatusCdAndDelYn(String statusCd, Boolean delYn) {
return Optional.empty();
}
@Override
public TrainRunRequest findTrainRunRequest(Long modelId) {
return queryFactory
.select(
Projections.constructor(
TrainRunRequest.class,
modelMasterEntity.requestPath, // datasetFolder
modelMasterEntity.uuid, // outputFolder
modelHyperParamEntity.inputSize,
modelHyperParamEntity.cropSize,
modelHyperParamEntity.batchSize,
modelHyperParamEntity.gpuIds,
modelHyperParamEntity.gpuCnt,
modelHyperParamEntity.learningRate,
modelHyperParamEntity.backbone,
modelConfigEntity.epochCount,
modelHyperParamEntity.trainNumWorkers,
modelHyperParamEntity.valNumWorkers,
modelHyperParamEntity.testNumWorkers,
modelHyperParamEntity.trainShuffle,
modelHyperParamEntity.trainPersistent,
modelHyperParamEntity.valPersistent,
modelHyperParamEntity.dropPathRate,
modelHyperParamEntity.frozenStages,
modelHyperParamEntity.neckPolicy,
modelHyperParamEntity.classWeight,
modelHyperParamEntity.decoderChannels,
modelHyperParamEntity.weightDecay,
modelHyperParamEntity.layerDecayRate,
modelHyperParamEntity.ignoreIndex,
modelHyperParamEntity.ddpFindUnusedParams,
modelHyperParamEntity.numLayers,
modelHyperParamEntity.metrics,
modelHyperParamEntity.saveBest,
modelHyperParamEntity.saveBestRule,
modelHyperParamEntity.valInterval,
modelHyperParamEntity.logInterval,
modelHyperParamEntity.visInterval,
modelHyperParamEntity.rotProb,
modelHyperParamEntity.rotDegree,
modelHyperParamEntity.flipProb,
modelHyperParamEntity.exchangeProb,
modelHyperParamEntity.brightnessDelta,
modelHyperParamEntity.contrastRange,
modelHyperParamEntity.saturationRange,
modelHyperParamEntity.hueDelta,
Expressions.nullExpression(Integer.class),
Expressions.nullExpression(String.class),
modelHyperParamEntity.uuid))
.from(modelMasterEntity)
.leftJoin(modelHyperParamEntity)
.on(modelHyperParamEntity.id.eq(modelMasterEntity.hyperParamId))
.leftJoin(modelConfigEntity)
.on(modelConfigEntity.model.id.eq(modelMasterEntity.id))
.where(modelMasterEntity.id.eq(modelId))
.fetchOne();
}
@Override
public Long findModelStep1InProgressCnt() {
return queryFactory
.select(modelMasterEntity.id.count())
.from(modelMasterEntity)
.where(modelMasterEntity.step1State.eq(TrainStatusType.IN_PROGRESS.getId()))
.fetchOne();
}
}

View File

@@ -0,0 +1,9 @@
package com.kamco.cd.training.postgres.repository.train;
import com.kamco.cd.training.postgres.entity.ModelMetricsTestEntity;
import org.springframework.data.jpa.repository.JpaRepository;
import org.springframework.stereotype.Repository;
@Repository
public interface ModelTestMetricsJobRepository
extends JpaRepository<ModelMetricsTestEntity, Long>, ModelTestMetricsJobRepositoryCustom {}

View File

@@ -0,0 +1,13 @@
package com.kamco.cd.training.postgres.repository.train;
import com.kamco.cd.training.train.dto.ModelTrainMetricsDto.ResponsePathDto;
import java.util.List;
public interface ModelTestMetricsJobRepositoryCustom {
void updateModelMetricsTrainSaveYn(Long modelId, String stepNo);
List<ResponsePathDto> getTestMetricSaveNotYetModelIds();
void insertModelMetricsTest(List<Object[]> batchArgs);
}

View File

@@ -0,0 +1,70 @@
package com.kamco.cd.training.postgres.repository.train;
import static com.kamco.cd.training.postgres.entity.QModelMasterEntity.modelMasterEntity;
import com.kamco.cd.training.common.enums.TrainStatusType;
import com.kamco.cd.training.postgres.entity.ModelMetricsTestEntity;
import com.kamco.cd.training.train.dto.ModelTrainMetricsDto.ResponsePathDto;
import com.querydsl.core.types.Projections;
import com.querydsl.jpa.impl.JPAQueryFactory;
import java.util.List;
import org.springframework.data.jpa.repository.support.QuerydslRepositorySupport;
import org.springframework.jdbc.core.JdbcTemplate;
public class ModelTestMetricsJobRepositoryImpl extends QuerydslRepositorySupport
implements ModelTestMetricsJobRepositoryCustom {
private final JPAQueryFactory queryFactory;
private final JdbcTemplate jdbcTemplate;
public ModelTestMetricsJobRepositoryImpl(
JPAQueryFactory queryFactory, JdbcTemplate jdbcTemplate) {
super(ModelMetricsTestEntity.class);
this.queryFactory = queryFactory;
this.jdbcTemplate = jdbcTemplate;
}
@Override
public void updateModelMetricsTrainSaveYn(Long modelId, String stepNo) {
queryFactory
.update(modelMasterEntity)
.set(
stepNo.equals("step1")
? modelMasterEntity.step1MetricSaveYn
: modelMasterEntity.step2MetricSaveYn,
true)
.where(modelMasterEntity.id.eq(modelId))
.execute();
}
@Override
public List<ResponsePathDto> getTestMetricSaveNotYetModelIds() {
return queryFactory
.select(
Projections.constructor(
ResponsePathDto.class, modelMasterEntity.id, modelMasterEntity.responsePath))
.from(modelMasterEntity)
.where(
modelMasterEntity.step2EndDttm.isNotNull(),
modelMasterEntity.step2State.eq(TrainStatusType.COMPLETED.getId()),
modelMasterEntity
.step2MetricSaveYn
.isNull()
.or(modelMasterEntity.step2MetricSaveYn.isFalse()))
.fetch();
}
@Override
public void insertModelMetricsTest(List<Object[]> batchArgs) {
String sql =
"""
insert into tb_model_metrics_test
(model_id, model, tp, fp, fn, precisions, recall, f1_score, accuracy, iou,
detection_count, gt_count
)
values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
""";
jdbcTemplate.batchUpdate(sql, batchArgs);
}
}

View File

@@ -0,0 +1,7 @@
package com.kamco.cd.training.postgres.repository.train;
import com.kamco.cd.training.postgres.entity.ModelTrainJobEntity;
import org.springframework.data.jpa.repository.JpaRepository;
public interface ModelTrainJobRepository
extends JpaRepository<ModelTrainJobEntity, Long>, ModelTrainJobRepositoryCustom {}

View File

@@ -0,0 +1,10 @@
package com.kamco.cd.training.postgres.repository.train;
import com.kamco.cd.training.postgres.entity.ModelTrainJobEntity;
import java.util.Optional;
public interface ModelTrainJobRepositoryCustom {
int findMaxAttemptNo(Long modelId);
Optional<ModelTrainJobEntity> findLatestByModelId(Long modelId);
}

View File

@@ -0,0 +1,43 @@
package com.kamco.cd.training.postgres.repository.train;
import com.kamco.cd.training.postgres.entity.ModelTrainJobEntity;
import com.kamco.cd.training.postgres.entity.QModelTrainJobEntity;
import com.querydsl.jpa.impl.JPAQueryFactory;
import jakarta.persistence.EntityManager;
import java.util.Optional;
import org.springframework.stereotype.Repository;
@Repository
public class ModelTrainJobRepositoryImpl implements ModelTrainJobRepositoryCustom {
private final JPAQueryFactory queryFactory;
public ModelTrainJobRepositoryImpl(EntityManager em) {
this.queryFactory = new JPAQueryFactory(em);
}
/** modelId의 attempt_no 최대값. (없으면 0) */
@Override
public int findMaxAttemptNo(Long modelId) {
QModelTrainJobEntity j = QModelTrainJobEntity.modelTrainJobEntity;
Integer max =
queryFactory.select(j.attemptNo.max()).from(j).where(j.modelId.eq(modelId)).fetchOne();
return max != null ? max : 0;
}
/**
* modelId의 최신 job 1건 (보통 id desc / queuedDttm desc 등) - attemptNo 기준으로도 가능하지만, 여기선 id desc가 가장
* 단순.
*/
@Override
public Optional<ModelTrainJobEntity> findLatestByModelId(Long modelId) {
QModelTrainJobEntity j = QModelTrainJobEntity.modelTrainJobEntity;
ModelTrainJobEntity job =
queryFactory.selectFrom(j).where(j.modelId.eq(modelId)).orderBy(j.id.desc()).fetchFirst();
return Optional.ofNullable(job);
}
}

View File

@@ -0,0 +1,9 @@
package com.kamco.cd.training.postgres.repository.train;
import com.kamco.cd.training.postgres.entity.ModelMetricsTrainEntity;
import org.springframework.data.jpa.repository.JpaRepository;
import org.springframework.stereotype.Repository;
@Repository
public interface ModelTrainMetricsJobRepository
extends JpaRepository<ModelMetricsTrainEntity, Long>, ModelTrainMetricsJobRepositoryCustom {}

View File

@@ -0,0 +1,15 @@
package com.kamco.cd.training.postgres.repository.train;
import com.kamco.cd.training.train.dto.ModelTrainMetricsDto.ResponsePathDto;
import java.util.List;
public interface ModelTrainMetricsJobRepositoryCustom {
List<ResponsePathDto> getTrainMetricSaveNotYetModelIds();
void insertModelMetricsTrain(List<Object[]> batchArgs);
void updateModelMetricsTrainSaveYn(Long modelId, String stepNo);
void insertModelMetricsValidation(List<Object[]> batchArgs);
}

View File

@@ -0,0 +1,82 @@
package com.kamco.cd.training.postgres.repository.train;
import static com.kamco.cd.training.postgres.entity.QModelMasterEntity.modelMasterEntity;
import com.kamco.cd.training.common.enums.TrainStatusType;
import com.kamco.cd.training.postgres.entity.ModelMetricsTrainEntity;
import com.kamco.cd.training.train.dto.ModelTrainMetricsDto.ResponsePathDto;
import com.querydsl.core.types.Projections;
import com.querydsl.jpa.impl.JPAQueryFactory;
import java.util.List;
import org.springframework.data.jpa.repository.support.QuerydslRepositorySupport;
import org.springframework.jdbc.core.JdbcTemplate;
public class ModelTrainMetricsJobRepositoryImpl extends QuerydslRepositorySupport
implements ModelTrainMetricsJobRepositoryCustom {
private final JPAQueryFactory queryFactory;
private final JdbcTemplate jdbcTemplate;
public ModelTrainMetricsJobRepositoryImpl(
JPAQueryFactory queryFactory, JdbcTemplate jdbcTemplate) {
super(ModelMetricsTrainEntity.class);
this.queryFactory = queryFactory;
this.jdbcTemplate = jdbcTemplate;
}
@Override
public List<ResponsePathDto> getTrainMetricSaveNotYetModelIds() {
return queryFactory
.select(
Projections.constructor(
ResponsePathDto.class, modelMasterEntity.id, modelMasterEntity.responsePath))
.from(modelMasterEntity)
.where(
modelMasterEntity.step1EndDttm.isNotNull(),
modelMasterEntity.step1State.eq(TrainStatusType.COMPLETED.getId()),
modelMasterEntity
.step1MetricSaveYn
.isNull()
.or(modelMasterEntity.step1MetricSaveYn.isFalse()))
.fetch();
}
@Override
public void insertModelMetricsTrain(List<Object[]> batchArgs) {
String sql =
"""
insert into tb_model_metrics_train
(model_id, epoch, iteration, loss, lr, duration_time)
values (?, ?, ?, ?, ?, ?)
""";
jdbcTemplate.batchUpdate(sql, batchArgs);
}
@Override
public void updateModelMetricsTrainSaveYn(Long modelId, String stepNo) {
queryFactory
.update(modelMasterEntity)
.set(
stepNo.equals("step1")
? modelMasterEntity.step1MetricSaveYn
: modelMasterEntity.step2MetricSaveYn,
true)
.where(modelMasterEntity.id.eq(modelId))
.execute();
}
@Override
public void insertModelMetricsValidation(List<Object[]> batchArgs) {
String sql =
"""
insert into tb_model_metrics_validation
(model_id, epoch, a_acc, m_fscore, m_precision, m_recall, m_iou, m_acc, changed_fscore, changed_precision, changed_recall,
unchanged_fscore, unchanged_precision, unchanged_recall
)
values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
""";
jdbcTemplate.batchUpdate(sql, batchArgs);
}
}

View File

@@ -1,12 +1,13 @@
package com.kamco.cd.training.postgres.repository.upload;
import com.kamco.cd.training.upload.dto.UploadDto;
import java.util.UUID;
public interface UploadSessionRepositoryCustom {
void insertUploadSession(UploadDto.UploadAddReq addReq);
UploadDto.uploadDto findByDatasetUid(Long datasetId, String uploadDivi);
UploadDto.uploadDto findByDatasetUid(String uploadDivi, UUID uuid);
UploadDto.uploadDto findByUuid(String uuid);

View File

@@ -61,7 +61,7 @@ public class UploadSessionRepositoryImpl extends QuerydslRepositorySupport
}
@Override
public UploadDto.uploadDto findByDatasetUid(Long datasetId, String uploadDivi) {
public UploadDto.uploadDto findByDatasetUid(String uploadDivi, UUID uuid) {
UploadDto.uploadDto foundContent =
queryFactory
@@ -83,9 +83,9 @@ public class UploadSessionRepositoryImpl extends QuerydslRepositorySupport
.from(uploadSessionEntity)
.where(
uploadSessionEntity
.datasetId
.eq(datasetId)
.and(uploadSessionEntity.uploadDivi.eq(uploadDivi)))
.uploadDivi
.eq(uploadDivi)
.and(uploadSessionEntity.uuid.eq(uuid)))
.limit(1)
.fetchOne();

View File

@@ -0,0 +1,189 @@
package com.kamco.cd.training.train;
import com.kamco.cd.training.config.api.ApiResponseDto;
import com.kamco.cd.training.train.service.TestJobService;
import com.kamco.cd.training.train.service.TrainJobService;
import io.swagger.v3.oas.annotations.Operation;
import io.swagger.v3.oas.annotations.Parameter;
import io.swagger.v3.oas.annotations.media.Content;
import io.swagger.v3.oas.annotations.media.Schema;
import io.swagger.v3.oas.annotations.responses.ApiResponse;
import io.swagger.v3.oas.annotations.responses.ApiResponses;
import io.swagger.v3.oas.annotations.tags.Tag;
import java.util.UUID;
import lombok.RequiredArgsConstructor;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
@Tag(name = "학습 실행 API", description = "모델학습관리 > 학습 실행 API")
@RequiredArgsConstructor
@RestController
@RequestMapping("/api/train")
public class TrainApiController {
private final TrainJobService trainJobService;
private final TestJobService testJobService;
@Operation(summary = "학습 실행", description = "학습 실행 API")
@ApiResponses(
value = {
@ApiResponse(
responseCode = "200",
description = "실행 성공",
content =
@Content(
mediaType = "application/json",
schema = @Schema(implementation = String.class))),
@ApiResponse(responseCode = "400", description = "잘못된 검색 조건", content = @Content),
@ApiResponse(responseCode = "500", description = "서버 오류", content = @Content)
})
@PostMapping("/run/{uuid}")
public ApiResponseDto<String> run(
@Parameter(description = "uuid", example = "80a0e544-36ed-4999-b705-97427f23337d")
@PathVariable
UUID uuid) {
Long modelId = trainJobService.getModelIdByUuid(uuid);
trainJobService.enqueue(modelId);
return ApiResponseDto.ok("ok");
}
@Operation(summary = "학습 재실행", description = "학습 재실행 API")
@ApiResponses(
value = {
@ApiResponse(
responseCode = "200",
description = "재실행 성공",
content =
@Content(
mediaType = "application/json",
schema = @Schema(implementation = String.class))),
@ApiResponse(responseCode = "400", description = "잘못된 검색 조건", content = @Content),
@ApiResponse(responseCode = "500", description = "서버 오류", content = @Content)
})
@PostMapping("/restart/{uuid}")
public ApiResponseDto<String> restart(
@Parameter(description = "uuid", example = "80a0e544-36ed-4999-b705-97427f23337d")
@PathVariable
UUID uuid) {
Long modelId = trainJobService.getModelIdByUuid(uuid);
Long jobId = trainJobService.restart(modelId);
return ApiResponseDto.ok("ok");
}
@Operation(summary = "학습 이어하기", description = "학습 이어하기 API")
@ApiResponses(
value = {
@ApiResponse(
responseCode = "200",
description = "이어하기 성공",
content =
@Content(
mediaType = "application/json",
schema = @Schema(implementation = String.class))),
@ApiResponse(responseCode = "400", description = "잘못된 검색 조건", content = @Content),
@ApiResponse(responseCode = "500", description = "서버 오류", content = @Content)
})
@PostMapping("/resume/{uuid}")
public ApiResponseDto<String> resume(
@Parameter(description = "uuid", example = "80a0e544-36ed-4999-b705-97427f23337d")
@PathVariable
UUID uuid) {
Long modelId = trainJobService.getModelIdByUuid(uuid);
Long jobId = trainJobService.resume(modelId);
return ApiResponseDto.ok("ok");
}
@Operation(summary = "학습 취소", description = "학습 취소 API")
@ApiResponses(
value = {
@ApiResponse(
responseCode = "200",
description = "취소 성공",
content =
@Content(
mediaType = "application/json",
schema = @Schema(implementation = String.class))),
@ApiResponse(responseCode = "400", description = "잘못된 검색 조건", content = @Content),
@ApiResponse(responseCode = "500", description = "서버 오류", content = @Content)
})
@PostMapping("/cancel/{uuid}")
public ApiResponseDto<String> cancel(
@Parameter(description = "uuid", example = "80a0e544-36ed-4999-b705-97427f23337d")
@PathVariable
UUID uuid) {
Long modelId = trainJobService.getModelIdByUuid(uuid);
trainJobService.cancel(modelId);
return ApiResponseDto.ok("ok");
}
@Operation(summary = "test 실행", description = "test 실행 API")
@ApiResponses(
value = {
@ApiResponse(
responseCode = "200",
description = "test 성공",
content =
@Content(
mediaType = "application/json",
schema = @Schema(implementation = String.class))),
@ApiResponse(responseCode = "400", description = "잘못된 검색 조건", content = @Content),
@ApiResponse(responseCode = "500", description = "서버 오류", content = @Content)
})
@PostMapping("/test/run/{epoch}/{uuid}")
public ApiResponseDto<String> run(
@Parameter(description = "best 에폭", example = "1") @PathVariable int epoch,
@Parameter(description = "uuid", example = "80a0e544-36ed-4999-b705-97427f23337d")
@PathVariable
UUID uuid) {
Long modelId = trainJobService.getModelIdByUuid(uuid);
testJobService.enqueue(modelId, uuid, epoch);
return ApiResponseDto.ok("ok");
}
@Operation(summary = "test 학습 취소", description = "학습 취소 API")
@ApiResponses(
value = {
@ApiResponse(
responseCode = "200",
description = "취소 성공",
content =
@Content(
mediaType = "application/json",
schema = @Schema(implementation = String.class))),
@ApiResponse(responseCode = "400", description = "잘못된 검색 조건", content = @Content),
@ApiResponse(responseCode = "500", description = "서버 오류", content = @Content)
})
@PostMapping("/test/cancel/{uuid}")
public ApiResponseDto<String> cancelTest(
@Parameter(description = "uuid", example = "80a0e544-36ed-4999-b705-97427f23337d")
@PathVariable
UUID uuid) {
Long modelId = trainJobService.getModelIdByUuid(uuid);
testJobService.cancel(modelId);
return ApiResponseDto.ok("ok");
}
@Operation(summary = "데이터셋 tmp 파일생성", description = "데이터셋 tmp 파일생성 API")
@ApiResponses(
value = {
@ApiResponse(
responseCode = "200",
description = "데이터셋 tmp 파일생성 성공",
content =
@Content(
mediaType = "application/json",
schema = @Schema(implementation = String.class))),
@ApiResponse(responseCode = "400", description = "잘못된 검색 조건", content = @Content),
@ApiResponse(responseCode = "500", description = "서버 오류", content = @Content)
})
@PostMapping("/create-tmp/{uuid}")
public ApiResponseDto<UUID> createTmpFile(
@Parameter(description = "uuid", example = "80a0e544-36ed-4999-b705-97427f23337d")
@PathVariable
UUID uuid) {
return ApiResponseDto.ok(trainJobService.createTmpFile(uuid));
}
}

View File

@@ -0,0 +1,16 @@
package com.kamco.cd.training.train.dto;
import lombok.AllArgsConstructor;
import lombok.Getter;
import lombok.NoArgsConstructor;
import lombok.Setter;
@Getter
@Setter
@AllArgsConstructor
@NoArgsConstructor
public class EvalRunRequest {
private String uuid;
private int epoch; // best_changed_fscore_epoch_1.pth
private Integer timeoutSeconds;
}

View File

@@ -0,0 +1,25 @@
package com.kamco.cd.training.train.dto;
import java.time.ZonedDateTime;
import java.util.Map;
import lombok.AllArgsConstructor;
import lombok.Getter;
@Getter
@AllArgsConstructor
public class ModelTrainJobDto {
private Long id;
private Long modelId;
private Integer attemptNo;
private String statusCd;
private Integer exitCode;
private String errorMessage;
private String containerName;
private Map<String, Object> paramsJson;
private ZonedDateTime queuedDttm;
private ZonedDateTime startedDttm;
private ZonedDateTime finishedDttm;
private Integer totalEpoch;
private Integer currentEpoch;
}

View File

@@ -0,0 +1,15 @@
package com.kamco.cd.training.train.dto;
/** 학습 실행이 예약되었음을 알리는 이벤트 객체 */
public class ModelTrainJobQueuedEvent {
private final Long jobId;
public ModelTrainJobQueuedEvent(Long jobId) {
this.jobId = jobId;
}
public Long getJobId() {
return jobId;
}
}

View File

@@ -0,0 +1,21 @@
package com.kamco.cd.training.train.dto;
import io.swagger.v3.oas.annotations.media.Schema;
import lombok.AllArgsConstructor;
import lombok.Getter;
import lombok.NoArgsConstructor;
import lombok.Setter;
public class ModelTrainMetricsDto {
@Schema(name = "ResponsePathDto", description = "AI 결과 저장된 path 경로 정보")
@Getter
@Setter
@NoArgsConstructor
@AllArgsConstructor
public static class ResponsePathDto {
private Long modelId;
private String responsePath;
}
}

View File

@@ -0,0 +1,94 @@
package com.kamco.cd.training.train.dto;
import java.util.UUID;
import lombok.AllArgsConstructor;
import lombok.Getter;
import lombok.NoArgsConstructor;
import lombok.Setter;
@Getter
@Setter
@NoArgsConstructor
@AllArgsConstructor
public class TrainRunRequest {
// ========================
// 기본
// ========================
private String datasetFolder;
private UUID outputFolder;
private String inputSize;
private String cropSize;
private Integer batchSize;
private String gpuIds;
private Integer gpus;
private Double learningRate;
private String backbone;
private Integer epochs;
// ========================
// Data
// ========================
private Integer trainNumWorkers;
private Integer valNumWorkers;
private Integer testNumWorkers;
private Boolean trainShuffle;
private Boolean trainPersistent;
private Boolean valPersistent;
// ========================
// Model Architecture
// ========================
private Double dropPathRate;
private Integer frozenStages;
private String neckPolicy;
private String classWeight;
private String decoderChannels;
// ========================
// Loss & Optimization
// ========================
private Double weightDecay;
private Double layerDecayRate;
private Integer ignoreIndex;
private Boolean ddpFindUnusedParams;
private Integer numLayers;
// ========================
// Evaluation
// ========================
private String metrics;
private String saveBest;
private String saveBestRule;
private Integer valInterval;
private Integer logInterval;
private Integer visInterval;
// ========================
// Augmentation
// ========================
private Double rotProb;
private String rotDegree;
private Double flipProb;
private Double exchangeProb;
private Integer brightnessDelta;
private String contrastRange;
private String saturationRange;
private Integer hueDelta;
// ========================
// 실행 타임아웃
// ========================
private Integer timeoutSeconds;
private String resumeFrom;
private UUID uuid;
public String getOutputFolder() {
return String.valueOf(this.outputFolder);
}
public String getUuid() {
return String.valueOf(this.uuid);
}
}

View File

@@ -0,0 +1,20 @@
package com.kamco.cd.training.train.dto;
import lombok.AllArgsConstructor;
import lombok.Getter;
import lombok.NoArgsConstructor;
import lombok.Setter;
/** 학습 실행 결과 반환 객체 */
@Getter
@Setter
@NoArgsConstructor
@AllArgsConstructor
public class TrainRunResult {
private String jobId;
private String containerName;
private int exitCode;
private String status;
private String logs;
}

View File

@@ -0,0 +1,406 @@
package com.kamco.cd.training.train.service;
import com.kamco.cd.training.train.dto.EvalRunRequest;
import com.kamco.cd.training.train.dto.TrainRunRequest;
import com.kamco.cd.training.train.dto.TrainRunResult;
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.TimeUnit;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import lombok.extern.log4j.Log4j2;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;
@Log4j2
@Service
public class DockerTrainService {
// 실행할 Docker 이미지명
@Value("${train.docker.image}")
private String image;
// 학습 요청 데이터가 위치한 호스트 디렉토리
@Value("${train.docker.requestDir}")
private String requestDir;
// 학습 결과가 저장될 호스트 디렉토리
@Value("${train.docker.responseDir}")
private String responseDir;
// 컨테이너 이름 prefix
@Value("${train.docker.containerPrefix}")
private String containerPrefix;
// 공유메모리 사이즈 설정 (대용량 학습시 필요)
@Value("${train.docker.shmSize:16g}")
private String shmSize;
// IPC host 사용 여부
@Value("${train.docker.ipcHost:true}")
private boolean ipcHost;
/** Docker 학습 컨테이너를 동기 실행 - 요청 스레드에서 docker run 실행 - 컨테이너 종료까지 대기 - stdout/stderr 로그 수집 후 결과 반환 */
public TrainRunResult runTrainSync(TrainRunRequest req, String containerName) throws Exception {
List<String> cmd = buildDockerRunCommand(containerName, req);
log.info("=== Docker Train Command ===");
log.info("Container: {}", containerName);
log.info("Command: {}", String.join(" ", cmd));
log.info("================================");
ProcessBuilder pb = new ProcessBuilder(cmd);
pb.redirectErrorStream(true);
Process p = pb.start();
// 로그는 별도 스레드에서 읽기 (메인 스레드가 readLine에 안 걸리게)
StringBuilder logBuilder = new StringBuilder();
Pattern epochPattern = Pattern.compile("(?i)\\bepoch\\s*\\[?(\\d+)\\s*/\\s*(\\d+)\\]?\\b");
Thread logThread =
new Thread(
() -> {
try (BufferedReader br =
new BufferedReader(
new InputStreamReader(p.getInputStream(), StandardCharsets.UTF_8))) {
String line;
while ((line = br.readLine()) != null) {
// 1) 로그 누적
synchronized (logBuilder) {
logBuilder.append(line).append('\n');
}
// 2) epoch 감지 + DB 업데이트
Matcher m = epochPattern.matcher(line);
if (m.find()) {
int currentEpoch = Integer.parseInt(m.group(1));
int totalEpoch = Integer.parseInt(m.group(2));
log.info("[EPOCH] container={} {}/{}", containerName, currentEpoch, totalEpoch);
// TODO 실행중인 에폭 저장 필요하면 만들어야함
// TODO 하지만 여기서 트랜젝션 걸리는 db 작업하면 안좋다고하는데..?
// modelTrainMngCoreService.updateCurrentEpoch(modelId,
// currentEpoch, totalEpoch);
}
}
} catch (Exception e) {
log.warn("logThread error: {}", e.toString());
}
},
"train-log-" + containerName);
// new Thread(
// () -> {
// try (BufferedReader br =
// new BufferedReader(
// new InputStreamReader(p.getInputStream(), StandardCharsets.UTF_8))) {
// String line;
// while ((line = br.readLine()) != null) {
// synchronized (log) {
// log.append(line).append('\n');
// }
// }
// } catch (Exception ignored) {
// }
// },
// "train-log-" + containerName);
logThread.setDaemon(true);
logThread.start();
int timeoutSeconds = req.getTimeoutSeconds() != null ? req.getTimeoutSeconds() : 7200;
boolean finished = p.waitFor(timeoutSeconds, TimeUnit.SECONDS);
if (!finished) {
// docker run 프로세스도 같이 끊어야 readLine이 풀림
p.destroy();
if (!p.waitFor(2, TimeUnit.SECONDS)) {
p.destroyForcibly();
}
killContainer(containerName);
String logs;
synchronized (logBuilder) {
logs = logBuilder.toString();
}
return new TrainRunResult(
null, // jobId (없으면 null)
containerName,
-1,
"TIMEOUT",
logs);
}
int exit = p.exitValue();
// 로그 스레드가 마무리할 시간을 조금 줌(없어도 되지만 로그 누락 방지용)
logThread.join(500);
String logs;
synchronized (logBuilder) {
logs = logBuilder.toString();
}
return new TrainRunResult(null, containerName, exit, exit == 0 ? "SUCCESS" : "FAILED", logs);
}
/**
* 학습 docker run command
*
* @param containerName
* @param req
* @return
*/
private List<String> buildDockerRunCommand(String containerName, TrainRunRequest req) {
List<String> c = new ArrayList<>();
c.add("docker");
c.add("run");
// 컨테이너 이름 지정
c.add("--name");
c.add(containerName + "-" + req.getUuid().substring(0, 8));
// 실행 종료 시 자동 삭제
c.add("--rm");
// GPU 전체 사용
c.add("--gpus");
c.add("all");
// IPC host 사용 여부
if (ipcHost) {
c.add("--ipc=host");
}
// 공유메모리 설정
c.add("--shm-size=" + shmSize);
// 메모리 관련 ulimit 설정
c.add("--ulimit");
c.add("memlock=-1");
c.add("--ulimit");
c.add("stack=67108864");
// 환경변수 설정
c.add("-e");
c.add("OPENCV_LOG_LEVEL=ERROR");
c.add("-e");
c.add("NCCL_DEBUG=INFO");
c.add("-e");
c.add("NCCL_IB_DISABLE=1");
c.add("-e");
c.add("NCCL_P2P_DISABLE=0");
c.add("-e");
c.add("NCCL_SOCKET_IFNAME=eth0");
// 요청/결과 디렉토리 볼륨 마운트
c.add("-v");
c.add(requestDir + ":/data");
c.add("-v");
c.add(responseDir + ":/checkpoints");
// 표준입력 유지 (-it 대신 -i만 사용)
c.add("-i");
// 사용할 이미지
c.add(image);
// ===== 컨테이너 내부 실행 명령 =====
c.add("python");
c.add("/workspace/change-detection-code/train_wrapper.py");
// ===== 기본 파라미터 =====
addArg(c, "--dataset-folder", req.getDatasetFolder());
addArg(c, "--output-folder", req.getOutputFolder());
addArg(c, "--input-size", req.getInputSize());
addArg(c, "--crop-size", req.getCropSize());
addArg(c, "--batch-size", req.getBatchSize());
addArg(c, "--gpu-ids", req.getGpuIds());
// addArg(c, "--gpus", req.getGpus());
addArg(c, "--lr", req.getLearningRate());
addArg(c, "--backbone", req.getBackbone());
addArg(c, "--epochs", req.getEpochs());
// ===== Data =====
addArg(c, "--train-num-workers", req.getTrainNumWorkers());
addArg(c, "--val-num-workers", req.getValNumWorkers());
addArg(c, "--test-num-workers", req.getTestNumWorkers());
addArg(c, "--train-shuffle", req.getTrainShuffle());
addArg(c, "--train-persistent", req.getTrainPersistent());
addArg(c, "--val-persistent", req.getValPersistent());
// ===== Model Architecture =====
addArg(c, "--drop-path-rate", req.getDropPathRate());
addArg(c, "--frozen-stages", req.getFrozenStages());
addArg(c, "--neck-policy", req.getNeckPolicy());
addArg(c, "--class-weight", req.getClassWeight());
addArg(c, "--decoder-channels", req.getDecoderChannels());
// ===== Loss & Optimization =====
addArg(c, "--weight-decay", req.getWeightDecay());
addArg(c, "--layer-decay-rate", req.getLayerDecayRate());
addArg(c, "--ignore-index", req.getIgnoreIndex());
addArg(c, "--ddp-find-unused-params", req.getDdpFindUnusedParams());
addArg(c, "--num-layers", req.getNumLayers());
// ===== Evaluation =====
addArg(c, "--metrics", req.getMetrics());
addArg(c, "--save-best", req.getSaveBest());
addArg(c, "--save-best-rule", req.getSaveBestRule());
addArg(c, "--val-interval", req.getValInterval());
addArg(c, "--log-interval", req.getLogInterval());
addArg(c, "--vis-interval", req.getVisInterval());
// ===== Augmentation =====
addArg(c, "--rot-prob", req.getRotProb());
addArg(c, "--rot-degree", req.getRotDegree());
addArg(c, "--flip-prob", req.getFlipProb());
addArg(c, "--exchange-prob", req.getExchangeProb());
addArg(c, "--brightness-delta", req.getBrightnessDelta());
addArg(c, "--contrast-range", req.getContrastRange());
addArg(c, "--saturation-range", req.getSaturationRange());
addArg(c, "--hue-delta", req.getHueDelta());
addArg(c, "--resume-from", req.getResumeFrom());
return c;
}
/** 인자 추가(키 + 값) - null / blank면 아예 추가 안 함 */
private void addArg(List<String> c, String key, Object value) {
if (value == null) return;
String s = String.valueOf(value).trim();
if (s.isEmpty()) return;
c.add(key + "=" + s);
}
/** 컨테이너 강제 종료 및 제거 */
public void killContainer(String containerName) {
try {
new ProcessBuilder("docker", "rm", "-f", containerName)
.redirectErrorStream(true)
.start()
.waitFor(10, TimeUnit.SECONDS);
} catch (Exception ignored) {
}
}
public TrainRunResult runEvalSync(EvalRunRequest req, String containerName) throws Exception {
List<String> cmd = buildDockerEvalCommand(containerName, req);
log.info("=== Docker Test Command ===");
log.info("Container: {}", containerName);
log.info("Command: {}", String.join(" ", cmd));
log.info("================================");
ProcessBuilder pb = new ProcessBuilder(cmd);
pb.redirectErrorStream(true);
Process p = pb.start();
StringBuilder log = new StringBuilder();
Thread logThread =
new Thread(
() -> {
try (BufferedReader br =
new BufferedReader(
new InputStreamReader(p.getInputStream(), StandardCharsets.UTF_8))) {
String line;
while ((line = br.readLine()) != null) {
synchronized (log) {
log.append(line).append('\n');
}
}
} catch (Exception ignored) {
}
});
logThread.setDaemon(true);
logThread.start();
int timeout = req.getTimeoutSeconds() != null ? req.getTimeoutSeconds() : 7200;
boolean finished = p.waitFor(timeout, TimeUnit.SECONDS);
if (!finished) {
p.destroyForcibly();
killContainer(containerName);
String logs;
synchronized (log) {
logs = log.toString();
}
return new TrainRunResult(null, containerName, -1, "TIMEOUT", logs);
}
int exit = p.exitValue();
logThread.join(500);
String logs;
synchronized (log) {
logs = log.toString();
}
return new TrainRunResult(null, containerName, exit, exit == 0 ? "SUCCESS" : "FAILED", logs);
}
/**
* 테스트 docker run command
*
* @param containerName
* @param req
* @return
*/
private List<String> buildDockerEvalCommand(String containerName, EvalRunRequest req) {
String uuid = req.getUuid();
Integer epoch = req.getEpoch();
if (uuid == null || uuid.isBlank()) throw new IllegalArgumentException("uuid is required");
if (epoch == null || epoch <= 0) throw new IllegalArgumentException("epoch must be > 0");
String modelFile = "best_changed_fscore_epoch_" + epoch + ".pth";
List<String> c = new ArrayList<>();
c.add("docker");
c.add("run");
c.add("--name");
c.add(containerName + "=" + req.getUuid().substring(0, 8));
c.add("--rm");
c.add("--gpus");
c.add("all");
if (ipcHost) c.add("--ipc=host");
c.add("--shm-size=" + shmSize);
c.add("-v");
c.add(requestDir + ":/data");
c.add("-v");
c.add(responseDir + ":/checkpoints");
c.add(image);
c.add("python");
c.add("/workspace/change-detection-code/run_evaluation_pipeline.py");
c.add("--dataset_dir");
c.add("/data/" + uuid);
c.add("--model");
c.add("/checkpoints/" + uuid + "/" + modelFile);
return c;
}
}

View File

@@ -0,0 +1,102 @@
package com.kamco.cd.training.train.service;
import com.kamco.cd.training.postgres.core.ModelTestMetricsJobCoreService;
import com.kamco.cd.training.train.dto.ModelTrainMetricsDto.ResponsePathDto;
import java.io.BufferedReader;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.List;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.csv.CSVFormat;
import org.apache.commons.csv.CSVParser;
import org.apache.commons.csv.CSVRecord;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;
@Slf4j
@Service
@RequiredArgsConstructor
public class ModelTestMetricsJobService {
private final ModelTestMetricsJobCoreService modelTestMetricsJobCoreService;
@Value("${spring.profiles.active}")
private String profile;
/**
* 실행중인 profile
*
* @return
*/
private boolean isLocalProfile() {
return "local".equalsIgnoreCase(profile);
}
// @Scheduled(cron = "0 * * * * *")
public void findTestValidMetricCsvFiles() {
// if (isLocalProfile()) {
// return;
// }
List<ResponsePathDto> modelIds =
modelTestMetricsJobCoreService.getTestMetricSaveNotYetModelIds();
if (modelIds.isEmpty()) {
return;
}
for (ResponsePathDto modelInfo : modelIds) {
String testPath = modelInfo.getResponsePath() + "/metrics/test.csv";
try (BufferedReader reader =
Files.newBufferedReader(Paths.get(testPath), StandardCharsets.UTF_8); ) {
CSVParser parser = CSVFormat.DEFAULT.withFirstRecordAsHeader().parse(reader);
List<Object[]> batchArgs = new ArrayList<>();
for (CSVRecord record : parser) {
String model = record.get("model");
long TP = Long.parseLong(record.get("TP"));
long FP = Long.parseLong(record.get("FP"));
long FN = Long.parseLong(record.get("FN"));
float precision = Float.parseFloat(record.get("precision"));
float recall = Float.parseFloat(record.get("recall"));
float f1_score = Float.parseFloat(record.get("f1_score"));
float accuracy = Float.parseFloat(record.get("accuracy"));
float iou = Float.parseFloat(record.get("iou"));
long detection_count = Long.parseLong(record.get("detection_count"));
long gt_count = Long.parseLong(record.get("gt_count"));
batchArgs.add(
new Object[] {
modelInfo.getModelId(),
model,
TP,
FP,
FN,
precision,
recall,
f1_score,
accuracy,
iou,
detection_count,
gt_count
});
}
modelTestMetricsJobCoreService.insertModelMetricsTest(batchArgs);
} catch (IOException e) {
throw new RuntimeException(e);
}
modelTestMetricsJobCoreService.updateModelMetricsTrainSaveYn(modelInfo.getModelId(), "step2");
}
}
}

Some files were not shown because too many files have changed in this diff Show More