kamco-cd-cron/shp-exporter/README.md

# PostgreSQL to Shapefile Converter

Spring Boot CLI application that queries PostgreSQL spatial data and generates shapefiles.

## Features

- Batch processing for multiple map_ids
- PostGIS geometry to Shapefile conversion
- **GeoServer REST API integration for automatic layer registration**
- Configurable via `application.yml`
- Generates all required shapefile files (.shp, .shx, .dbf, .prj)
- Supports EPSG:5186 (Korean 2000 / Central Belt) coordinate reference system
- GeoJSON export support

## Prerequisites

- Java 17 or higher
- PostgreSQL database with PostGIS extension
- Access to the KAMCO database at 192.168.2.127:15432

## Configuration

Edit `src/main/resources/application.yml` to configure:

```yaml
converter:
  spring.profiles.active: 'dev' #profiles.active
  inference-id: 'D5E46F60FC40B1A8BE0CD1F3547AA6'  # Inference ID (used for output folder structure)
  map-ids:
    - '35813030'    # Add your map_ids here (text type)
  batch-ids:
    - 252
    - 253
    - 257
  output-base-dir: '/kamco-nfs/dataset/export/'
  crs: 'EPSG:5186'  # Korean 2000 / Central Belt

geoserver:
  base-url: 'https://kamco.geo-dev.gs.dabeeo.com'
  workspace: 'cd'
  datastore: 'inference_result'
  overwrite-existing: true  # Delete existing layers before re-registering
  connection-timeout: 30000  # 30 seconds
  read-timeout: 60000        # 60 seconds
```

```md
## Converter Mode (`converter.mode`)

`converter.mode`는 대량 `map_ids` 처리 시
OS 커맨드라인 길이 제한(`Argument list too long`) 문제를 방지하기 위해 추가 하였습니다.

### Supported Modes

#### MERGED
- `batch-ids`에 해당하는 **모든 데이터를 하나의 Shapefile로 병합 생성**
- `map-ids`가 설정되어 있어도 **무시됨**
- 단일 결과 파일이 필요한 경우 적합

#### MAP_IDS
- 명시적으로 전달한 `map-ids`만 대상으로 Shapefile 생성
- `converter.map-ids` **필수**
- `map-ids` 개수가 많을 경우 OS 커맨드라인 길이 제한에 걸릴 수 있음

#### RESOLVE
- `batch-ids` 기준으로 **JAR 내부에서 map_ids를 조회**한 뒤 Shapefile 생성
- `map-ids`를 커맨드라인 인자로 전달하지 않음
- 대량 데이터 처리 시 가장 안전한 방식

### Default Behavior (mode 미지정 시)

- `converter.map-ids`가 비어 있으면 → **MERGED**
- `converter.map-ids`가 있으면 → **MAP_IDS**

### Command Line Parameters

You can override configuration values using command line arguments:

**Using Gradle (recommended - no quoting issues):**

```bash
./gradlew bootRun --args="--converter.inference-id=D5E46F60FC40B1A8BE0CD1F3547AA6 --converter.map-ids[0]=35813030 --converter.batch-ids[0]=252 --converter.batch-ids[1]=253 --converter.batch-ids[2]=257 --converter.mode=MERGED"
```

**Using JAR (zsh shell - quote arguments with brackets):**

```bash
java -jar build/libs/makesample-1.0.0.jar \
  '--converter.inference-id=D5E46F60FC40B1A8BE0CD1F3547AA6' \
  '--converter.map-ids[0]=35813030' \
  '--converter.batch-ids[0]=252' \
  '--converter.batch-ids[1]=253' \
  '--converter.batch-ids[2]=257' \
  '--converter.mode=MERGED'
```

**Using JAR (bash shell - no quotes needed):**

```bash
java -jar build/libs/makesample-1.0.0.jar \
  --converter.inference-id=D5E46F60FC40B1A8BE0CD1F3547AA6 \
  --converter.map-ids[0]=35813030 \
  --converter.batch-ids[0]=252 \
  --converter.batch-ids[1]=253 \
  --converter.batch-ids[2]=257 \
  --converter.mode=MERGED
```

**Note for zsh users:** zsh interprets square brackets `[]` as glob patterns. Always quote arguments containing brackets when using zsh.

## Building

```bash
./gradlew build
```

## Running

### Generate Shapefiles

```bash
./gradlew bootRun
```

Or run the JAR directly:

```bash
java -jar build/libs/makesample-1.0.0.jar
```

### Register Shapefile to GeoServer

First, set GeoServer credentials as environment variables:

```bash
export GEOSERVER_USERNAME=admin
export GEOSERVER_PASSWORD=geoserver
```

Then register a shapefile:

```bash
./gradlew bootRun --args="--upload-shp /kamco-nfs/dataset/export/D5E46F60FC40B1A8BE0CD1F3547AA6/35813030/35813030.shp --layer inference_35813030"
```

Or using the JAR:

```bash
java -jar build/libs/makesample-1.0.0.jar \
  --upload-shp /path/to/shapefile.shp \
  --layer layer_name
```

**GeoServer Registration Process:**
1. Verifies workspace 'cd' exists (must be pre-created in GeoServer)
2. Creates datastore 'inference_result' if it doesn't exist
3. Deletes existing layer if `overwrite-existing: true`
4. Publishes shapefile via REST API
5. Verifies successful registration
6. Automatically enables WMS, WFS, WMTS services

**Important Notes:**
- Workspace 'cd' must exist in GeoServer before registration
- Environment variables `GEOSERVER_USERNAME` and `GEOSERVER_PASSWORD` must be set
- Shapefile path must be absolute
- GeoServer must have file system access to the shapefile location

## Output

Shapefiles will be created in directories structured as `output-base-dir/inference-id/map-id/`:

```
/kamco-nfs/dataset/export/D5E46F60FC40B1A8BE0CD1F3547AA6/35813030/
├── 35813030.shp    # Shapefile geometry
├── 35813030.shx    # Shape index
├── 35813030.dbf    # Attribute data
└── 35813030.prj    # Projection information
```

## Database Query

The application executes the following query for each map_id:

```sql
SELECT uid, map_id, probability, before_year, after_year,
       before_c, before_p, after_c, after_p, ST_AsText(geometry) as geometry_wkt
FROM inference_results_testing
WHERE batch_id = ANY(?) AND map_id = ?
```

### Database Schema
- **geometry**: `geometry(Polygon, 5186)` - EPSG:5186 좌표계
- **map_id**: `text` - 문자열 타입
- **before_year, after_year**: `bigint` - Long 타입
- **batch_id**: `bigint` - Long 타입
- **uid**: `uuid` - UUID 타입

## Field Mapping

Shapefile field names are limited to 10 characters:

| Database Column | DB Type  | Shapefile Field | Shapefile Type |
|-----------------|----------|-----------------|----------------|
| uid             | uuid     | chnDtctId       | String         |
| map_id          | text     | mpqd_no         | String         |
| probability     | float8   | chn_dtct_p      | Double         |
| before_year     | bigint   | cprs_yr         | Long           |
| after_year      | bigint   | crtr_yr         | Long           |
| before_c        | text     | bf_cls_cd       | String         |
| before_p        | float8   | bf_cls_pro      | Double         |
| after_c         | text     | af_cls_cd       | String         |
| after_p         | float8   | af_cls_pro      | Double         |
| geometry        | geom     | the_geom        | Polygon        |

## Error Handling

### Shapefile Generation
- **No results**: Logs warning and continues to next map_id
- **Mixed geometry types**: Throws exception (shapefiles require homogeneous geometry)
- **Database connection failure**: Application exits with error
- **Invalid geometry**: Logs warning and continues processing

### GeoServer Registration
- **Workspace not found**: Logs error with remediation steps (must be pre-created)
- **Authentication failure**: Logs error prompting to verify environment variables
- **Network timeout**: Logs connection error with timeout details
- **Layer already exists**: Automatically deletes and re-registers if `overwrite-existing: true`
- **Registration failure**: Logs error but does not stop application (non-blocking)

## Validating Output

### Shapefile Validation

Open the generated shapefiles in QGIS or ArcGIS to verify:

1. Geometry displays correctly
2. Attribute table contains all expected fields
3. CRS is EPSG:5186 (Korean 2000 / Central Belt)

### GeoServer Layer Validation

After registering to GeoServer, verify the layer:

1. **GeoServer Admin Console**: https://kamco.geo-dev.gs.dabeeo.com/geoserver/web
2. Navigate to **Layers** → Find your layer (e.g., `cd:inference_35813030`)
3. Preview the layer using **Layer Preview**
4. Verify services are enabled:
   - WMS: `https://kamco.geo-dev.gs.dabeeo.com/geoserver/cd/wms`
   - WFS: `https://kamco.geo-dev.gs.dabeeo.com/geoserver/cd/wfs`
   - WMTS: `https://kamco.geo-dev.gs.dabeeo.com/geoserver/cd/wmts`

**Example WMS GetMap Request:**
```
https://kamco.geo-dev.gs.dabeeo.com/geoserver/cd/wms?
  service=WMS&
  version=1.1.0&
  request=GetMap&
  layers=cd:inference_35813030&
  bbox=<bounds>&
  width=768&
  height=768&
  srs=EPSG:5186&
  format=image/png
```

## Development

### Code Formatting

The project uses Google Java Format with 2-space indentation:

```bash
./gradlew spotlessApply
```

### Project Structure

```
src/main/java/com/kamco/makesample/
├── MakeSampleApplication.java                    # Main application class
├── cli/
│   └── ConverterCommandLineRunner.java           # CLI entry point
├── config/
│   ├── ConverterProperties.java                  # Shapefile converter configuration
│   ├── GeoServerProperties.java                  # GeoServer configuration
│   ├── GeoServerCredentials.java                 # GeoServer authentication
│   └── RestTemplateConfig.java                   # HTTP client configuration
├── exception/
│   ├── ShapefileConversionException.java
│   ├── GeometryConversionException.java
│   ├── MixedGeometryException.java
│   └── GeoServerRegistrationException.java       # GeoServer registration errors
├── model/
│   └── InferenceResult.java                      # Domain model
├── repository/
│   └── InferenceResultRepository.java            # Data access layer
├── service/
│   ├── GeometryConverter.java                    # PostGIS to JTS conversion
│   ├── ShapefileConverterService.java            # Orchestration service
│   └── GeoServerRegistrationService.java         # GeoServer REST API integration
└── writer/
    ├── ShapefileWriter.java                       # GeoTools shapefile writer
    └── GeoJsonWriter.java                         # GeoJSON export writer
```

## Dependencies

- Spring Boot 3.5.7
  - spring-boot-starter
  - spring-boot-starter-jdbc
  - spring-boot-starter-web (for RestTemplate)
  - spring-boot-starter-validation (for @NotBlank annotations)
- GeoTools 30.0
  - gt-shapefile
  - gt-referencing
  - gt-epsg-hsql
  - gt-geojson
- PostgreSQL JDBC Driver
- PostGIS JDBC 2.5.1
- JTS (Java Topology Suite) 1.19.0

## Troubleshooting

### GeoServer Registration Issues

**Problem: "Workspace not found: cd"**
```
Solution: Create workspace 'cd' in GeoServer admin console before registration
Steps:
1. Login to GeoServer admin: https://kamco.geo-dev.gs.dabeeo.com/geoserver/web
2. Go to Workspaces → Add new workspace
3. Name: cd, Namespace URI: http://cd
4. Click Save
```

**Problem: "GeoServer credentials not configured"**
```
Solution: Set environment variables before running
export GEOSERVER_USERNAME=admin
export GEOSERVER_PASSWORD=geoserver
```

**Problem: "Layer already exists and overwrite is disabled"**
```
Solution: Enable overwrite in application.yml
geoserver:
  overwrite-existing: true
```

**Problem: Connection timeout to GeoServer**
```
Solution: Increase timeout values in application.yml
geoserver:
  connection-timeout: 60000  # 60 seconds
  read-timeout: 120000       # 120 seconds
```

**Problem: "Registration failed. Layer not found after publication"**
```
Possible causes:
1. GeoServer cannot access shapefile path (check file system permissions)
2. Shapefile is corrupted or invalid
3. Network issue interrupted registration

Solution:
1. Verify GeoServer has read access to shapefile directory
2. Validate shapefile using QGIS or ogr2ogr
3. Check GeoServer logs for detailed error messages
```

### Database Connection Issues

**Problem: "Connection refused to 192.168.2.127:15432"**
```
Solution: Verify PostgreSQL is running and accessible
psql -h 192.168.2.127 -p 15432 -U kamco_cds -d kamco_cds
```

**Problem: "No results found for map_id"**
```
Solution: Verify data exists in database
SELECT COUNT(*) FROM inference_results_testing
WHERE batch_id IN (252, 253, 257) AND map_id = '35813030';
```

## License

KAMCO Internal Use Only