req/orig.md

# Requirements for journal-uploader

[[_TOC_]]

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED",
"MAY", and "OPTIONAL" in this document are to be interpreted as described in
[RFC 2119](https://datatracker.ietf.org/doc/html/rfc2119).

## Purpose
The journal-uploader has two main functionalities.
- Take a stream of log messages and filter them depending on their severity
- Upload journal logs for a specified time when activated through cloud call

## Requirements
### 1. Traced Logging
#### 1.1 File Monitoring
- **1.1.1 Continuous Monitoring:** The tool **_MUST_** continuously monitor a designated directory.

#### 1.2 File Detection
- **1.2.1 Detection of New Files:** The tool **_MUST_** detect the addition of new files in the monitored directory.
- **1.2.2 Avoid Re-processing:** The tool **_MUST NOT_** process files that have already been processed.

#### 1.3 File Processing
- **1.3.1 Reading Log Messages:** When a new file is processed, each log message **_SHOULD_** be put into a buffer.
- **1.3.2 Filtering Log Messages:** The tool will search for messages of a defined priority (Trigger Priority).
  Each message of this priority, as well as all messages before and after, which are inside a defined timespan, **_MUST_**
  get written into a file. Every other message **_SHOULD_** gets dropped.
- **1.3.3 No Duplicate Log Messages:** The tool **_SHALL_** make sure that no log entry will be written to the file twice.

#### 1.4 Traced Log Rotation
- **1.4.1 Rotating Files:** When the size of the current traced log file exceeds a certain threshold,
  it **_MUST_** be closed and a new file **_MUST_** be opened for writing.
- **1.4.2 Compression of Rotated Files:** Each traced log file **_MUST_** get compressed after it got rotated.
- **1.4.3 Rotating Directory:** When the directory size exceeds a certain threshold, the tool **_MUST_** delete the oldest
  files in the directory, until the size is below the threshold again.

### 2. Remote Journal Logging
#### 2.1 Service Activation
- **2.1.1 Cloud Activation:** The remote journal logging **_SHALL_** be startable through a function call from the cloud.
  The api call has the duration and max interval as arguments.
- **2.1.2 Duration:** The remote journal logging **_SHOULD_** stay active, until it reaches the specified duration.
- **2.1.3 Max Interval:** If no upload was done after the amount of time specified in max interval,
  a log rotation **_SHALL_** be triggered, which will in turn get picked up by the file monitoring.
- **2.1.4 Analytics Not Accepted:** If the user has not accepted the usage of their data, the cloud call **_MUST_**
  result in an error.

#### 2.2 File Monitoring
- **2.2.1 Continuous Monitoring:** The tool **_SHOULD_** continuously monitor a designated directory.

#### 2.3 File Detection
- **2.3.1 Detection of New Files:** The tool **_MUST_** detect the addition of new files in the monitored directory.
- **2.3.2 Avoid Re-processing:** The tool **_MUST NOT_** process files that have already been processed.

#### 2.4 File Processing
- **2.4.1 File Upload:** When a file gets detected, it **_SHOULD_** get uploaded to the cloud.
- **2.4.2 No Duplicate Files:** Already processed files **_MUST NOT_** get uploaded again.
- **2.4.3 Revoking Analytics:** If the user revokes the usage of their data, the service **_MAY_** continue running
  but **_MUST NOT_** upload any data until the user allows the usage of their data again.
- **2.4.4 Duration Expired:** After the specified duration is expired, the service **_SHOULD_** stop uploading files.

### 3. Configuration
- **3.1 Configurable Journal Directory:** Users **_SHOULD_** be able to specify the directory to be monitored for
  journal files.
- **3.2 Configurable Output Directory:** Users **_SHOULD_** be able to specify the directory into which the final files
  will be written.
- **3.3 Configurable Trigger Priority:** Users **_SHOULD_** be able to specify which priority triggers the filtering.
- **3.4 Configurable Journal Context:** Users **_SHOULD_** be able to specify how many seconds of context will be added
  to traced logs when encountering a trigger priority.
- **3.5 Configurable Max File Size:** Users **_SHOULD_** be able to specify the max file size, at which a file gets rotated.
- **3.6 Configurable Max Directory Size:** Users **_SHOULD_** be able to specify the max directory size, at which a
  directory gets rotated.
- **3.7 Configurable File Monitoring Interval:** Users **_SHOULD_** be able to specify an interval, which **_SHOULD_** change
  how long the tool waits before checking if new files are available.

### 4. Performance Requirements
- **4.1 Efficiency:** The tool **_SHOULD_** efficiently monitor and process files without excessive resource consumption.
- **4.2 Interval Delay:** The tool **_SHOULD_** do its work with no more than 10 seconds delay after its interval.

### 5. Data Protection
- **5.1 No Insecure Connection:** The tool **_MUST_** send data only through a secure connection.
- **5.2 GDPR compliance:** The tool **_MUST NOT_** upload data if the user has not agreed to share this information.

### 6. Testing
- **6.1 Unit Tests:** Comprehensive unit tests **_SHOULD_** be written to cover major functionalities.
- **6.2 Integration Tests:** Integration tests **_SHOULD_** be conducted to ensure all parts of the tool work together
  seamlessly.

## Definitions
- Default Journal Directory: /run/log/journal/<machine_id>
  - Machine ID can be found at /etc/machine-id
- Default Output Directory: /run/log/filtered-journal

## Config Defaults
- **Journal Directory**
  - type: Path
  - **Required**: This value **_MUST_** be provided as a start parameter.

- **Output Directory**
  - type: Path
  - **Required**: This value **_MUST_** be provided as a start parameter.

- **Trigger Priority**
  - type: Enum
  - Valid Values: _Emergency, Alert, Critical, Error, Warning, Notice, Info, Debug_
  - Default Value: _Warning_

- **Journal Context**
  - type: Integer
  - unit: Seconds
  - Default Value: _15_

- **Max File Size**
  - type: Integer
  - unit: Bytes
  - Default Value: _8388608_ (8 MB)

- **Max Directory Size**
  - type: Integer
  - unit: Bytes
  - Default Value: _75497472_ (72 MB)

- **File Monitoring Interval**
  - type: Integer
  - unit: Seconds
  - Default Value: _10_