# Requirements for journal-uploader [[_TOC_]] The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC 2119](https://datatracker.ietf.org/doc/html/rfc2119). ## Purpose The journal-uploader has two main functionalities. - Take a stream of log messages and filter them depending on their severity - Upload journal logs for a specified time when activated through cloud call ## Requirements ### 1. Traced Logging #### 1.1 File Monitoring - **1.1.1 Continuous Monitoring:** The tool **_MUST_** continuously monitor a designated directory. #### 1.2 File Detection - **1.2.1 Detection of New Files:** The tool **_MUST_** detect the addition of new files in the monitored directory. - **1.2.2 Avoid Re-processing:** The tool **_MUST NOT_** process files that have already been processed. #### 1.3 File Processing - **1.3.1 Reading Log Messages:** When a new file is processed, each log message **_SHOULD_** be put into a buffer. - **1.3.2 Filtering Log Messages:** The tool will search for messages of a defined priority (Trigger Priority). Each message of this priority, as well as all messages before and after, which are inside a defined timespan, **_MUST_** get written into a file. Every other message **_SHOULD_** gets dropped. - **1.3.3 No Duplicate Log Messages:** The tool **_SHALL_** make sure that no log entry will be written to the file twice. #### 1.4 Traced Log Rotation - **1.4.1 Rotating Files:** When the size of the current traced log file exceeds a certain threshold, it **_MUST_** be closed and a new file **_MUST_** be opened for writing. - **1.4.2 Compression of Rotated Files:** Each traced log file **_MUST_** get compressed after it got rotated. - **1.4.3 Rotating Directory:** When the directory size exceeds a certain threshold, the tool **_MUST_** delete the oldest files in the directory, until the size is below the threshold again. ### 2. Remote Journal Logging #### 2.1 Service Activation - **2.1.1 Cloud Activation:** The remote journal logging **_SHALL_** be startable through a function call from the cloud. The api call has the duration and max interval as arguments. - **2.1.2 Duration:** The remote journal logging **_SHOULD_** stay active, until it reaches the specified duration. - **2.1.3 Max Interval:** If no upload was done after the amount of time specified in max interval, a log rotation **_SHALL_** be triggered, which will in turn get picked up by the file monitoring. - **2.1.4 Analytics Not Accepted:** If the user has not accepted the usage of their data, the cloud call **_MUST_** result in an error. #### 2.2 File Monitoring - **2.2.1 Continuous Monitoring:** The tool **_SHOULD_** continuously monitor a designated directory. #### 2.3 File Detection - **2.3.1 Detection of New Files:** The tool **_MUST_** detect the addition of new files in the monitored directory. - **2.3.2 Avoid Re-processing:** The tool **_MUST NOT_** process files that have already been processed. #### 2.4 File Processing - **2.4.1 File Upload:** When a file gets detected, it **_SHOULD_** get uploaded to the cloud. - **2.4.2 No Duplicate Files:** Already processed files **_MUST NOT_** get uploaded again. - **2.4.3 Revoking Analytics:** If the user revokes the usage of their data, the service **_MAY_** continue running but **_MUST NOT_** upload any data until the user allows the usage of their data again. - **2.4.4 Duration Expired:** After the specified duration is expired, the service **_SHOULD_** stop uploading files. ### 3. Configuration - **3.1 Configurable Journal Directory:** Users **_SHOULD_** be able to specify the directory to be monitored for journal files. - Required: This value **_MUST_** be provided as a start parameter. - **3.2 Configurable Output Directory:** Users **_SHOULD_** be able to specify the directory into which the final files will be written. - Required: This value **_MUST_** be provided as a start parameter. - **3.3 Configurable Trigger Priority:** Users **_SHOULD_** be able to specify which priority triggers the filtering. - Default Value: _Warning_ - **3.4 Configurable Journal Context:** Users **_SHOULD_** be able to specify how many seconds of context will be added to traced logs when encountering a trigger priority. - Default Value: _10_ - **3.5 Configurable Max File Size:** Users **_SHOULD_** be able to specify the max file size, at which a file gets rotated. - Default Value: _8388608_ (8 MB) - **3.6 Configurable Max Directory Size:** Users **_SHOULD_** be able to specify the max directory size, at which a directory gets rotated. - Default Value: _75497472_ (72 MB) - **3.7 Configurable File Monitoring Interval:** Users **_SHOULD_** be able to specify an interval, which **_SHOULD_** change how long the tool waits before checking if new files are available. - Default Value: _10_ ### 4. Performance Requirements - **4.1 Efficiency:** The tool **_SHOULD_** efficiently monitor and process files without excessive resource consumption. - **4.2 Interval Delay:** The tool **_SHOULD_** do its work with no more than 10 seconds delay after its interval. ### 5. Data Protection - **5.1 No Insecure Connection:** The tool **_MUST_** send data only through a secure connection. - **5.2 GDPR compliance:** The tool **_MUST NOT_** upload data if the user has not agreed to share this information. ### 6. Testing - **6.1 Unit Tests:** Comprehensive unit tests **_SHOULD_** be written to cover major functionalities. - **6.2 Integration Tests:** Integration tests **_SHOULD_** be conducted to ensure all parts of the tool work together seamlessly. ## Definitions - Default Journal Directory: /run/log/journal/ - Machine ID can be found at /etc/machine-id - Default Output Directory: /run/log/filtered-journal ## Config Defaults - **Journal Directory** - type: Path - **Required**: This value **_MUST_** be provided as a start parameter. - **Output Directory** - type: Path - **Required**: This value **_MUST_** be provided as a start parameter. - **Trigger Priority** - type: Enum - Valid Values: _Emergency, Alert, Critical, Error, Warning, Notice, Info, Debug_ - Default Value: _Warning_ - **Journal Context** - type: Integer - unit: Seconds - Default Value: _15_ - **Max File Size** - type: Integer - unit: Bytes - Default Value: _8388608_ (8 MB) - **Max Directory Size** - type: Integer - unit: Bytes - Default Value: _75497472_ (72 MB) - **File Monitoring Interval** - type: Integer - unit: Seconds - Default Value: _10_