IAmHardliner
/
req
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

update

This commit is contained in:
Biedermann Steve 2024-05-08 14:43:29 +02:00
parent 9948b07f76
commit fd8ea49ac7
4 changed files with 195 additions and 83 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

120
out.md Normal file
View File

@ -0,0 +1,120 @@
# Requirements for journal-uploader
[[_TOC_]]
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED",
"MAY", and "OPTIONAL" in this document are to be interpreted as described in
[RFC 2119](https://datatracker.ietf.org/doc/html/rfc2119).
## Description
The journal-uploader has two main functionalities.
- Take a stream of log messages and filter them depending on their severity
- Upload journal logs for a specified time when activated through cloud call
## Requirements
### [_TOPIC-1_] Journal Watcher
#### [_SUB-1.1_] File Monitoring
- **[_REQ-1.1.1_] Continuous Monitoring:** The tool **_MUST_** continuously monitor a designated directory.
#### [_SUB-1.2_] File Detection
- **[_REQ-1.2.1_] Detection of New Files:** The tool **_MUST_** detect the addition of new files in the monitored directory.
- **[_REQ-1.2.2_] Avoid Re-processing:** The tool **_MUST NOT_** process files that have already been processed.
### [_TOPIC-2_] Traced Logging
#### [_SUB-2.1_] File Processing
- **[_REQ-2.1.1_] Reading Log Messages:** When a new file is processed, each log message **_SHOULD_** be put into a buffer.
- **[_REQ-2.1.2_] Filtering Log Messages:** The tool will search for messages of a defined priority (Trigger Priority).
Each message of this priority, as well as all messages before and after, which are inside a defined timespan, **_MUST_**
get written into a file. Every other message **_SHOULD_** gets dropped.
- **[_REQ-2.1.3_] No Duplicate Log Messages:** The tool **_SHALL_** make sure that no log entry will be written to the file twice.
#### [_SUB-2.2_] Traced Log Rotation
- **[_REQ-2.2.1_] Rotating Files:** When the size of the current traced log file exceeds a certain threshold,
it **_MUST_** be closed and a new file **_MUST_** be opened for writing.
- **[_REQ-2.2.2_] Compression of Rotated Files:** Each traced log file **_MUST_** get compressed after it got rotated.
- **[_REQ-2.2.3_] Rotating Directory:** When the directory size exceeds a certain threshold, the tool **_MUST_** delete the oldest
files in the directory, until the size is below the threshold again.
### [_TOPIC-3_] Remote Journal Logging
#### [_SUB-3.1_] Service Activation
- **[_REQ-3.1.1_] Cloud Activation:** The remote journal logging **_SHALL_** be startable through a function call from the cloud.
The api call has the duration and max interval as arguments.
- **[_REQ-3.1.2_] Duration:** The remote journal logging **_SHOULD_** stay active, until it reaches the specified duration.
- **[_REQ-3.1.3_] Max Interval:** If no upload was done after the amount of time specified in max interval,
a log rotation **_SHALL_** be triggered, which will in turn get picked up by the file monitoring.
- **[_REQ-3.1.4_] Analytics Not Accepted:** If the user has not accepted the usage of their data, the cloud call **_MUST_**
result in an error.
#### [_SUB-3.2_] File Processing
- **[_REQ-3.2.1_] File Upload:** When a file gets detected, it **_SHOULD_** get uploaded to the cloud.
- **[_REQ-3.2.2_] No Duplicate Files:** Already processed files **_MUST NOT_** get uploaded again.
- **[_REQ-3.2.3_] Revoking Analytics:** If the user revokes the usage of their data, the service **_MAY_** continue running
but **_MUST NOT_** upload any data until the user allows the usage of their data again.
- **[_REQ-3.2.4_] Duration Expired:** After the specified duration is expired, the service **_SHOULD_** stop uploading files.
### [_TOPIC-4_] Configuration
- **[_CONF-4.1_] Journal Directory:** Users **_SHOULD_** be able to specify the directory to be monitored for journal files.
- **[_CONF-4.2_] Output Directory:** Users **_SHOULD_** be able to specify the directory into which the final files will be written.
- **[_CONF-4.3_] Trigger Priority:** Users **_SHOULD_** be able to specify which priority triggers the filtering.
- **[_CONF-4.4_] Journal Context:** Users **_SHOULD_** be able to specify how many seconds of context will be added to traced logs when encountering a trigger priority.
- **[_CONF-4.5_] Max File Size:** Users **_SHOULD_** be able to specify the max file size, at which a file gets rotated.
- **[_CONF-4.6_] Max Directory Size:** Users **_SHOULD_** be able to specify the max directory size, at which a directory gets rotated.
- **[_CONF-4.7_] File Monitoring Interval:** Users **_SHOULD_** be able to specify an interval, which **_SHOULD_** change
how long the tool waits before checking if new files are available.
### [_TOPIC-5_] Performance Requirements
- **[_PERF-5.1_] Efficiency:** The tool **_SHOULD_** efficiently monitor and process files without excessive resource consumption.
- **[_PERF-5.2_] Interval Delay:** The tool **_SHOULD_** do its work with no more than 10 seconds delay after its interval.
### [_TOPIC-6_] Security & Data Protection
- **[_SEC-6.1_] No Insecure Connection:** The tool **_MUST_** send data only through a secure connection.
- **[_SEC-6.2_] GDPR compliance:** The tool **_MUST NOT_** upload data if the user has not agreed to share this information.
### [_TOPIC-7_] Testing
- **[_TEST-7.1_] Unit Tests:** Comprehensive unit tests **_SHOULD_** be written to cover major functionalities.
- **[_TEST-7.2_] Integration Tests:** Integration tests **_SHOULD_** be conducted to ensure all parts of the tool work together seamlessly.
## Definitions
- Default Journal Directory: /run/log/journal/<machine_id>
- Machine ID can be found at /etc/machine-id
- Default Output Directory: /run/log/filtered-journal
## Config Defaults
- **Journal Directory**
- Type: Path
- **Required**: This value **_MUST_** be provided as a start parameter.
- **Output Directory**
- Type: Path
- **Required**: This value **_MUST_** be provided as a start parameter.
- **Trigger Priority**
- Type: Enum
- Valid Values: _Emergency, Alert, Critical, Error, Warning, Notice, Info, Debug_
- Default Value: _Warning_
- **Journal Context**
- Type: Integer
- Unit: Seconds
- Default Value: _15_
- **Max File Size**
- Type: Integer
- Unit: Bytes
- Default Value: _8388608_ (8 MB)
- **Max Directory Size**
- Type: Integer
- Unit: Bytes
- Default Value: _75497472_ (72 MB)
- **File Monitoring Interval**
- Type: Integer
- Unit: Seconds
- Default Value: _10_

107
req.yml
View File

@ -4,163 +4,152 @@ description: |-
- Take a stream of log messages and filter them depending on their severity - Take a stream of log messages and filter them depending on their severity
- Upload journal logs for a specified time when activated through cloud call - Upload journal logs for a specified time when activated through cloud call
topics: topics:
FEAT-1: TOPIC-1:
name: Traced Logging name: Journal Watcher
subtopics: subtopics:
SUB-1: SUB-1.1:
name: File Monitoring name: File Monitoring
requirements: requirements:
REQ-1: REQ-1.1.1:
name: Continuous Monitoring name: Continuous Monitoring
description: The tool must continuously monitor a designated directory. description: The tool must continuously monitor a designated directory.
SUB-2: SUB-1.2:
name: File Detection name: File Detection
requirements: requirements:
REQ-1: REQ-1.2.1:
name: Detection of New Files name: Detection of New Files
description: The tool must detect the addition of new files in the monitored directory. description: The tool must detect the addition of new files in the monitored directory.
REQ-2: REQ-1.2.2:
name: Avoid Re-processing name: Avoid Re-processing
description: The tool must not process files that have already been processed. description: The tool must not process files that have already been processed.
SUB-3: TOPIC-2:
name: Traced Logging
subtopics:
SUB-2.1:
name: File Processing name: File Processing
requirements: requirements:
REQ-1: REQ-2.1.1:
name: Reading Log Messages name: Reading Log Messages
description: When a new file is processed, each log message should be put into a buffer. description: When a new file is processed, each log message should be put into a buffer.
REQ-2: REQ-2.1.2:
name: Filtering Log Messages name: Filtering Log Messages
description: |- description: |-
The tool will search for messages of a defined priority (Trigger Priority). The tool will search for messages of a defined priority (Trigger Priority).
Each message of this priority, as well as all messages before and after, which are inside a defined timespan, must Each message of this priority, as well as all messages before and after, which are inside a defined timespan, must
get written into a file. Every other message should gets dropped. get written into a file. Every other message should gets dropped.
REQ-3: REQ-2.1.3:
name: No Duplicate Log Messages name: No Duplicate Log Messages
description: The tool shall make sure that no log entry will be written to the file twice. description: The tool shall make sure that no log entry will be written to the file twice.
SUB-4: SUB-2.2:
name: Traced Log Rotation name: Traced Log Rotation
requirements: requirements:
REQ-1: REQ-2.2.1:
name: Rotating Files name: Rotating Files
description: |- description: |-
When the size of the current traced log file exceeds a certain threshold, When the size of the current traced log file exceeds a certain threshold,
it must be closed and a new file must be opened for writing. it must be closed and a new file must be opened for writing.
REQ-2: REQ-2.2.2:
name: Compression of Rotated Files name: Compression of Rotated Files
description: Each traced log file must get compressed after it got rotated. description: Each traced log file must get compressed after it got rotated.
REQ-3: REQ-2.2.3:
name: Rotating Directory name: Rotating Directory
description: |- description: |-
When the directory size exceeds a certain threshold, the tool must delete the oldest When the directory size exceeds a certain threshold, the tool must delete the oldest
files in the directory, until the size is below the threshold again. files in the directory, until the size is below the threshold again.
FEAT-2: TOPIC-3:
name: Remote Journal Logging name: Remote Journal Logging
subtopics: subtopics:
SUB-1: SUB-3.1:
name: Service Activation name: Service Activation
requirements: requirements:
REQ-1: REQ-3.1.1:
name: Cloud Activation name: Cloud Activation
description: |- description: |-
The remote journal logging shall be startable through a function call from the cloud. The remote journal logging shall be startable through a function call from the cloud.
The api call has the duration and max interval as arguments. The api call has the duration and max interval as arguments.
REQ-2: REQ-3.1.2:
name: Duration name: Duration
description: The remote journal logging should stay active, until it reaches the specified duration. description: The remote journal logging should stay active, until it reaches the specified duration.
REQ-3: REQ-3.1.3:
name: Max Interval name: Max Interval
description: |- description: |-
If no upload was done after the amount of time specified in max interval, If no upload was done after the amount of time specified in max interval,
a log rotation shall be triggered, which will in turn get picked up by the file monitoring. a log rotation shall be triggered, which will in turn get picked up by the file monitoring.
REQ-4: REQ-3.1.4:
name: Analytics Not Accepted name: Analytics Not Accepted
description: |- description: |-
If the user has not accepted the usage of their data, the cloud call must If the user has not accepted the usage of their data, the cloud call must
result in an error. result in an error.
SUB-2: SUB-3.2:
name: File Monitoring
requirements:
REQ-1:
name: Continuous Monitoring
description: The tool should continuously monitor a designated directory.
SUB-3:
name: File Detection
requirements:
REQ-1:
name: Detection of New Files
description: The tool must detect the addition of new files in the monitored directory.
REQ-2:
name: Avoid Re-processing
description: The tool must not process files that have already been processed.
SUB-4:
name: File Processing name: File Processing
requirements: requirements:
REQ-1: REQ-3.2.1:
name: File Upload name: File Upload
description: When a file gets detected, it should get uploaded to the cloud. description: When a file gets detected, it should get uploaded to the cloud.
REQ-2: REQ-3.2.2:
name: No Duplicate Files name: No Duplicate Files
description: Already processed files must not get uploaded again. description: Already processed files must not get uploaded again.
REQ-3: REQ-3.2.3:
name: Revoking Analytics name: Revoking Analytics
description: |- description: |-
If the user revokes the usage of their data, the service may continue running If the user revokes the usage of their data, the service may continue running
but must not upload any data until the user allows the usage of their data again. but must not upload any data until the user allows the usage of their data again.
REQ-4: REQ-3.2.4:
name: Duration Expired name: Duration Expired
description: After the specified duration is expired, the service should stop uploading files. description: After the specified duration is expired, the service should stop uploading files.
FEAT-3:
TOPIC-4:
name: Configuration name: Configuration
requirements: requirements:
REQ-1: CONF-4.1:
name: Journal Directory name: Journal Directory
description: Users should be able to specify the directory to be monitored for journal files. description: Users should be able to specify the directory to be monitored for journal files.
REQ-2: CONF-4.2:
name: Output Directory name: Output Directory
description: Users should be able to specify the directory into which the final files will be written. description: Users should be able to specify the directory into which the final files will be written.
REQ-3: CONF-4.3:
name: Trigger Priority name: Trigger Priority
description: Users should be able to specify which priority triggers the filtering. description: Users should be able to specify which priority triggers the filtering.
REQ-4: CONF-4.4:
name: Journal Context name: Journal Context
description: Users should be able to specify how many seconds of context will be added description: Users should be able to specify how many seconds of context will be added
to traced logs when encountering a trigger priority. to traced logs when encountering a trigger priority.
REQ-5: CONF-4.5:
name: Max File Size name: Max File Size
description: Users should be able to specify the max file size, at which a file gets rotated. description: Users should be able to specify the max file size, at which a file gets rotated.
REQ-6: CONF-4.6:
name: Max Directory Size name: Max Directory Size
description: Users should be able to specify the max directory size, at which a directory gets rotated. description: Users should be able to specify the max directory size, at which a directory gets rotated.
REQ-7: CONF-4.7:
name: File Monitoring Interval name: File Monitoring Interval
description: |- description: |-
Users should be able to specify an interval, which should change Users should be able to specify an interval, which should change
how long the tool waits before checking if new files are available. how long the tool waits before checking if new files are available.
FEAT-4: TOPIC-5:
name: Performance Requirements name: Performance Requirements
requirements: requirements:
REQ-1: PERF-5.1:
name: Efficiency name: Efficiency
description: The tool should efficiently monitor and process files without excessive resource consumption. description: The tool should efficiently monitor and process files without excessive resource consumption.
REQ-2: PERF-5.2:
name: Interval Delay name: Interval Delay
description: The tool should do its work with no more than 10 seconds delay after its interval. description: The tool should do its work with no more than 10 seconds delay after its interval.
FEAT-5: TOPIC-6:
name: Data Protection name: Security & Data Protection
requirements: requirements:
REQ-1: SEC-6.1:
name: No Insecure Connection name: No Insecure Connection
description: The tool must send data only through a secure connection. description: The tool must send data only through a secure connection.
REQ-2: SEC-6.2:
name: GDPR compliance name: GDPR compliance
description: The tool must not upload data if the user has not agreed to share this information. description: The tool must not upload data if the user has not agreed to share this information.
FEAT-6: TOPIC-7:
name: Testing name: Testing
requirements: requirements:
REQ-1: TEST-7.1:
name: Unit Tests name: Unit Tests
description: Comprehensive unit tests should be written to cover major functionalities. description: Comprehensive unit tests should be written to cover major functionalities.
REQ-2: TEST-7.2:
name: Integration Tests name: Integration Tests
description: Integration tests should be conducted to ensure all parts of the tool work together seamlessly. description: Integration tests should be conducted to ensure all parts of the tool work together seamlessly.

21
src/lib.rs
View File

@ -2,27 +2,6 @@ use indexmap::{indexmap, IndexMap};
use serde::{Deserialize, Serialize, Serializer}; use serde::{Deserialize, Serialize, Serializer};
use stringlit::s; use stringlit::s;
#[allow(dead_code)]
pub const WORD_DESCRIPTION: &str = //
r#"The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED",
"MAY", and "OPTIONAL" in this document are to be interpreted as described in
[RFC 2119](https://datatracker.ietf.org/doc/html/rfc2119).
"#;
#[allow(dead_code)]
pub const HIGHLIGHTED_WORDS: [&str; 10] = [
"must not",
"must",
"required",
"shall not",
"shall",
"should not",
"should",
"recommended",
"may",
"optional",
];
pub fn my_trim<S>(v: &String, s: S) -> Result<S::Ok, S::Error> pub fn my_trim<S>(v: &String, s: S) -> Result<S::Ok, S::Error>
where where
S: Serializer, S: Serializer,

30
src/main.rs
View File

@ -2,6 +2,25 @@ use indexmap::IndexMap;
use req::*; use req::*;
use stringlit::s; use stringlit::s;
pub const WORD_DESCRIPTION: &str = //
r#"The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED",
"MAY", and "OPTIONAL" in this document are to be interpreted as described in
[RFC 2119](https://datatracker.ietf.org/doc/html/rfc2119).
"#;
pub const HIGHLIGHTED_WORDS: [&str; 10] = [
"must not",
"must",
"required",
"shall not",
"shall",
"should not",
"should",
"recommended",
"may",
"optional",
];
fn nl() -> String { fn nl() -> String {
s!("") s!("")
} }
@ -9,7 +28,7 @@ fn nl() -> String {
fn add_requirements(output: &mut Vec<String>, requirements: &IndexMap<String, Requirement>) { fn add_requirements(output: &mut Vec<String>, requirements: &IndexMap<String, Requirement>) {
for (id, requirement) in requirements { for (id, requirement) in requirements {
output.push(format!( output.push(format!(
"- **{id} {}:** {}", "- **[_{id}_] {}:** {}",
requirement.name, requirement.description requirement.name, requirement.description
)); ));
} }
@ -17,7 +36,7 @@ fn add_requirements(output: &mut Vec<String>, requirements: &IndexMap<String, Re
fn add_topics(output: &mut Vec<String>, topics: &IndexMap<String, Topic>, level: usize) { fn add_topics(output: &mut Vec<String>, topics: &IndexMap<String, Topic>, level: usize) {
for (id, topic) in topics { for (id, topic) in topics {
output.push(format!("{} {id} {}", "#".repeat(level), topic.name)); output.push(format!("{} [_{id}_] {}", "#".repeat(level), topic.name));
if !topic.requirements.is_empty() { if !topic.requirements.is_empty() {
add_requirements(output, &topic.requirements); add_requirements(output, &topic.requirements);
output.push(nl()); output.push(nl());
@ -89,7 +108,12 @@ fn main() -> anyhow::Result<()> {
} }
} }
println!("{}", output.join("\n")); let mut output = output.join("\n");
for word in HIGHLIGHTED_WORDS {
output = output.replace(word, &format!("**_{}_**", word.to_uppercase()));
}
println!("{output}");
Ok(()) Ok(())
} }