… mismatch

data directory creation should not happen in case of deployment mismatch
staging should be overwritten in case of new staging

… mismatch

    data directory creation should not happen in case of deployment mismatch
    staging should be overwritten in case of new staging
    default staging and data directory should not be created if env var has different path

…en staging and remote

Co-authored-by: Nitish Tiwari <[email protected]>

…blehq#629)

Bumps [h2](https://github.com/hyperium/h2) from 0.3.17 to 0.3.24.
- [Release notes](https://github.com/hyperium/h2/releases)
- [Changelog](https://github.com/hyperium/h2/blob/v0.3.24/CHANGELOG.md)
- [Commits](hyperium/h2@v0.3.17...v0.3.24)

---
updated-dependencies:
- dependency-name: h2
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…ehq#623)

fixes parseablehq#615

Standardise the error message and also use the new logg.ing
domain for short URLs.

…eablehq#634)

Add P_MODE with options `ingest`, `query` and `all`. Default mode is `all`. 
There are still more changes required for both modes to work well. Will be
added in next subsequent PRs.

Fixes parseablehq#617

create parquet file by grouping all arrow files (in staging) for the duration 
provided in env variable P_STORAGE_UPLOAD_INTERVAL also check 
if arrow files vector is not empty, then sort the arrow files and create key 
for parquet file from last file from sorted arrow files vector

Fixes parseablehq#616

Previously in Query Mode, All log stream endpoints were allowed.
But is it better that only ingester is allowed to create streams.

Fixes parseablehq#641

Signed-off-by: Nitish Tiwari <[email protected]>

Earlier, separate scheduler was initialized for each 
stream on load time or whenever retention period is set.
Now, a single scheduler is initialized which checks retention 
config of all the streams and performs the retention cleanup.

Fixes parseablehq#636

…blehq#653)

added const of 60 secs to be used for local to storage sync

fixes parseablehq#651

fixes: parseablehq#312

fixes parseablehq#650

…parseablehq#661)

fixes parseablehq#654

fixes parseablehq#660

includes console release v0.4.0

It is better that users generate themselves as needed.

Signed-off-by: Nitish Tiwari <[email protected]>

…snapshot" (parseablehq#666)

Reverts parseablehq#661 because with this change we're backward incompatible with older 
versions where .stream.json doesn't contain retention field.

This reverts commit 121bf01.

Signed-off-by: Nitish Tiwari <[email protected]>

Changes does in the PR -
1. adds the first_event_at property (from the min value of p_timestamp of the first parquet file listed in the first manifest file from the snapshot of the stream.json) to the stats api and writes it to the stream.json file at the request of get stats.
2. updates the first_event_at in case of retention

Fixes : parseablehq#587

…parseablehq#673)

current: when server restarts, the retention config gets deleted from stream.json
change: retention config does not get deleted on server restart
Fixes: parseablehq#654

Bumps [mio](https://github.com/tokio-rs/mio) from 0.8.8 to 0.8.11.
- [Release notes](https://github.com/tokio-rs/mio/releases)
- [Changelog](https://github.com/tokio-rs/mio/blob/master/CHANGELOG.md)
- [Commits](tokio-rs/mio@v0.8.8...v0.8.11)

---
updated-dependencies:
- dependency-name: mio
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…seablehq#683)

This PR adds enhancement to use a user provided timestamp for partition 
in ingesting logs instead of using server time.

User needs to add custom header X-P-Time-Partition (optional) at stream 
creation api to allow ingestion/query using timestamp column from the 
log data instead of server time p_timestamp

This is time_partition field name is stored in stream.json and in memory 
STREAM_INFO in ingest api. Server checks if timestamp column name exists in 
the log event, if not, throw exception. Also, checks if timestamp value can be 
parsed into datetime, if not, throw exception arrow file name gets the date, 
hr, mm from the timestamp field (if defined in stream) else file name gets 
the date, hr, mm from the server time parquet file name gets a random 
number attached to it. This is because a lot of log data can have same 
date, hr, mm value of the timestamp field and with this random number, 
parquet will not get overwritten in the console, query from and to date will 
be matched against the value of the timestamp column of the log data (if 
defined in the stream), else from and to date will be matched against the 
p_timestamp column.

Fixes parseablehq#671 
Fixes parseablehq#685

Fixes parseablehq#690 

Signed-off-by: Nitish Tiwari <[email protected]>

modified function that checks if query has starttime before the 1st manifest lower bound time
to ensure server uses manifest for count(*) where starttime is greater than manifest creation date

fix for ingestion with time partition
fix for query with time partition
fix for query optimization
fixed get_first_event call for time partition
random number generation logic changed while creating parquet file name

* removed the X-P-Time-Partition header in log stream creation API
* removed the logic that partitions the ingested log based on the 
X-P-Time-Partition header value of which was stored in `stream.json`
* Query still uses the logic to make query compatible with the external 
tool that ingests based on time partition

…eablehq#716)

This PR allow historical ingestion only when date column provided in
header x-p-time-partition and server time are within the same minute, 
no change for default ingestion.

Add API - GET /logstream/{logstream}/info
Response -
{
    "created-at": "2024-03-27T15:58:28.418792+05:30",
    "first-event-at": "2024-03-27T15:59:08.980+05:30",
    "cache_enabled": false,
    "time_partition": "source_time"
}

Also removed logic of first-event-at and created-at 
from stats API response

set header x-p-static-schema-flag = true and schema in body in below format -
{
   "fields":[
    {
        "name": name of the field",
        "data_type": "data type of the field, out of these (int, double, float, boolean, string, datetime, string_list, int_list, double_list, float_list, boolean_list)",
    }
   ]
}
once provided, schema is persisted in the storage and in metadata
if static schema provided, ingest api will verify if event log schema matches the static schema provided in stream creation
if schema does not match, ingestion is rejected

if field is defined as float,
below values (examples) can be accepted -
100,-100.45, 100.23, "200.45"

…ablehq#730)

delete ingester if offline

* fix: stats response

* fix: s3 get objects

Refactor object storage to filter objects by starts_with_pattern

…#734)

* Add node_url field to Cli struct and update related code

* Update required flag for Node URL in CLI

* updated logic to have server address (ip:port) in parquet file name similar to other json files

---------

Co-authored-by: Nikhil Sinha <[email protected]>

* remove staging query from the query result (for distributed)

* Refactor get_schema method to handle missing schema in object storage

Bumps [h2](https://github.com/hyperium/h2) from 0.3.24 to 0.3.26.
- [Release notes](https://github.com/hyperium/h2/releases)
- [Changelog](https://github.com/hyperium/h2/blob/v0.3.26/CHANGELOG.md)
- [Commits](hyperium/h2@v0.3.24...v0.3.26)

---
updated-dependencies:
- dependency-name: h2
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* fix analytics for the cluster
* Add active_ingesters and inactive_ingesters metrics
* updated ingesters' count and event related metrics

This PR adds fixes for 

1. Default role not assigned to the OAuth user if group does not exist
2. Use user name used instead of id

fixes parseablehq#638
fixes parseablehq#868

Signed-off-by: Nitish Tiwari <[email protected]>

fix for staging size metrics not resetting to 0 even when 
local to storage sync is completed and no arrow/parquet 
file is left in staging folder

* Refactor object storage to use filter_func instead of 
starts_with_pattern in get_objects method
* Refactor fetch_schema method to use object storage instead of HTTP requests
* Refactor metadata.rs and storage.rs
* refactor ingest logic
* fetch stream info from store if stream info is not present in memory.
error if stream info does not exist in S3 and memory

Co-authored-by: Nikhil Sinha <[email protected]>

…seablehq#746)

1. fixed banner spacing
2. modified server mode: All to Standalone, Ingest to Distributed (Ingest), Query to Distributed (Query)
3. updated server mode in about API response
4. updated logic for env var P_INGESTOR_URL to use HOSTNAME and PORT from env
5. remove put cache api from querier
6. added put cache api to ingestor
7. renamed ingester to ingestor
8. corrected cache flow for ingestors and standalone
9. removed query, other logstream apis for ingestors

also fixed P_INGESTOR_URL fetch from env variables

with this PR, when delete stream is called, querier 
deletes the stream folder from the storage then calls 
delete stream API for each ingestor. Finally, ingestor 
deletes the stream from its local map

This PR ensures all metadata and data files (json and parquet) use 
a simple sha256 based hash name mechanism. Each ingestor 
allocates itself a unique hash which is used in all file names
relevant to that ingestor. This hash is persisted in metadata file
content also and is supposed to be the same for the lifecycle 
of the ingestor.
---------

Co-authored-by: Nikhil Sinha <[email protected]>

caching for distributed mode can be 
enabled from querier UI. Querier calls the PUT 
/cache API to all ingestors. An ingestor, first checks 
if stream exists, if not found in local map, checks 
in S3 and creates stream. Then checks if caching 
env vars are set. If yes, add cache_enabled flag to 
STREAM_INFO and update its stream.json in S3

Fixes: parseablehq#764

fix for stream info api