Skip to content

Commit

Permalink
Test data integrity in object store (#47)
Browse files Browse the repository at this point in the history
This PR adds a test case to

- Generate logs dynamically using `flog`
- Ingest across minute boundaries
- Wait for sync
- Download parquet from the store created by Parseable for the minute
- Compare the sent logs with the ones loaded from the downloaded parquet

Co-authored-by: theteachr <[email protected]>
  • Loading branch information
nitisht and theteachr authored Dec 19, 2023
1 parent 3d730c0 commit c7bc0c3
Show file tree
Hide file tree
Showing 5 changed files with 1,359 additions and 3 deletions.
20 changes: 19 additions & 1 deletion go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,28 @@ module quest

go 1.21.1

require github.com/stretchr/testify v1.8.4
require (
github.com/minio/minio-go v6.0.14+incompatible
github.com/stretchr/testify v1.8.4
github.com/xitongsys/parquet-go v1.6.2
github.com/xitongsys/parquet-go-source v0.0.0-20230919034749-0b16411e6349
)

require (
github.com/apache/arrow/go/arrow v0.0.0-20200730104253-651201b0f516 // indirect
github.com/apache/thrift v0.14.2 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/go-ini/ini v1.25.4 // indirect
github.com/golang/snappy v0.0.3 // indirect
github.com/klauspost/compress v1.15.9 // indirect
github.com/mitchellh/go-homedir v1.1.0 // indirect
github.com/pierrec/lz4/v4 v4.1.8 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/smartystreets/goconvey v1.8.1 // indirect
golang.org/x/crypto v0.9.0 // indirect
golang.org/x/net v0.10.0 // indirect
golang.org/x/sys v0.8.0 // indirect
golang.org/x/text v0.9.0 // indirect
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
)
Loading

0 comments on commit c7bc0c3

Please sign in to comment.