Test data integrity

[nitisht]

• Send logs to Parseable
• Wait for sync
• Download parquet from the store created by Parseable for the minute
• Compare the sent logs with the ones loaded from the downloaded parquet

[github-actions]

CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅

[theteachr]

I have read the CLA Document and I hereby sign the CLA

[nitisht]

recheck

[nitisht]

@trueleo have you tested? does it work as expected

[trueleo]

@theteachr It seems to be failing with

docker run --network=host trueleo8/quest:latest smoke http://localhost:8000 admin admin 
=== RUN   TestIntegrity
--- FAIL: TestIntegrity (120.11s)
2023/12/19 12:21:54 ERROR couldn't create MinIO client error="address http://localhost:9000: too many colons in address"
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x7b5822]

goroutine 19 [running]:
testing.tRunner.func1.2({0x8025a0, 0xbac4d0})
        /usr/local/go/src/testing/testing.go:1545 +0x238
testing.tRunner.func1()
        /usr/local/go/src/testing/testing.go:1548 +0x397
panic({0x8025a0?, 0xbac4d0?})
        /usr/local/go/src/runtime/panic.go:914 +0x21f
quest.downloadParquetFiles({0x7ffdf254beab, 0xa}, {{0x7ffdf254bed9, 0x15}, {0x7ffdf254befb, 0xa}, {0x7ffdf254bf12, 0xa}, {0x7ffdf254bf2b, 0x9}})
        /tests/integrity_test.go:155 +0x102
quest.TestIntegrity(0x0?)
        /tests/integrity_test.go:121 +0x47b
testing.tRunner(0xc000107380, 0x8c8670)
        /usr/local/go/src/testing/testing.go:1595 +0xff
created by testing.(*T).Run in goroutine 1
        /usr/local/go/src/testing/testing.go:1648 +0x3ad

I am running with default params and created a bucket named parseable . There is some data in that bucket now, so it seems ingest step worked but afterward that it failed on the next step.