Fluent-bit on ECS Fargate - intermittent issue #6949

vijaynauti · 2023-03-03T16:43:56Z

vijaynauti
Mar 3, 2023

Hi Team,

My application is deployed on ECS Fargate based container. Where I have configured Fluent-bit for moving my ECS logs to OpenSearch and CloudWatch Log Group.

For Up and running ECS task it is able to send logs to both the destination, but when there is new deployment and there is some issue for task to come up buz of some error at application code end and ECS is continue to spinning up new task to maintain the desired count.
On this situation most of the time, Log router is not pushing any logs to destination but some time it is pushing. Kind of intermittent , Can someone please help with this

Below is my configuration:

--extra.com
[SERVICE]
flush 1
Grace 30
log_level debug
[OUTPUT]
Name cloudwatch
Match **
region us-east-1
log_key log
log_group_name ${LogGroup_Name}
log_stream_prefix ${env}-
auto_create_group true
[OUTPUT]
Name OpenSearch
Match **
Host ${ES_Host}
Port 443
Index graceit-${Service_Name}-dev-%Y-%m-%d
Type _doc
AWS_Auth On
AWS_Region us-east-1
tls On
Suppress_Type_Name On
retry_limit 3
Buffer_Size 600KB

Dockerfile
FROM amazon/aws-for-fluent-bit:latest
ADD extra.conf /fluent-bit/configs/extra.conf
ADD entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]

@PettitWesley

PettitWesley · 2023-03-03T21:09:45Z

PettitWesley
Mar 3, 2023
Collaborator

Can you give us the Fluent Bit log output?

there is some issue for task to come up buz of some error at application code end and ECS is continue to spinning up new task to maintain the desired count.
On this situation most of the time, Log router is not pushing any logs to destination but some time it is pushing. Kind of intermittent

May be sometimes the app is just causing the task to exit immediately and the app didn't produce any logs?

Enable debug logging: https://github.com/aws/aws-for-fluent-bit/blob/mainline/troubleshooting/debugging.md#enable-debug-logging

This will tell you if Fluent Bit even ingested any records. You should see a message like "update chunk with new size"

10 replies

PettitWesley Mar 8, 2023
Collaborator

what is the intermittent issue?

vijaynauti Mar 10, 2023
Author

@PettitWesley: I have elaborated the issue on above, Pasting here again. Checked with AWS team as well. but confirmed its issue with Fluent-bit level.

For Up and running ECS task it is able to send logs to both the destination, but when there is new deployment and there is some issue for task to come up buz of some error at application code end and ECS is continue to spinning up new task to maintain the desired count.
On this situation most of the time, Log router is not pushing any logs to destination but some time it is pushing. Kind of intermittent

PettitWesley Mar 10, 2023
Collaborator

Please send me debug log output from Fluent Bit: https://github.com/aws/aws-for-fluent-bit/blob/mainline/troubleshooting/debugging.md#enable-debug-logging

vijaynauti Mar 14, 2023
Author

Hello @PettitWesley , Please find output.. this is the same where it didn't push any logs.

2023-03-14T19:18:45.903+05:30
AWS for Fluent Bit Container Image Version 2.31.4
AWS for Fluent Bit Container Image Version 2.31.4
2023-03-14T19:18:46.039+05:30
Fluent Bit v1.9.10
Fluent Bit v1.9.10
2023-03-14T19:18:46.039+05:30

right (C) 2015-2022 The Fluent Bit Authors
right (C) 2015-2022 The Fluent Bit Authors
2023-03-14T19:18:46.039+05:30
Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
2023-03-14T19:18:46.039+05:30
https://fluentbit.io
https://fluentbit.io
2023-03-14T19:18:46.039+05:30
[2023/03/14 13:48:46] [ warn] [env] variable ${env} is used but not set
[2023/03/14 13:48:46] [ warn] [env] variable ${env} is used but not set
2023-03-14T19:18:46.039+05:30
[2023/03/14 13:48:46] [ info] Configuration:
[2023/03/14 13:48:46] [ info] Configuration:
2023-03-14T19:18:46.039+05:30
[2023/03/14 13:48:46] [ info] flush time | 1.000000 seconds
[2023/03/14 13:48:46] [ info] flush time | 1.000000 seconds
2023-03-14T19:18:46.039+05:30
[2023/03/14 13:48:46] [ info] grace | 30 seconds
[2023/03/14 13:48:46] [ info] grace | 30 seconds
2023-03-14T19:18:46.039+05:30
[2023/03/14 13:48:46] [ info] daemon | 0
[2023/03/14 13:48:46] [ info] daemon | 0
2023-03-14T19:18:46.039+05:30
[2023/03/14 13:48:46] [ info] ___________
[2023/03/14 13:48:46] [ info] ___________
2023-03-14T19:18:46.039+05:30
[2023/03/14 13:48:46] [ info] inputs:
[2023/03/14 13:48:46] [ info] inputs:
2023-03-14T19:18:46.040+05:30
[2023/03/14 13:48:46] [ info] tcp
[2023/03/14 13:48:46] [ info] tcp
2023-03-14T19:18:46.040+05:30
[2023/03/14 13:48:46] [ info] forward
[2023/03/14 13:48:46] [ info] forward
2023-03-14T19:18:46.040+05:30
[2023/03/14 13:48:46] [ info] forward
[2023/03/14 13:48:46] [ info] forward
2023-03-14T19:18:46.040+05:30
[2023/03/14 13:48:46] [ info] ___________
[2023/03/14 13:48:46] [ info] ___________
2023-03-14T19:18:46.040+05:30
[2023/03/14 13:48:46] [ info] filters:
[2023/03/14 13:48:46] [ info] filters:
2023-03-14T19:18:46.040+05:30
[2023/03/14 13:48:46] [ info] record_modifier.0
[2023/03/14 13:48:46] [ info] record_modifier.0
2023-03-14T19:18:46.040+05:30
[2023/03/14 13:48:46] [ info] ___________
[2023/03/14 13:48:46] [ info] ___________
2023-03-14T19:18:46.040+05:30
[2023/03/14 13:48:46] [ info] outputs:
[2023/03/14 13:48:46] [ info] outputs:
2023-03-14T19:18:46.040+05:30
[2023/03/14 13:48:46] [ info] cloudwatch.0
[2023/03/14 13:48:46] [ info] cloudwatch.0
2023-03-14T19:18:46.040+05:30
[2023/03/14 13:48:46] [ info] opensearch.1
[2023/03/14 13:48:46] [ info] opensearch.1
2023-03-14T19:18:46.040+05:30
[2023/03/14 13:48:46] [ info] null.2
[2023/03/14 13:48:46] [ info] null.2
2023-03-14T19:18:46.040+05:30
[2023/03/14 13:48:46] [ info] ___________
[2023/03/14 13:48:46] [ info] ___________
2023-03-14T19:18:46.040+05:30
[2023/03/14 13:48:46] [ info] collectors:
[2023/03/14 13:48:46] [ info] collectors:
2023-03-14T19:18:46.045+05:30
[2023/03/14 13:48:46] [ info] [fluent bit] version=1.9.10, commit=03164e8742, pid=23
[2023/03/14 13:48:46] [ info] [fluent bit] version=1.9.10, commit=03164e8742, pid=23
2023-03-14T19:18:46.047+05:30
[2023/03/14 13:48:46] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2023/03/14 13:48:46] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
2023-03-14T19:18:46.049+05:30
[2023/03/14 13:48:46] [ info] [storage] version=1.3.0, type=memory-only, sync=normal, checksum=disabled, max_chunks_up=128
[2023/03/14 13:48:46] [ info] [storage] version=1.3.0, type=memory-only, sync=normal, checksum=disabled, max_chunks_up=128
2023-03-14T19:18:46.049+05:30
[2023/03/14 13:48:46] [ info] [cmetrics] version=0.3.7
[2023/03/14 13:48:46] [ info] [cmetrics] version=0.3.7
2023-03-14T19:18:46.049+05:30
[2023/03/14 13:48:46] [debug] [tcp:tcp.0] created event channels: read=27 write=28
[2023/03/14 13:48:46] [debug] [tcp:tcp.0] created event channels: read=27 write=28
2023-03-14T19:18:46.050+05:30
[2023/03/14 13:48:46] [ info] [input:tcp:tcp.0] listening on 127.0.0.1:8877
[2023/03/14 13:48:46] [ info] [input:tcp:tcp.0] listening on 127.0.0.1:8877
2023-03-14T19:18:46.050+05:30
[2023/03/14 13:48:46] [debug] [forward:forward.1] created event channels: read=30 write=31
[2023/03/14 13:48:46] [debug] [forward:forward.1] created event channels: read=30 write=31
2023-03-14T19:18:46.050+05:30
[2023/03/14 13:48:46] [ info] [input:forward:forward.1] listening on unix:///var/run/fluent.sock
[2023/03/14 13:48:46] [ info] [input:forward:forward.1] listening on unix:///var/run/fluent.sock
2023-03-14T19:18:46.050+05:30
[2023/03/14 13:48:46] [debug] [forward:forward.2] created event channels: read=33 write=34
[2023/03/14 13:48:46] [debug] [forward:forward.2] created event channels: read=33 write=34
2023-03-14T19:18:46.050+05:30
[2023/03/14 13:48:46] [debug] [in_fw] Listen='127.0.0.1' TCP_Port=24224
[2023/03/14 13:48:46] [debug] [in_fw] Listen='127.0.0.1' TCP_Port=24224
2023-03-14T19:18:46.052+05:30
time="2023-03-14T13:48:46Z" level=info msg="[cloudwatch 0] plugin parameter log_group_name = 'uai2005812/smartshop/dev-router-management-service-v1'"
time="2023-03-14T13:48:46Z" level=info msg="[cloudwatch 0] plugin parameter log_group_name = 'uai2005812/smartshop/dev-router-management-service-v1'"
2023-03-14T19:18:46.052+05:30
time="2023-03-14T13:48:46Z" level=info msg="[cloudwatch 0] plugin parameter default_log_group_name = 'fluentbit-default'"
time="2023-03-14T13:48:46Z" level=info msg="[cloudwatch 0] plugin parameter default_log_group_name = 'fluentbit-default'"
2023-03-14T19:18:46.053+05:30
time="2023-03-14T13:48:46Z" level=info msg="[cloudwatch 0] plugin parameter log_stream_prefix = '-'"
time="2023-03-14T13:48:46Z" level=info msg="[cloudwatch 0] plugin parameter log_stream_prefix = '-'"
2023-03-14T19:18:46.053+05:30
time="2023-03-14T13:48:46Z" level=info msg="[cloudwatch 0] plugin parameter log_stream_name = ''"
time="2023-03-14T13:48:46Z" level=info msg="[cloudwatch 0] plugin parameter log_stream_name = ''"
2023-03-14T19:18:46.053+05:30
time="2023-03-14T13:48:46Z" level=info msg="[cloudwatch 0] plugin parameter default_log_stream_name = '/fluentbit-default'"
time="2023-03-14T13:48:46Z" level=info msg="[cloudwatch 0] plugin parameter default_log_stream_name = '/fluentbit-default'"
2023-03-14T19:18:46.053+05:30
time="2023-03-14T13:48:46Z" level=info msg="[cloudwatch 0] plugin parameter region = 'us-east-1'"
time="2023-03-14T13:48:46Z" level=info msg="[cloudwatch 0] plugin parameter region = 'us-east-1'"
2023-03-14T19:18:46.053+05:30
time="2023-03-14T13:48:46Z" level=info msg="[cloudwatch 0] plugin parameter log_key = 'log'"
time="2023-03-14T13:48:46Z" level=info msg="[cloudwatch 0] plugin parameter log_key = 'log'"
2023-03-14T19:18:46.053+05:30
time="2023-03-14T13:48:46Z" level=info msg="[cloudwatch 0] plugin parameter role_arn = ''"
time="2023-03-14T13:48:46Z" level=info msg="[cloudwatch 0] plugin parameter role_arn = ''"
2023-03-14T19:18:46.053+05:30
time="2023-03-14T13:48:46Z" level=info msg="[cloudwatch 0] plugin parameter auto_create_group = 'true'"
time="2023-03-14T13:48:46Z" level=info msg="[cloudwatch 0] plugin parameter auto_create_group = 'true'"
2023-03-14T19:18:46.053+05:30
time="2023-03-14T13:48:46Z" level=info msg="[cloudwatch 0] plugin parameter auto_create_stream = 'true'"
time="2023-03-14T13:48:46Z" level=info msg="[cloudwatch 0] plugin parameter auto_create_stream = 'true'"
2023-03-14T19:18:46.053+05:30
time="2023-03-14T13:48:46Z" level=info msg="[cloudwatch 0] plugin parameter new_log_group_tags = ''"
time="2023-03-14T13:48:46Z" level=info msg="[cloudwatch 0] plugin parameter new_log_group_tags = ''"
2023-03-14T19:18:46.053+05:30
time="2023-03-14T13:48:46Z" level=info msg="[cloudwatch 0] plugin parameter log_retention_days = '0'"
time="2023-03-14T13:48:46Z" level=info msg="[cloudwatch 0] plugin parameter log_retention_days = '0'"
2023-03-14T19:18:46.053+05:30
time="2023-03-14T13:48:46Z" level=info msg="[cloudwatch 0] plugin parameter endpoint = ''"
time="2023-03-14T13:48:46Z" level=info msg="[cloudwatch 0] plugin parameter endpoint = ''"
2023-03-14T19:18:46.053+05:30
time="2023-03-14T13:48:46Z" level=info msg="[cloudwatch 0] plugin parameter sts_endpoint = ''"
time="2023-03-14T13:48:46Z" level=info msg="[cloudwatch 0] plugin parameter sts_endpoint = ''"
2023-03-14T19:18:46.053+05:30
time="2023-03-14T13:48:46Z" level=info msg="[cloudwatch 0] plugin parameter external_id = ''"
time="2023-03-14T13:48:46Z" level=info msg="[cloudwatch 0] plugin parameter external_id = ''"
2023-03-14T19:18:46.053+05:30
time="2023-03-14T13:48:46Z" level=info msg="[cloudwatch 0] plugin parameter credentials_endpoint = "
time="2023-03-14T13:48:46Z" level=info msg="[cloudwatch 0] plugin parameter credentials_endpoint = "
2023-03-14T19:18:46.053+05:30
time="2023-03-14T13:48:46Z" level=info msg="[cloudwatch 0] plugin parameter log_format = ''"
time="2023-03-14T13:48:46Z" level=info msg="[cloudwatch 0] plugin parameter log_format = ''"
2023-03-14T19:18:46.057+05:30
[2023/03/14 13:48:46] [ info] [input:forward:forward.2] listening on 127.0.0.1:24224
[2023/03/14 13:48:46] [ info] [input:forward:forward.2] listening on 127.0.0.1:24224
2023-03-14T19:18:46.057+05:30
[2023/03/14 13:48:46] [debug] [cloudwatch:cloudwatch.0] created event channels: read=36 write=37
[2023/03/14 13:48:46] [debug] [cloudwatch:cloudwatch.0] created event channels: read=36 write=37
2023-03-14T19:18:46.057+05:30
[2023/03/14 13:48:46] [debug] [opensearch:opensearch.1] created event channels: read=38 write=39
[2023/03/14 13:48:46] [debug] [opensearch:opensearch.1] created event channels: read=38 write=39
2023-03-14T19:18:46.063+05:30
[2023/03/14 13:48:46] [debug] [out_es] Enabled AWS Auth
[2023/03/14 13:48:46] [debug] [out_es] Enabled AWS Auth
2023-03-14T19:18:46.082+05:30
[2023/03/14 13:48:46] [debug] [aws_credentials] Initialized Env Provider in standard chain
[2023/03/14 13:48:46] [debug] [aws_credentials] Initialized Env Provider in standard chain
2023-03-14T19:18:46.083+05:30
[2023/03/14 13:48:46] [debug] [aws_credentials] Initialized AWS Profile Provider in standard chain
[2023/03/14 13:48:46] [debug] [aws_credentials] Initialized AWS Profile Provider in standard chain
2023-03-14T19:18:46.083+05:30
[2023/03/14 13:48:46] [debug] [aws_credentials] Not initializing EKS provider because AWS_ROLE_ARN was not set
[2023/03/14 13:48:46] [debug] [aws_credentials] Not initializing EKS provider because AWS_ROLE_ARN was not set
2023-03-14T19:18:46.083+05:30
[2023/03/14 13:48:46] [debug] [aws_credentials] Configuring HTTP provider with 169.254.170.2:80/v2/credentials/c60fa98d-0aca-4bb9-a1cd-984b6a4d9a10
[2023/03/14 13:48:46] [debug] [aws_credentials] Configuring HTTP provider with 169.254.170.2:80/v2/credentials/c60fa98d-0aca-4bb9-a1cd-984b6a4d9a10
2023-03-14T19:18:46.083+05:30
[2023/03/14 13:48:46] [debug] [aws_credentials] Initialized ECS Provider in standard chain
[2023/03/14 13:48:46] [debug] [aws_credentials] Initialized ECS Provider in standard chain
2023-03-14T19:18:46.083+05:30
[2023/03/14 13:48:46] [debug] [aws_credentials] Initialized EC2 Provider in standard chain
[2023/03/14 13:48:46] [debug] [aws_credentials] Initialized EC2 Provider in standard chain
2023-03-14T19:18:46.083+05:30
[2023/03/14 13:48:46] [debug] [aws_credentials] Sync called on the http provider
[2023/03/14 13:48:46] [debug] [aws_credentials] Sync called on the http provider
2023-03-14T19:18:46.083+05:30
[2023/03/14 13:48:46] [debug] [aws_credentials] Sync called on the EC2 provider
[2023/03/14 13:48:46] [debug] [aws_credentials] Sync called on the EC2 provider
2023-03-14T19:18:46.083+05:30
[2023/03/14 13:48:46] [debug] [aws_credentials] Init called on the env provider
[2023/03/14 13:48:46] [debug] [aws_credentials] Init called on the env provider
2023-03-14T19:18:46.083+05:30
[2023/03/14 13:48:46] [debug] [aws_credentials] Init called on the profile provider
[2023/03/14 13:48:46] [debug] [aws_credentials] Init called on the profile provider
2023-03-14T19:18:46.083+05:30
[2023/03/14 13:48:46] [debug] [aws_credentials] Reading shared config file.
[2023/03/14 13:48:46] [debug] [aws_credentials] Reading shared config file.
2023-03-14T19:18:46.083+05:30
[2023/03/14 13:48:46] [debug] [aws_credentials] Shared config file /root/.aws/config does not exist
[2023/03/14 13:48:46] [debug] [aws_credentials] Shared config file /root/.aws/config does not exist
2023-03-14T19:18:46.083+05:30
[2023/03/14 13:48:46] [debug] [aws_credentials] Reading shared credentials file.
[2023/03/14 13:48:46] [debug] [aws_credentials] Reading shared credentials file.
2023-03-14T19:18:46.083+05:30
[2023/03/14 13:48:46] [debug] [aws_credentials] Shared credentials file /root/.aws/credentials does not exist
[2023/03/14 13:48:46] [debug] [aws_credentials] Shared credentials file /root/.aws/credentials does not exist
2023-03-14T19:18:46.083+05:30
[2023/03/14 13:48:46] [debug] [aws_credentials] Init called on the http provider
[2023/03/14 13:48:46] [debug] [aws_credentials] Init called on the http provider
2023-03-14T19:18:46.087+05:30
[2023/03/14 13:48:46] [debug] [http_client] not using http_proxy for header
[2023/03/14 13:48:46] [debug] [http_client] not using http_proxy for header
2023-03-14T19:18:46.088+05:30
[2023/03/14 13:48:46] [debug] [upstream] KA connection utils: change the color of ] and remove unused variable #40 to 169.254.170.2:80 is now available
[2023/03/14 13:48:46] [debug] [upstream] KA connection utils: change the color of ] and remove unused variable #40 to 169.254.170.2:80 is now available
2023-03-14T19:18:46.088+05:30
[2023/03/14 13:48:46] [debug] [aws_credentials] Async called on the http provider
[2023/03/14 13:48:46] [debug] [aws_credentials] Async called on the http provider
2023-03-14T19:18:46.088+05:30
[2023/03/14 13:48:46] [debug] [aws_credentials] Async called on the EC2 provider
[2023/03/14 13:48:46] [debug] [aws_credentials] Async called on the EC2 provider
2023-03-14T19:18:46.088+05:30
[2023/03/14 13:48:46] [debug] [aws_credentials] upstream_set called on the http provider
[2023/03/14 13:48:46] [debug] [aws_credentials] upstream_set called on the http provider
2023-03-14T19:18:46.088+05:30
[2023/03/14 13:48:46] [debug] [aws_credentials] upstream_set called on the EC2 provider
[2023/03/14 13:48:46] [debug] [aws_credentials] upstream_set called on the EC2 provider
2023-03-14T19:18:46.088+05:30
[2023/03/14 13:48:46] [debug] [output:opensearch:opensearch.1] host=vpc-uai2005812-smartshop-dev-log-h4ciz4npheebnofcj3oy4p7ot4.us-east-1.es.amazonaws.com port=443 uri=/_bulk index=smartshop-router-management-dev-%Y-%m-%d type=_doc
[2023/03/14 13:48:46] [debug] [output:opensearch:opensearch.1] host=vpc-uai2005812-smartshop-dev-log-h4ciz4npheebnofcj3oy4p7ot4.us-east-1.es.amazonaws.com port=443 uri=/_bulk index=smartshop-router-management-dev-%Y-%m-%d type=_doc
2023-03-14T19:18:46.088+05:30
[2023/03/14 13:48:46] [debug] [null:null.2] created event channels: read=41 write=42
[2023/03/14 13:48:46] [debug] [null:null.2] created event channels: read=41 write=42
2023-03-14T19:18:46.088+05:30
[2023/03/14 13:48:46] [debug] [router] match rule tcp.0:cloudwatch.0
[2023/03/14 13:48:46] [debug] [router] match rule tcp.0:cloudwatch.0
2023-03-14T19:18:46.088+05:30
[2023/03/14 13:48:46] [debug] [router] match rule tcp.0:opensearch.1
[2023/03/14 13:48:46] [debug] [router] match rule tcp.0:opensearch.1
2023-03-14T19:18:46.088+05:30
[2023/03/14 13:48:46] [debug] [router] match rule tcp.0:null.2
[2023/03/14 13:48:46] [debug] [router] match rule tcp.0:null.2
2023-03-14T19:18:46.089+05:30
[2023/03/14 13:48:46] [debug] [router] match rule forward.1:cloudwatch.0
[2023/03/14 13:48:46] [debug] [router] match rule forward.1:cloudwatch.0
2023-03-14T19:18:46.089+05:30
[2023/03/14 13:48:46] [debug] [router] match rule forward.1:opensearch.1
[2023/03/14 13:48:46] [debug] [router] match rule forward.1:opensearch.1
2023-03-14T19:18:46.089+05:30
[2023/03/14 13:48:46] [debug] [router] match rule forward.2:cloudwatch.0
[2023/03/14 13:48:46] [debug] [router] match rule forward.2:cloudwatch.0
2023-03-14T19:18:46.089+05:30
[2023/03/14 13:48:46] [debug] [router] match rule forward.2:opensearch.1
[2023/03/14 13:48:46] [debug] [router] match rule forward.2:opensearch.1
2023-03-14T19:18:46.089+05:30
[2023/03/14 13:48:46] [ info] [sp] stream processor started
[2023/03/14 13:48:46] [ info] [sp] stream processor started
2023-03-14T19:18:46.100+05:30
[2023/03/14 13:48:46] [ info] [output:null:null.2] worker #0 started
[2023/03/14 13:48:46] [ info] [output:null:null.2] worker #0 started
2023-03-14T19:18:46.775+05:30
[2023/03/14 13:48:46] [debug] [input chunk] update output instances with new chunk size diff=127
[2023/03/14 13:48:46] [debug] [input chunk] update output instances with new chunk size diff=127
2023-03-14T19:18:47.297+05:30
[2023/03/14 13:48:47] [debug] [task] created task=0x7f0c9dd07c20 id=0 OK
[2023/03/14 13:48:47] [debug] [task] created task=0x7f0c9dd07c20 id=0 OK
2023-03-14T19:18:47.484+05:30
time="2023-03-14T13:48:47Z" level=info msg="[cloudwatch 0] Log group uai2005812/smartshop/dev-router-management-service-v1 already exists\n"
time="2023-03-14T13:48:47Z" level=info msg="[cloudwatch 0] Log group uai2005812/smartshop/dev-router-management-service-v1 already exists\n"
2023-03-14T19:19:08.940+05:30
[2023/03/14 13:49:08] [engine] caught signal (SIGTERM)
[2023/03/14 13:49:08] [engine] caught signal (SIGTERM)
2023-03-14T19:19:08.940+05:30
[2023/03/14 13:49:08] [ info] [input] pausing forward.1
[2023/03/14 13:49:08] [ info] [input] pausing forward.1
2023-03-14T19:19:08.940+05:30
[2023/03/14 13:49:08] [ info] [input] pausing forward.2
[2023/03/14 13:49:08] [ info] [input] pausing forward.2
2023-03-14T19:19:13.781+05:30
time="2023-03-14T13:49:13Z" level=info msg="[cloudwatch 0] Log stream -router-management-firelens-164ab29dd5b14cafaf47c89f58c101ac does not exist in log group uai2005812/smartshop/dev-router-management-service-v1"
time="2023-03-14T13:49:13Z" level=info msg="[cloudwatch 0] Log stream -router-management-firelens-164ab29dd5b14cafaf47c89f58c101ac does not exist in log group uai2005812/smartshop/dev-router-management-service-v1"
2023-03-14T19:19:13.808+05:30
time="2023-03-14T13:49:13Z" level=info msg="[cloudwatch 0] Created log stream -router-management-firelens-164ab29dd5b14cafaf47c89f58c101ac in group uai2005812/smartshop/dev-router-management-service-v1"
time="2023-03-14T13:49:13Z" level=info msg="[cloudwatch 0] Created log stream -router-management-firelens-164ab29dd5b14cafaf47c89f58c101ac in group uai2005812/smartshop/dev-router-management-service-v1"
2023-03-14T19:19:13.853+05:30
[2023/03/14 13:49:13] [debug] [upstream] KA connection utils: change the color of ] and remove unused variable #40 to 169.254.170.2:80 has been disconnected by the remote service
[2023/03/14 13:49:13] [debug] [upstream] KA connection utils: change the color of ] and remove unused variable #40 to 169.254.170.2:80 has been disconnected by the remote service
2023-03-14T19:19:13.856+05:30
[2023/03/14 13:49:13] [ warn] [engine] service will shutdown in max 30 seconds
[2023/03/14 13:49:13] [ warn] [engine] service will shutdown in max 30 seconds
2023-03-14T19:19:13.856+05:30
[2023/03/14 13:49:13] [debug] [out flush] cb_destroy coro_id=0
[2023/03/14 13:49:13] [debug] [out flush] cb_destroy coro_id=0
2023-03-14T19:19:13.864+05:30
[2023/03/14 13:49:13] [debug] [http_client] not using http_proxy for header
[2023/03/14 13:49:13] [debug] [http_client] not using http_proxy for header
2023-03-14T19:19:13.864+05:30
[2023/03/14 13:49:13] [debug] [output:opensearch:opensearch.1] Signing request with AWS Sigv4
[2023/03/14 13:49:13] [debug] [output:opensearch:opensearch.1] Signing request with AWS Sigv4
2023-03-14T19:19:13.864+05:30
[2023/03/14 13:49:13] [debug] [aws_credentials] Retrieving credentials from the HTTP provider..
[2023/03/14 13:49:13] [debug] [aws_credentials] Retrieving credentials from the HTTP provider..
2023-03-14T19:19:13.886+05:30
[2023/03/14 13:49:13] [debug] [output:opensearch:opensearch.1] HTTP Status=200 URI=/_bulk
[2023/03/14 13:49:13] [debug] [output:opensearch:opensearch.1] HTTP Status=200 URI=/_bulk
2023-03-14T19:19:13.886+05:30
[2023/03/14 13:49:13] [debug] [output:opensearch:opensearch.1] OpenSearch response
[2023/03/14 13:49:13] [debug] [output:opensearch:opensearch.1] OpenSearch response
2023-03-14T19:19:13.886+05:30
{
"took": 5,
"errors": false,
"items": [
{
"create": {
"_index": "smartshop-router-management-dev-2023-03-14",
"_type": "_doc",
"_id": "nr9h4IYBoiVndgIh14UX",
"_version": 1,
"result": "created",
"_shards": {
"total": 2,
"successful": 2,
"failed": 0
},
"_seq_no": 1379,
"_primary_term": 1,
"status": 201
}
}
]
}
{"took":5,"errors":false,"items":[{"create":{"_index":"smartshop-router-management-dev-2023-03-14","_type":"_doc","_id":"nr9h4IYBoiVndgIh14UX","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":1379,"_primary_term":1,"status":201}}]}
2023-03-14T19:19:13.886+05:30
[2023/03/14 13:49:13] [debug] [upstream] KA connection hello_world: Fixed segfault caused by flb_lib_stop() #35 to vpc-uai2005812-smartshop-dev-log-h4ciz4npheebnofcj3oy4p7ot4.us-east-1.es.amazonaws.com:443 is now available
[2023/03/14 13:49:13] [debug] [upstream] KA connection hello_world: Fixed segfault caused by flb_lib_stop() #35 to vpc-uai2005812-smartshop-dev-log-h4ciz4npheebnofcj3oy4p7ot4.us-east-1.es.amazonaws.com:443 is now available
2023-03-14T19:19:13.886+05:30
[2023/03/14 13:49:13] [debug] [out flush] cb_destroy coro_id=0
[2023/03/14 13:49:13] [debug] [out flush] cb_destroy coro_id=0
2023-03-14T19:19:13.886+05:30
[2023/03/14 13:49:13] [debug] [task] destroy task=0x7f0c9dd07c20 (task_id=0)
[2023/03/14 13:49:13] [debug] [task] destroy task=0x7f0c9dd07c20 (task_id=0)
2023-03-14T19:19:14.293+05:30
[2023/03/14 13:49:14] [ info] [engine] service has stopped (0 pending tasks)
[2023/03/14 13:49:14] [ info] [engine] service has stopped (0 pending tasks)
2023-03-14T19:19:14.293+05:30
[2023/03/14 13:49:14] [debug] [GO] running exit callback
[2023/03/14 13:49:14] [debug] [GO] running exit callback
2023-03-14T19:19:14.293+05:30
[2023/03/14 13:49:14] [debug] [socket] could not validate socket status for hello_world: Fixed segfault caused by flb_lib_stop() #35
[2023/03/14 13:49:14] [debug] [socket] could not validate socket status for hello_world: Fixed segfault caused by flb_lib_stop() #35
2023-03-14T19:19:14.294+05:30
[2023/03/14 13:49:14] [ info] [output:null:null.2] thread worker #0 stopping...
[2023/03/14 13:49:14] [ info] [output:null:null.2] thread worker #0 stopping...
2023-03-14T19:19:14.294+05:30
[2023/03/14 13:49:14] [ info] [output:null:null.2] thread worker #0 stopped
[2023/03/14 13:49:14] [ info] [output:null:null.2] thread worker #0 stopped

vijaynauti Mar 23, 2023
Author

@PettitWesley Can you check above please.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fluent-bit on ECS Fargate - intermittent issue #6949

{{title}}

Replies: 1 comment 10 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

Fluent-bit on ECS Fargate - intermittent issue #6949

vijaynauti Mar 3, 2023

Replies: 1 comment · 10 replies

PettitWesley Mar 3, 2023 Collaborator

PettitWesley Mar 8, 2023 Collaborator

vijaynauti Mar 10, 2023 Author

@PettitWesley: I have elaborated the issue on above, Pasting here again. Checked with AWS team as well. but confirmed its issue with Fluent-bit level.

PettitWesley Mar 10, 2023 Collaborator

vijaynauti Mar 14, 2023 Author

vijaynauti Mar 23, 2023 Author

vijaynauti
Mar 3, 2023

Replies: 1 comment 10 replies

PettitWesley
Mar 3, 2023
Collaborator

PettitWesley Mar 8, 2023
Collaborator

vijaynauti Mar 10, 2023
Author

PettitWesley Mar 10, 2023
Collaborator

vijaynauti Mar 14, 2023
Author

vijaynauti Mar 23, 2023
Author