Skip to content
This repository has been archived by the owner on Dec 14, 2022. It is now read-only.

[QUESTION] Disable flink schema upload operation #535

Open
nikolasten opened this issue Apr 14, 2022 · 2 comments
Open

[QUESTION] Disable flink schema upload operation #535

nikolasten opened this issue Apr 14, 2022 · 2 comments
Assignees
Labels
type/feature Indicates new functionality

Comments

@nikolasten
Copy link

We are running a flink sink on topic that it does not have any pulsar schema. Basically we are publishing just json strings.
If we dont add service account as admin role on tenant, flink fails with :

java.lang.RuntimeException: Failed to get schema information for persistent://production/backoffice/internal.notification-bet-activity
    at org.apache.flink.streaming.connectors.pulsar.internal.SchemaUtils.uploadPulsarSchema(SchemaUtils.java:65)
    at org.apache.flink.streaming.connectors.pulsar.FlinkPulsarSinkBase.uploadSchema(FlinkPulsarSinkBase.java:302)
    at org.apache.flink.streaming.connectors.pulsar.FlinkPulsarSinkBase.open(FlinkPulsarSinkBase.java:267)
    at org.apache.flink.streaming.connectors.pulsar.FlinkPulsarSink.open(FlinkPulsarSink.java:41)
    at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:34)
    at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:102)
    at org.apache.flink.streaming.api.operators.StreamSink.open(StreamSink.java:46)
    at org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:442)
    at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:585)
    at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55)
    at org.apache.flink.streaming.runtime.tasks.StreamTask.executeRestore(StreamTask.java:565)
    at org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:650)
    at org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:540)
    at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:759)
    at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566)
    at java.base/java.lang.Thread.run(Unknown Source)
Caused by: org.apache.pulsar.client.admin.PulsarAdminException$NotAuthorizedException: Don't have permission to administrate resources on this tenant

Is it possible to disable schema uploads done by flink connector?
If not, what is preferred way to add this permission and not to add full admin access on tenant?

@nikolasten nikolasten added the type/feature Indicates new functionality label Apr 14, 2022
@nikolasten
Copy link
Author

I see that upload schema call in releases after 1.13.1.4 is removed https://github.com/streamnative/pulsar-flink/blob/release-1.13.1.4/pulsar-flink-connector/src/main/java/org/apache/flink/streaming/connectors/pulsar/FlinkPulsarSinkBase.java#L267
When will the stable version of 1.13.1.5 is going to be released? At the moment there are only release candidate versions

@nlu90
Copy link
Contributor

nlu90 commented Apr 27, 2022

@imaffe Could you help release a stable 1.13 once the pipeline is finished.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
type/feature Indicates new functionality
Projects
None yet
Development

No branches or pull requests

3 participants