Skip to content

Commit

Permalink
adjustment adapter name from selectdb to doris
Browse files Browse the repository at this point in the history
  • Loading branch information
catpineapple committed Nov 17, 2023
1 parent af640cc commit 1e54eb0
Show file tree
Hide file tree
Showing 62 changed files with 1,771 additions and 517 deletions.
15 changes: 12 additions & 3 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
### Python template
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
Expand All @@ -20,7 +21,6 @@ parts/
sdist/
var/
wheels/
pip-wheel-metadata/
share/python-wheels/
*.egg-info/
.installed.cfg
Expand Down Expand Up @@ -50,7 +50,7 @@ coverage.xml
*.py,cover
.hypothesis/
.pytest_cache/
test.env
cover/

# Translations
*.mo
Expand All @@ -73,6 +73,7 @@ instance/
docs/_build/

# PyBuilder
.pybuilder/
target/

# Jupyter Notebook
Expand All @@ -83,7 +84,9 @@ profile_default/
ipython_config.py

# pyenv
.python-version
# For a library or package, you might want to ignore these files since the code is
# intended to run in multiple environments; otherwise, check them in:
# .python-version

# pipenv
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
Expand Down Expand Up @@ -128,3 +131,9 @@ dmypy.json

# Pyre type checker
.pyre/

# pytype static type analyzer
.pytype/

# Cython debug symbols
cython_debug/
3 changes: 3 additions & 0 deletions .idea/.gitignore

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

413 changes: 413 additions & 0 deletions .idea/dbnavigator.xml

Large diffs are not rendered by default.

13 changes: 13 additions & 0 deletions .idea/dbt-doris.iml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 6 additions & 0 deletions .idea/inspectionProfiles/profiles_settings.xml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 4 additions & 0 deletions .idea/misc.xml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 8 additions & 0 deletions .idea/modules.xml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 6 additions & 0 deletions .idea/vcs.xml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@

# For more on configuring pre-commit hooks (see https://pre-commit.com/)

# TODO: remove global exclusion of tests when testing overhaul is complete
Expand Down Expand Up @@ -52,4 +53,3 @@ repos:
args: [--show-error-codes, --pretty, --ignore-missing-imports]
files: ^dbt/adapters
language: system

10 changes: 4 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,17 +6,15 @@

dbt is the T in ELT. Organize, cleanse, denormalize, filter, rename, and pre-aggregate the raw data in your warehouse so that it's ready for analysis.

## SelectDB
This repo contains the base code to help you start to build out your dbt-selectdb adapter plugin, for more information on how to build out the adapter please follow the [docs](https://docs.getdbt.com/docs/contributing/building-a-new-adapter)
## Apache Doris
This repo contains the base code to help you start to build out your dbt-doris adapter plugin, for more information on how to build out the adapter please follow the [docs](https://docs.getdbt.com/docs/contributing/building-a-new-adapter)

** Note ** this `README` is meant to be replaced with what information would be required to use your adpater once your at a point todo so.

** Note **
### Adapter Scaffold default Versioning
This adapter plugin follows [semantic versioning](https://semver.org/). The first version of this plugin is v0.1.0, in order to be compatible with dbt Core v1.3.0.

It's also brand new! For selectdb-specific functionality, we will aim for backwards-compatibility wherever possible. We are likely to be iterating more quickly than most major-version-1 software projects. To that end, backwards-incompatible changes will be clearly communicated and limited to minor versions (once every three months).

## Getting Started

#### Setting up Locally
Expand All @@ -30,7 +28,7 @@ It's also brand new! For selectdb-specific functionality, we will aim for backwa
## Join the dbt Community

- Be part of the conversation in the [dbt Community Slack](http://community.getdbt.com/)
- If one doesn't exist feel free to request a #db-selectdb channel be made in the [#channel-requests](https://getdbt.slack.com/archives/C01D8J8AJDA) on dbt community slack channel.
- If one doesn't exist feel free to request a #db-doris channel be made in the [#channel-requests](https://getdbt.slack.com/archives/C01D8J8AJDA) on dbt community slack channel.
- Read more on the [dbt Community Discourse](https://discourse.getdbt.com)

## Reporting bugs and contributing code
Expand All @@ -40,4 +38,4 @@ It's also brand new! For selectdb-specific functionality, we will aim for backwa

## Code of Conduct

Everyone interacting in the dbt project's codebases, issue trackers, chat rooms, and mailing lists is expected to follow the [dbt Code of Conduct](https://community.getdbt.com/code-of-conduct).# dbt-selectdb
Everyone interacting in the dbt project's codebases, issue trackers, chat rooms, and mailing lists is expected to follow the [dbt Code of Conduct](https://community.getdbt.com/code-of-conduct).# dbt-doris
33 changes: 33 additions & 0 deletions dbt/adapters/doris/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
#!/usr/bin/env python
# encoding: utf-8

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

from dbt.adapters.doris.connections import DorisConnectionManager # noqa
from dbt.adapters.doris.connections import DorisCredentials
from dbt.adapters.doris.impl import DorisAdapter

from dbt.adapters.base import AdapterPlugin
from dbt.include import doris


Plugin = AdapterPlugin(
adapter=DorisAdapter,
credentials=DorisCredentials,
include_path=doris.PACKAGE_PATH,
)
25 changes: 25 additions & 0 deletions dbt/adapters/doris/__version__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
#!/usr/bin/env python
# encoding: utf-8

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.


# this 'version' must be set !!!
# otherwise the adapters will not be found after the 'dbt init xxx' command

version = "0.3.4"
33 changes: 33 additions & 0 deletions dbt/adapters/doris/column.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
#!/usr/bin/env python
# encoding: utf-8

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

from dataclasses import dataclass

from dbt.adapters.base.column import Column


@dataclass
class DorisColumn(Column):
@property
def quoted(self) -> str:
return "`{}`".format(self.column)

def __repr__(self) -> str:
return f"<DorisColumn {self.name} ({self.data_type})>"
Original file line number Diff line number Diff line change
@@ -1,3 +1,23 @@
#!/usr/bin/env python
# encoding: utf-8

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

from contextlib import contextmanager
from dataclasses import dataclass
from typing import ContextManager, Optional, Union
Expand All @@ -10,11 +30,11 @@
from dbt.contracts.connection import AdapterResponse, Connection, ConnectionState
from dbt.events import AdapterLogger

logger = AdapterLogger("selectdb")
logger = AdapterLogger("doris")


@dataclass
class SelectdbCredentials(Credentials):
class DorisCredentials(Credentials):
host: str = "127.0.0.1"
port: int = 9030
username: str = "root"
Expand All @@ -25,7 +45,7 @@ class SelectdbCredentials(Credentials):

@property
def type(self):
return "selectdb"
return "doris"

def _connection_keys(self):
return "host", "port", "user", "schema"
Expand All @@ -36,16 +56,16 @@ def unique_field(self) -> str:

def __post_init__(self):
if self.database is not None and self.database != self.schema:
raise exceptions.RuntimeException(
raise exceptions.DbtRuntimeError(
f" schema: {self.schema} \n"
f" database: {self.database} \n"
f"On SelectDB, database must be omitted or have the same value as"
f"On Doris, database must be omitted or have the same value as"
f" schema."
)


class SelectdbConnectionManager(SQLConnectionManager):
TYPE = "selectdb"
class DorisConnectionManager(SQLConnectionManager):
TYPE = "doris"

@classmethod
def open(cls, connection: Connection) -> Connection:
Expand Down Expand Up @@ -82,7 +102,7 @@ def open(cls, connection: Connection) -> Connection:
connection.handle = None
connection.state = 'fail'

raise dbt.exceptions.FailedToConnectException(str(e))
raise exceptions.FailedToConnectError(str(e))
return connection

@classmethod
Expand All @@ -106,23 +126,24 @@ def get_response(cls, cursor) -> Union[AdapterResponse, str]:
rows_affected=num_rows,
)

@contextmanager
def exception_handler(self, sql: str) -> ContextManager:
@contextmanager
def exception_handler(self, sql: str) -> ContextManager:
try:
yield
except mysql.connector.DatabaseError as e:
logger.debug(f"SelectDB database error: {e}, sql: {sql}")
raise exceptions.DatabaseException(str(e)) from e
logger.debug(f"Doris database error: {e}, sql: {sql}")
raise exceptions.DbtDatabaseError(str(e)) from e
except Exception as e:
logger.debug(f"Error running SQL: {sql}")
if isinstance(e, exceptions.RuntimeException):
if isinstance(e, exceptions.DbtRuntimeError):
raise e
raise exceptions.RuntimeException(str(e)) from e
raise exceptions.DbtRuntimeError(str(e)) from e

@classmethod
def begin(self):
"""
SelectDB's inserting always transaction, ignore it
https://doris.apache.org/docs/data-operate/import/import-scenes/load-atomicity/
Doris's inserting always transaction, ignore it
"""
pass

Expand Down
Loading

0 comments on commit 1e54eb0

Please sign in to comment.