Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

Commit 1a2d2d7

Browse filesBrowse files
Ricardo Mendestswast
Ricardo Mendes
authored and
tswast
committed
Add samples for Data Catalog lookup_entry (GoogleCloudPlatform#2148)
* Add samples for Data Catalog lookup_entry * Add tests for Data Catalog lookup_entry * Add samples for lookup_entry by SQL Resource * Add README.rst * Improve command line interface * Removed the "lookup-" prefix from commands * Handle the --sql-resource optional argument by subparsers * Refer to GCP public assets in tests
1 parent b4b6816 commit 1a2d2d7
Copy full SHA for 1a2d2d7

File tree

Expand file treeCollapse file tree

5 files changed

+348
-0
lines changed
Filter options
Expand file treeCollapse file tree

5 files changed

+348
-0
lines changed

‎datacatalog/cloud-client/README.rst

Copy file name to clipboard
+121Lines changed: 121 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,121 @@
1+
.. This file is automatically generated. Do not edit this file directly.
2+
3+
Google Cloud Data Catalog Python Samples
4+
===============================================================================
5+
6+
.. image:: https://gstatic.com/cloudssh/images/open-btn.png
7+
:target: https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/GoogleCloudPlatform/python-docs-samples&page=editor&open_in_editor=datacatalog/cloud-client/README.rst
8+
9+
10+
This directory contains samples for Google Cloud Data Catalog. `Google Cloud Data Catalog`_ is a fully managed and scalable metadata management service that empowers organizations to quickly discover, manage, and understand all their data in Google Cloud.
11+
12+
13+
14+
15+
.. _Google Cloud Data Catalog: https://cloud.google.com/data-catalog/docs
16+
17+
Setup
18+
-------------------------------------------------------------------------------
19+
20+
21+
Authentication
22+
++++++++++++++
23+
24+
This sample requires you to have authentication setup. Refer to the
25+
`Authentication Getting Started Guide`_ for instructions on setting up
26+
credentials for applications.
27+
28+
.. _Authentication Getting Started Guide:
29+
https://cloud.google.com/docs/authentication/getting-started
30+
31+
Install Dependencies
32+
++++++++++++++++++++
33+
34+
#. Clone python-docs-samples and change directory to the sample directory you want to use.
35+
36+
.. code-block:: bash
37+
38+
$ git clone https://github.com/GoogleCloudPlatform/python-docs-samples.git
39+
40+
#. Install `pip`_ and `virtualenv`_ if you do not already have them. You may want to refer to the `Python Development Environment Setup Guide`_ for Google Cloud Platform for instructions.
41+
42+
.. _Python Development Environment Setup Guide:
43+
https://cloud.google.com/python/setup
44+
45+
#. Create a virtualenv. Samples are compatible with Python 2.7 and 3.4+.
46+
47+
.. code-block:: bash
48+
49+
$ virtualenv env
50+
$ source env/bin/activate
51+
52+
#. Install the dependencies needed to run the samples.
53+
54+
.. code-block:: bash
55+
56+
$ pip install -r requirements.txt
57+
58+
.. _pip: https://pip.pypa.io/
59+
.. _virtualenv: https://virtualenv.pypa.io/
60+
61+
Samples
62+
-------------------------------------------------------------------------------
63+
64+
Lookup entry
65+
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
66+
67+
.. image:: https://gstatic.com/cloudssh/images/open-btn.png
68+
:target: https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/GoogleCloudPlatform/python-docs-samples&page=editor&open_in_editor=datacatalog/cloud-client/lookup_entry.py,datacatalog/cloud-client/README.rst
69+
70+
71+
72+
73+
To run this sample:
74+
75+
.. code-block:: bash
76+
77+
$ python lookup_entry.py
78+
79+
usage: lookup_entry.py [-h]
80+
project_id
81+
{bigquery-dataset,bigquery-table,pubsub-topic} ...
82+
83+
This application demonstrates how to perform basic operations on entries
84+
with the Cloud Data Catalog API.
85+
86+
For more information, see the README.md under /datacatalog and the
87+
documentation at https://cloud.google.com/data-catalog/docs.
88+
89+
positional arguments:
90+
project_id Your Google Cloud project ID
91+
{bigquery-dataset,bigquery-table,pubsub-topic}
92+
bigquery-dataset Retrieves Data Catalog entry for the given BigQuery
93+
Dataset.
94+
bigquery-table Retrieves Data Catalog entry for the given BigQuery
95+
Table.
96+
pubsub-topic Retrieves Data Catalog entry for the given Pub/Sub
97+
Topic.
98+
99+
optional arguments:
100+
-h, --help show this help message and exit
101+
102+
103+
104+
105+
106+
The client library
107+
-------------------------------------------------------------------------------
108+
109+
This sample uses the `Google Cloud Client Library for Python`_.
110+
You can read the documentation for more details on API usage and use GitHub
111+
to `browse the source`_ and `report issues`_.
112+
113+
.. _Google Cloud Client Library for Python:
114+
https://googlecloudplatform.github.io/google-cloud-python/
115+
.. _browse the source:
116+
https://github.com/GoogleCloudPlatform/google-cloud-python
117+
.. _report issues:
118+
https://github.com/GoogleCloudPlatform/google-cloud-python/issues
119+
120+
121+
.. _Google Cloud SDK: https://cloud.google.com/sdk/

‎datacatalog/cloud-client/README.rst.in

Copy file name to clipboard
+23Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
# This file is used to generate README.rst
2+
3+
product:
4+
name: Google Cloud Data Catalog
5+
short_name: Data Catalog
6+
url: https://cloud.google.com/data-catalog/docs
7+
description: >
8+
`Google Cloud Data Catalog`_ is a fully managed and scalable metadata
9+
management service that empowers organizations to quickly discover, manage,
10+
and understand all their data in Google Cloud.
11+
12+
setup:
13+
- auth
14+
- install_deps
15+
16+
samples:
17+
- name: Lookup entry
18+
file: lookup_entry.py
19+
show_help: true
20+
21+
cloud_client_library: true
22+
23+
folder: datacatalog/cloud-client

‎datacatalog/cloud-client/lookup_entry.py

Copy file name to clipboard
+150Lines changed: 150 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,150 @@
1+
#!/usr/bin/env python
2+
3+
# Copyright 2019 Google Inc. All Rights Reserved.
4+
#
5+
# Licensed under the Apache License, Version 2.0 (the "License");
6+
# you may not use this file except in compliance with the License.
7+
# You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
17+
"""This application demonstrates how to perform basic operations on entries
18+
with the Cloud Data Catalog API.
19+
20+
For more information, see the README.md under /datacatalog and the
21+
documentation at https://cloud.google.com/data-catalog/docs.
22+
"""
23+
24+
import argparse
25+
26+
27+
def lookup_bigquery_dataset(project_id, dataset_id):
28+
"""Retrieves Data Catalog entry for the given BigQuery Dataset."""
29+
from google.cloud import datacatalog_v1beta1
30+
31+
datacatalog = datacatalog_v1beta1.DataCatalogClient()
32+
33+
resource_name = '//bigquery.googleapis.com/projects/{}/datasets/{}'\
34+
.format(project_id, dataset_id)
35+
36+
return datacatalog.lookup_entry(linked_resource=resource_name)
37+
38+
39+
def lookup_bigquery_dataset_sql_resource(project_id, dataset_id):
40+
"""Retrieves Data Catalog entry for the given BigQuery Dataset by
41+
sql_resource.
42+
"""
43+
from google.cloud import datacatalog_v1beta1
44+
45+
datacatalog = datacatalog_v1beta1.DataCatalogClient()
46+
47+
sql_resource = 'bigquery.dataset.`{}`.`{}`'.format(project_id, dataset_id)
48+
49+
return datacatalog.lookup_entry(sql_resource=sql_resource)
50+
51+
52+
def lookup_bigquery_table(project_id, dataset_id, table_id):
53+
"""Retrieves Data Catalog entry for the given BigQuery Table."""
54+
from google.cloud import datacatalog_v1beta1
55+
56+
datacatalog = datacatalog_v1beta1.DataCatalogClient()
57+
58+
resource_name = '//bigquery.googleapis.com/projects/{}/datasets/{}' \
59+
'/tables/{}'\
60+
.format(project_id, dataset_id, table_id)
61+
62+
return datacatalog.lookup_entry(linked_resource=resource_name)
63+
64+
65+
def lookup_bigquery_table_sql_resource(project_id, dataset_id, table_id):
66+
"""Retrieves Data Catalog entry for the given BigQuery Table by
67+
sql_resource.
68+
"""
69+
from google.cloud import datacatalog_v1beta1
70+
71+
datacatalog = datacatalog_v1beta1.DataCatalogClient()
72+
73+
sql_resource = 'bigquery.table.`{}`.`{}`.`{}`'.format(
74+
project_id, dataset_id, table_id)
75+
76+
return datacatalog.lookup_entry(sql_resource=sql_resource)
77+
78+
79+
def lookup_pubsub_topic(project_id, topic_id):
80+
"""Retrieves Data Catalog entry for the given Pub/Sub Topic."""
81+
from google.cloud import datacatalog_v1beta1
82+
83+
datacatalog = datacatalog_v1beta1.DataCatalogClient()
84+
85+
resource_name = '//pubsub.googleapis.com/projects/{}/topics/{}'\
86+
.format(project_id, topic_id)
87+
88+
return datacatalog.lookup_entry(linked_resource=resource_name)
89+
90+
91+
def lookup_pubsub_topic_sql_resource(project_id, topic_id):
92+
"""Retrieves Data Catalog entry for the given Pub/Sub Topic by
93+
sql_resource.
94+
"""
95+
from google.cloud import datacatalog_v1beta1
96+
97+
datacatalog = datacatalog_v1beta1.DataCatalogClient()
98+
99+
sql_resource = 'pubsub.topic.`{}`.`{}`'.format(project_id, topic_id)
100+
101+
return datacatalog.lookup_entry(sql_resource=sql_resource)
102+
103+
104+
if __name__ == '__main__':
105+
parser = argparse.ArgumentParser(
106+
description=__doc__,
107+
formatter_class=argparse.RawDescriptionHelpFormatter
108+
)
109+
110+
parser.add_argument('project_id', help='Your Google Cloud project ID')
111+
112+
subparsers = parser.add_subparsers(dest='command')
113+
114+
bigquery_dataset_parser = subparsers.add_parser(
115+
'bigquery-dataset', help=lookup_bigquery_dataset.__doc__)
116+
bigquery_dataset_parser.add_argument('dataset_id')
117+
bigquery_dataset_parser.add_argument('--sql-resource', action='store_true',
118+
help='Perform lookup by SQL Resource')
119+
120+
bigquery_table_parser = subparsers.add_parser(
121+
'bigquery-table', help=lookup_bigquery_table.__doc__)
122+
bigquery_table_parser.add_argument('dataset_id')
123+
bigquery_table_parser.add_argument('table_id')
124+
bigquery_table_parser.add_argument('--sql-resource', action='store_true',
125+
help='Perform lookup by SQL Resource')
126+
127+
pubsub_topic_parser = subparsers.add_parser(
128+
'pubsub-topic', help=lookup_pubsub_topic.__doc__)
129+
pubsub_topic_parser.add_argument('topic_id')
130+
pubsub_topic_parser.add_argument('--sql-resource', action='store_true',
131+
help='Perform lookup by SQL Resource')
132+
133+
args = parser.parse_args()
134+
135+
entry = None
136+
137+
if args.command == 'bigquery-dataset':
138+
lookup_method = lookup_bigquery_dataset_sql_resource \
139+
if args.sql_resource else lookup_bigquery_dataset
140+
entry = lookup_method(args.project_id, args.dataset_id)
141+
elif args.command == 'bigquery-table':
142+
lookup_method = lookup_bigquery_table_sql_resource \
143+
if args.sql_resource else lookup_bigquery_table
144+
entry = lookup_method(args.project_id, args.dataset_id, args.table_id)
145+
elif args.command == 'pubsub-topic':
146+
lookup_method = lookup_pubsub_topic_sql_resource \
147+
if args.sql_resource else lookup_pubsub_topic
148+
entry = lookup_method(args.project_id, args.topic_id)
149+
150+
print(entry.name)

‎datacatalog/cloud-client/lookup_entry_test.py

Copy file name to clipboard
+53Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
#!/usr/bin/env python
2+
3+
# Copyright 2019 Google Inc. All Rights Reserved.
4+
#
5+
# Licensed under the Apache License, Version 2.0 (the "License");
6+
# you may not use this file except in compliance with the License.
7+
# You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
17+
import lookup_entry
18+
19+
BIGQUERY_PROJECT = 'bigquery-public-data'
20+
BIGQUERY_DATASET = 'new_york_taxi_trips'
21+
BIGQUERY_TABLE = 'taxi_zone_geom'
22+
23+
PUBSUB_PROJECT = 'pubsub-public-data'
24+
PUBSUB_TOPIC = 'taxirides-realtime'
25+
26+
27+
def test_lookup_bigquery_dataset():
28+
assert lookup_entry.lookup_bigquery_dataset(
29+
BIGQUERY_PROJECT, BIGQUERY_DATASET)
30+
31+
32+
def test_lookup_bigquery_dataset_sql_resource():
33+
assert lookup_entry.lookup_bigquery_dataset_sql_resource(
34+
BIGQUERY_PROJECT, BIGQUERY_DATASET)
35+
36+
37+
def test_lookup_bigquery_table():
38+
assert lookup_entry.lookup_bigquery_table(
39+
BIGQUERY_PROJECT, BIGQUERY_DATASET, BIGQUERY_TABLE)
40+
41+
42+
def test_lookup_bigquery_table_sql_resource():
43+
assert lookup_entry.lookup_bigquery_table_sql_resource(
44+
BIGQUERY_PROJECT, BIGQUERY_DATASET, BIGQUERY_TABLE)
45+
46+
47+
def test_lookup_pubsub_topic():
48+
assert lookup_entry.lookup_pubsub_topic(PUBSUB_PROJECT, PUBSUB_TOPIC)
49+
50+
51+
def test_lookup_pubsub_topic_sql_resource():
52+
assert lookup_entry.lookup_pubsub_topic_sql_resource(
53+
PUBSUB_PROJECT, PUBSUB_TOPIC)

‎datacatalog/cloud-client/requirements.txt

Copy file name to clipboard
+1Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
google-cloud-datacatalog==0.1.0

0 commit comments

Comments
0 (0)
Morty Proxy This is a proxified and sanitized view of the page, visit original site.