前回、bqコマンドを使ってデータセットを管理できることを確認しました。今回はcloud functionsを使ってエンドポイントを実装する予定でしたが、いろいろあってAWSを利用することになりました。その中でKinesis Data FirehoseからS3にデータをPUTする前にAWS Lambdaでデータを整形する必要が出てきました。その時に気づいた（当たり前かもしれない）ことを記載します。
exports.handler = async (event, context) => { /* Process the list of records and transform them */ const output = event.records.map((record) => ({ /* This transformation is the "identity" transformation, the data is left intact */ recordId: record.recordId, result: 'Ok', data: record.data, })); console.log(`Processing completed. Successful records ${output.length}.`); return { records: output }; };
python-build: use readline from homebrew Downloading Python-3.7.9.tar.xz... -> https://www.python.org/ftp/python/3.7.9/Python-3.7.9.tar.xz Installing Python-3.7.9... python-build: use readline from homebrew Installed Python-3.7.9 to /Users/user/.anyenv/envs/pyenv/versions/3.7.9
3.7.9を利用するように設定します。
1 2 3
$ pyenv local 3.7.9 $ python --version Python 3.7.9
$ tar xzf ~/Downloads/google-cloud-sdk-312.0.0-darwin-x86_64.tar.gz -C . $ cd google-cloud-sdk $ ls LICENSE completion.zsh.inc path.bash.inc README data path.fish.inc RELEASE_NOTES deb path.zsh.inc VERSION install.bat platform bin install.sh properties completion.bash.inc lib rpm
$ cd .. $ ./google-cloud-sdk/install.sh Welcome to the Google Cloud SDK!
To help improve the quality of this product, we collect anonymized usage data and anonymized stacktraces when crashes are encountered; additional information is available at <https://cloud.google.com/sdk/usage-statistics>. This data is handled in accordance with our privacy policy <https://policies.google.com/privacy>. You may choose to opt in this collection now (by choosing 'Y' at the below prompt), or at any time in the future by running the following command:
gcloud config set disable_usage_reporting false
Do you want to help improve the Google Cloud SDK (y/N)? y
Your current Cloud SDK version is: 312.0.0 The latest available version is: 315.0.0
┌───────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ Components │ ├──────────────────┬──────────────────────────────────────────────────────┬──────────────────────────┬──────────┤ │ Status │ Name │ ID │ Size │ ├──────────────────┼──────────────────────────────────────────────────────┼──────────────────────────┼──────────┤ │ Update Available │ BigQuery Command Line Tool │ bq │ < 1 MiB │ │ Update Available │ Cloud SDK Core Libraries │ core │ 15.4 MiB │ │ Update Available │ Cloud Storage Command Line Tool │ gsutil │ 3.5 MiB │ │ Not Installed │ App Engine Go Extensions │ app-engine-go │ 4.8 MiB │ │ Not Installed │ Appctl │ appctl │ 18.5 MiB │ │ Not Installed │ Cloud Bigtable Command Line Tool │ cbt │ 7.6 MiB │ │ Not Installed │ Cloud Bigtable Emulator │ bigtable │ 6.6 MiB │ │ Not Installed │ Cloud Datalab Command Line Tool │ datalab │ < 1 MiB │ │ Not Installed │ Cloud Datastore Emulator │ cloud-datastore-emulator │ 18.4 MiB │ │ Not Installed │ Cloud Firestore Emulator │ cloud-firestore-emulator │ 42.1 MiB │ │ Not Installed │ Cloud Pub/Sub Emulator │ pubsub-emulator │ 56.3 MiB │ │ Not Installed │ Cloud SQL Proxy │ cloud_sql_proxy │ 7.3 MiB │ │ Not Installed │ Emulator Reverse Proxy │ emulator-reverse-proxy │ 14.5 MiB │ │ Not Installed │ Google Cloud Build Local Builder │ cloud-build-local │ 6.2 MiB │ │ Not Installed │ Google Container Registry's Docker credential helper │ docker-credential-gcr │ 1.8 MiB │ │ Not Installed │ Kind │ kind │ 4.4 MiB │ │ Not Installed │ Kustomize │ kustomize │ 22.8 MiB │ │ Not Installed │ Minikube │ minikube │ 24.0 MiB │ │ Not Installed │ Nomos CLI │ nomos │ 17.6 MiB │ │ Not Installed │ Skaffold │ skaffold │ 15.3 MiB │ │ Not Installed │ anthos-auth │ anthos-auth │ 16.2 MiB │ │ Not Installed │ gcloud Alpha Commands │ alpha │ < 1 MiB │ │ Not Installed │ gcloud Beta Commands │ beta │ < 1 MiB │ │ Not Installed │ gcloud app Java Extensions │ app-engine-java │ 59.5 MiB │ │ Not Installed │ gcloud app PHP Extensions │ app-engine-php │ 21.9 MiB │ │ Not Installed │ gcloud app Python Extensions │ app-engine-python │ 6.1 MiB │ │ Not Installed │ gcloud app Python Extensions (Extra Libraries) │ app-engine-python-extras │ 27.1 MiB │ │ Not Installed │ kpt │ kpt │ 11.7 MiB │ │ Not Installed │ kubectl │ kubectl │ < 1 MiB │ │ Not Installed │ pkg │ pkg │ │ └──────────────────┴──────────────────────────────────────────────────────┴──────────────────────────┴──────────┘ To install or remove components at your current SDK version [312.0.0], run: $ gcloud components install COMPONENT_ID $ gcloud components remove COMPONENT_ID
To update your SDK installation to the latest version [315.0.0], run: $ gcloud components update
Modify profile to update your $PATH and enable shell command completion?
Do you want to continue (Y/n)? Y
The Google Cloud SDK installer will now prompt you to update an rc file to bring the Google Cloud CLIs into your environment.
Enter a path to an rc file to update, or leave blank to use [/Users/user/.bash_profile]: Backing up [/Users/user/.bash_profile] to [/Users/user/.bash_profile.backup]. [/Users/user/.bash_profile] has been updated.
==> Start a new shell for the changes to take effect.
For more information on how to get started, please visit: https://cloud.google.com/sdk/docs/quickstarts
最新版のSDKに更新
インストールスクリプト実行中の出力に
1 2
To update your SDK installation to the latest version [315.0.0], run: $ gcloud components update
Not setting default zone/region (this feature makes it easier to use [gcloud compute] by setting an appropriate default value for the --zone and --region flag). See https://cloud.google.com/compute/docs/gcloud-compute section on how to set default compute region and zone manually. If you would like [gcloud init] to be able to do this for you the next time you run it, make sure the Compute Engine API is enabled for your project on the https://console.developers.google.com/apis page.
Created a default .boto configuration file at [/Users/user/.boto]. See this file and [https://cloud.google.com/storage/docs/gsutil/commands/config] for more information about configuring Google Cloud Storage. Your Google Cloud SDK is configured and ready to use!
* Commands that require authentication will use shibagaki.motoaki.rt@mynavi.jp by default * Commands will reference project `mynavi-cm-adfair` by default Run `gcloud help config` to learn how to change individual settings
This gcloud configuration is called [default]. You can create additional configurations if you work with multiple accounts and/or projects. Run `gcloud topic configurations` to learn more.
Some things to try next:
* Run `gcloud --help` to see the Cloud Platform services you can interact with. And run `gcloud help COMMAND` to get help on any gcloud command. * Run `gcloud topic --help` to learn about advanced features of the SDK like arg files and output formatting
デフォルトのzone/regionの設定がされていないと言われますが、 Your Google Cloud SDK is configured and ready to use!とも書かれているので、こちらで使えるかどうか試してみます。
bqコマンドの確認
bqコマンドがあるかどうか確認します
1 2
$ which bq /Users/user/google-cloud-sdk/bin/bq
パスは通っているようです
1 2
$ bq version This is BigQuery CLI 2.0.62
大丈夫そうですね。
まとめ
Google BigQueryを読み進めていくためにbqコマンドを利用できるようにしました。読み進めていきながら実際にコマンドを実行したいと思います。
def authenticate authenticate_or_request_with_http_token do |token, _options| @user = User.find_by_token token head status: 401 if @user.blank? end end
def authenticate authenticate_or_request_with_http_basic do |_user, password| @user = User.find_by_token password head status: 401 if @user.blank? end end
# Disable serving static files from the `/public` folder by default since # Apache or NGINX already handles this. config.public_file_server.enabled = ENV['RAILS_SERVE_STATIC_FILES'].present?
