cloudflare/GCS-Logshare-Setup-Script
Publicmirrored fromhttps://github.com/cloudflare/GCS-Logshare-Setup-Script
README.md
37lines · modecode
Updated logshare-cli cmd line option to match spec. Updated a few things in the readme such as removing /received because it's the default endpoint now. Updated main.sh script output73222c3
8 years ago
| 1 | # GCS Logshare Setup Script |
| 2 | |
| 3 | ## Description |
| 4 | The main.sh script from your local machine creates a GCC VM that runs [Cloudflare's Logshare](https://github.com/cloudflare/logshare) tool on a cron schedule to push Enterprise Logs to BigQuery. Logs are written to GCS and then automatically pushed into BigQuery using the [GCS-To-Big-Query](https://github.com/cloudflare/GCS-To-Big-Query) code. |
| 5 | |
| 6 | ## Defaults |
| 7 | 1. The cron job pushed runs every 1 minute and pulls logs from 10 minutes ago to 11 minutes ago. |
| 8 | 2. The ELS endpoint's fields are subject to change. At the time the VM is built, we cache a local version of the available fields. This is subject to change, but can be updated manually by modifying the fields.txt file. |
| 9 | |
| 10 | ## Requirements |
| 11 | * [jq](https://github.com/stedolan/jq) |
| 12 | * Python 2.7.9 |
| 13 | * curl |
| 14 | |
| 15 | ## Setup Instructions |
| 16 | |
| 17 | 1. Select or create a Cloud Platform project: |
| 18 | * https://console.cloud.google.com/ |
| 19 | 2. Clone the GCS Automation Script on your local machine: |
| 20 | * ``` git clone https://github.com/cloudflare/GCS-Logshare-Setup-Script.git ``` |
| 21 | 3. Enable the Service Management API for your project: |
| 22 | * https://console.developers.google.com/apis/api/servicemanagement.googleapis.com/overview |
| 23 | 4. Configure and enable your Google Billing profile: |
| 24 | * https://support.google.com/cloud/answer/6293499#enable-billing |
| 25 | 5. Enable the following Google APIs [here](https://console.cloud.google.com/apis/library): |
| 26 | * Google Cloud Storage, |
| 27 | * Google BigQuery, |
| 28 | * Cloud Function |
| 29 | 6. Create a copy of default.config.json and rename to config.json |
| 30 | * ``` mv config.default.json config.json ``` |
| 31 | 7. Modify config.json with your cloudflare account details |
| 32 | * Cloudflare_api_key - Cloudflare API Key |
| 33 | * Cloudflare_api_email - Cloudflare user account email address |
| 34 | * Zone_name - domain name; example: mydomain.com |
| 35 | * Gcs_project_id - Google Cloud Project ID |
| 36 | 8. Run the main orchestration script: |
| 37 | * ``` bash main.sh ``` |