cloudflare/GCS-Logshare-Setup-Script

# GCS Logshare Setup Script

## Description
The main.sh script from your local machine creates a GCC VM that runs [Cloudflare's Logshare](https://github.com/cloudflare/logshare) tool on a cron schedule to push Enterprise Logs to BigQuery. Logs are written to GCS and then automatically pushed into BigQuery using the [GCS-To-Big-Query](https://github.com/cloudflare/GCS-To-Big-Query) code.

## Defaults
1. The cron job pushed runs every 1 minute and pulls logs from 10 minutes ago to 11 minutes ago. 
2. The ELS endpoint's fields are subject to change. At the time the VM is built, we cache a local version of the available fields. This is subject to change, but can be updated manually by modifying the fields.txt file.

## Requirements
   * [jq](https://github.com/stedolan/jq)
   * Python 2.7.9
   * curl

## Setup Instructions

1. Select or create a Cloud Platform project:
    * https://console.cloud.google.com/
2. Clone the GCS Automation Script on your local machine: 
    * ``` git clone https://github.com/cloudflare/GCS-Logshare-Setup-Script.git ```
3. Enable the Service Management API for your project:
    * https://console.developers.google.com/apis/api/servicemanagement.googleapis.com/overview
4. Configure and enable your Google Billing profile:
    * https://support.google.com/cloud/answer/6293499#enable-billing  
5. Enable the following Google APIs [here](https://console.cloud.google.com/apis/library): 
   * Google Cloud Storage,
   * Google BigQuery,
   * Cloud Function
6. Create a copy of default.config.json and rename to config.json
   * ``` mv config.default.json config.json ```
7. Modify config.json with your cloudflare account details
   * Cloudflare_api_key  - Cloudflare API Key
   * Cloudflare_api_email - Cloudflare user account email address
   * Zone_name - domain name; example: mydomain.com
   * Gcs_project_id - Google Cloud Project ID
8. Run the main orchestration script: 
   * ``` bash main.sh ```