Raw logs: export CDN resource logs to a storage

Raw Logs is an option that enables an automatic export of CDN resource logs to your storage. Logs contain information about user requests sent to cache servers and pre-cache servers (if the Origin Shielding is enabled).

Info

This is a paid feature. To activate Raw Logs, contact the Gcore support team. After activation, enable Raw Logs in the Gcore Customer Portal and configure export to S3, FTP, or SFTP storage.

Raw logs settings

In this section, you can find general information about log settings, overview of log statuses, and details on how Raw Logs are exported and calculated.

Add logs from origin shielding

If you are using the Origin Shielding feature, you’ll see the Add logs from origin shielding checkbox when configuring logs. We recommend that you select this option as it ensures that the logs report will include both requests to cache services and requests to the pre-cache server. Thus, you’ll receive more detailed information on resource usage.

When setting up Raw Logs, select the checkbox to enable Add logs from origin shielding.

Tip

If you don’t see the Origin Shielding option on the Raw Logs page, this feature is not activated for your account. For details on how to activate Origin Shielding, check our dedicated guide.

Raw logs statuses

You can check the status of the Raw Logs in the Gcore Customer Portal:

Pending : A status for the time interval between the connection to storage and the first log export.
OK : Logs are exported from at least one CDN server.
Failed : An error occurred while connecting to a storage, or the service failed to export logs within 24 hours.
Pause : The Raw Logs feature is paused.

Export time intervals

Logs are exported at the end of each hour. If you activate Raw Logs at 00:30, the first logs will be exported between 00:45 and 01:00 and the next ones—between 01:45 and 02:00.

If CDN servers are not requested and you didn’t select the Do not send empty logs checkbox when configuring Raw Logs, an empty log file (± 20 bytes) will be sent to your storage.

Log traffic calculation

Logs can generate various types of analytics, such as delivered traffic. To understand what the totals mean, we recommend that you familiarize yourself with the formulas for calculating logs. For details about CDN reports, check out the View statistics on CDN resources guide.

The formula for calculating traffic depends on whether you’ve selected the Add logs from origin shielding option for Raw Logs.

If Origin Shielding is disabled

If this feature is disabled, the formula will look like this:

total_bytes = upstream_bytes + sent_bytes

Where:

upstream_bytes are equal to the $upstream_response_length log field and refer to the response length from an origin in bytes
sent_bytes are equal to the $bytes_sent log field and refer to the number of bytes sent to a user from the edge (cache) servers

For example, if $upstream_response_length is 10485760 (bytes) and $bytes_sent is 1514848 (bytes), the final value in the Raw logs report will be 12,000,608 (bytes).

If Origin Shielding is enabled

If this feature is enabled, the formula will look like this:

total_bytes = upstream_bytes + sent_bytes + shield_bytes

Where:

upstream_bytes are equal to the $upstream_response_length log field and refer to the response length from an origin in bytes
sent_bytes are equal to the $bytes_sent log field and refer to the number of bytes sent to a user from the edge (cache) servers
shield_bytes are equal to the $bytes_sent log field and refer to the number of bytes sent to a user from the pre-cache server

Info

The final value of log data may differ slightly from the billing statistics as there may be cases where not all logs are received, such as: * You are using the Origin Shielding feature but didn’t select the Add logs from origin shieldin g option. * You have a rate limit set on your storage side, and when CDN started generating logs, some logs weren’t downloaded because of the rate limitation.

Log path example

s3://log-bucket-name/2019/08/20/15/nodename_primarycname.domain.ru_access.log.gz

Log example

"0.0.0.0" "-" "-" "[26/Apr/2019:09:47:40 +0000]" "GET /ContentCommon/images/image.png HTTP/1.1"  
"200" "1514283" "https://example.com/videos/10" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1)  
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 YaBrowser/16.10.0.2309 Safari/537.36"  
"1514848" "[dh-up-gc18]" "https" "origin.cdn.com" "1.500" "0.445" "157" "bytes=0-1901653" "[dh]"  
"MISS" "10485760" "0.0.0.0:80" "2510" "7399" "-" "-" "KZ" "-" "shield_no" "0.0.0.0" "80" "206" "-" "0.000"  
"0.200" "0.0.0.0" "asnumber" "106980391" "1" "00-d5fe1dc9035165ce36952daf29686b6c-14330be33197dd1a-01" "-" "-"  
"ECDHE-RSA-AES256-GCM-SHA384" "28a4184139cb43cdc79006cf2d1a4ac93bdc****" "r"  
"application/json" "21" "PL" "45" "10" "100700"

Log format

"$remote_addr" "-" "$remote_user" "[$time_local]" "$request" "$status"  
"$body_bytes_sent" "$http_referer" "$http_user_agent" "$bytes_sent"  
"$edgename" "$scheme" "$host" "$request_time"  
"$upstream_response_time" "$request_length" "$http_range" "[$responding_node]"  
"$upstream_cache_status" "$upstream_response_length" "$upstream_addr"  
"$gcdn_api_client_id" "$gcdn_api_resource_id" "$uid_got" "$uid_set"  
"$geoip_country_code" "$geoip_city" "$shield_type" "$server_addr" "$server_port"  
"$upstream_status" "-" "$upstream_connect_time" "$upstream_header_time"  
"$shard_addr" "$geoip2_data_asnumber" "$connection" "$connection_requests"  
"$http_traceparent" "$http_x_forwarded_proto" "$gcdn_internal_status_code" "$ssl_cipher"  
"$ssl_session_id" "$ssl_session_reused" "$sent_http_content_type" "$tcpinfo_rtt" 
"$server_country_code" "$gcdn_tcpinfo_snd_cwnd" "$gcdn_tcpinfo_total_retrans" "$gcdn_rule_id"

It’s OK if you find a field that’s not listed in the example. We occasionally add new fields to the end of the line. If some fields are added to logs, you’ll receive an email about the update.

Log fields

The following table contains a complete list of available log fields. Fields formatted in italics relate to our internal CDN system, so you can ignore them.

You can check other fields—they can be helpful for traffic analysis or statistics.

View log fields

Field	Log value example	Description
`$remote_addr`	`0.0.0.0`	User’s IP address
`$remote_user` (internal system variable)	`-`	Username used in Basic authentication
`[$time_local]`	`[26/Apr/2019:09:47:40 +0000]`	Local time in Common Log Format
`$request`	`GET /ContentCommon/images/image.png HTTP/1.1`	HTTP method, requested file path, and HTTP version
`$status`	`200`	Response status code from a CDN server
`$body_bytes_sent`	`1514283`	Number of bytes sent to a user, excluding the response header size
`$http_referer`	`https://example.com/videos/10`	Referrer – a URL requested by a user
`$http_user_agent`	`Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 YaBrowser/16.10.0.2309 Safari/537.36`	User agent that was used to send a request (browser or other application)
`$bytes_sent`	`1514848`	Number of bytes sent to a user
`$edgename`	`[dh-up-gc18]`	CDN server that forwarded a requested file
`$scheme`	`https`	Protocol (HTTP or HTTPS) of a request
`$host`	`cdn.example.com`	Requested hostname of a CDN resource
`$request_time`	`1.500`	Request processing time in seconds (accurate to milliseconds); time elapsed between the first bytes of a request being processed and logging after the last bytes were sent to a user
`$upstream_response_time`	`0.445`	Number of seconds (accurate to milliseconds) it took to receive a response from an origin. In case of multiple responses, commas and colons are used
`$request_length`	`157`	Request length (including request line, header, and request body)
`$http_range`	`bytes=0-1901653`	File fragment size in a Range request
`[$responding_node]`	`dh`	Responding data center
`$upstream_cache_status`	`MISS`	Status of a requested file in CDN cache: HIT: response served from the CDN cache. STALE: outdated response that failed to update (origin not responding or responding incorrectly). UPDATING: outdated response still updating from a previous request. REVALIDATED: response matching one on an origin (based on the `proxy_cache_revalidate` directive). EXPIRED: response has expired in cache but still matches one on an origin; a request was sent to an origin for re-caching. MISS: response served directly from an origin rather than from a cache. BYPASS: response for the first file request after clearing the cache. Appears when the file is requested by each CDN server (the first request from that server results in BYPASS, subsequent requests result in HIT).
`$upstream_response_length`	`10485760`	Response length from an origin in bytes. In case of multiple responses, commas and colons are used
`$upstream_addr`	`0.0.0.0:80`	Origin’s IP address and port
`$gcdn_api_client_id` (internal system variable)	`123`	Your ID in our system
`$gcdn_api_resource_id` (internal system variable)	`01`	Your CDN-resource ID in our system
`$uid_got` (internal system variable)	`-`	Cookie name and received user ID
`$uid_set` (internal system variable)	`-`	Cookie name and provided user ID
`$geoip_country_code`	`KZ`	User’s country code according to the ISO 3166 standard (Alpha-2 code)
`$geoip_city`	`-`	User’s city code
`$shield_type` (internal system variable)	`shield_no`	Indicates whether Origin Shielding is enabled: shield_old – enabled shield_no – disabled
`$server_addr` (internal system variable)	`0.0.0.0`	IP address of an Anycast zone or CDN server
`$server_port` (internal system variable)	`80`	Requested port
`$upstream_status`	`206`	Origin response code
`$upstream_connect_time`	`0.000`	Number of seconds (accurate to milliseconds) it took to access an origin server
`$upstream_header_time`	`0.200`	Number of seconds (accurate to milliseconds) it took to receive a response header from an origin server
`$shard_addr` (internal system variable)	`0.0.0.0`	IP address of a CDN server that was first to accept a request if the Cache Sharding feature is enabled
`$geoip2_data_asnumber`	`asnumber`	Number of an autonomous system that sent a request
`$connection` (internal system variable)	`2897494295`	Connection serial number
`$connection_requests` (internal system variable)	`1`	Current number of requests made through a connection
`$http_traceparent`	`00-d5fe1dc9035165ce36952daf29686b6c-14330be33197dd1a-01`	Unique request identifier. More info in the Traceparent guide
`$http_x_forwarded_proto`	`-`	Initial protocol of an incoming request (HTTP or HTTPS)
`$gcdn_internal_status_code` (internal system variables)	`-`	Initial status code. Possible values are `-` or `100700`
`$ssl_cipher` (internal system variable)	`ECDHE-RSA-AES256-GCM-SHA384`	Cipher name used for an established SSL connection
`$ssl_session_id` (internal system variable)	`28a4184139cb43cdc79006cf2d1a4ac93bdc****`	Session ID of an established SSL connection
`$ssl_session_reused` (internal system variable)	`r`	Shows whether a session was reused (`r`) or not (`.`)
`$sent_http_content_type`	`application/json`	Value of the `Content-Type` HTTP header, indicating the MIME type of a transmitted file
`$tcpinfo_rtt`	`21`	Average time (latency) it takes to transfer a packet to/from a server. The unit of time is microseconds.
`$server_country_code`	`PL`	Server’s country code according to the ISO 3166 standard (Alpha-2 code).
`$gcdn_tcpinfo_snd_cwnd`	`45`	Size of the TCP Congestion window, i.e., the maximum number of TCP segments that the connection can send before an acknowledgment is required
`$tcpi_total_retrans`	`10`	Total number of retransmitted packets over the life of the connection
`$gcdn_rule_id`	`100700`	Initial rule ID (beta). Possible values are `-` or `100700`

Export logs to S3 storage

1. In the Gcore Customer Portal, navigate to CDN > Logs.

2. Enable the Receive Raw Logs toggle to open log settings.

3. Keep the Do not send empty logs option selected if you don’t want to receive empty logs. Otherwise, uncheck it.

4. If you use our Origin Shielding feature, you’ll see a checkbox Add logs from origin shielding. Select the checkbox to receive logs from both edge servers and pre-cache shielding servers.

5. Select to receive logs using S3.

6. Choose Amazon as the storage type.

7. Specify your access key ID. In your Amazon personal account, it’s called “AWS access key ID”. For details on how to find your key ID, check the official Amazon guide. An access key ID and secret access key are required to configure log export to your storage.

8. Specify your secret access key. In your Amazon account, it’s called “AWS secret access key”. For details on how to find your access key, check the official Amazon guide.

9. (Optional) Enter your AWS region—the location of a server where your storage is hosted. For most storage types, the region is determined automatically, so you might leave the field empty. However, we recommend specifying the region to ensure that your logs are exported successfully.

10. Choose how to organize your storage: put logs of all CDN resources into one bucket or use separate buckets for each CDN resource.

11. Specify buckets for log export. Make sure to indicate an existing bucket. Otherwise, your logs won’t be exported.
Enter a folder name if you want to export logs to a specific folder within a bucket.

12. Click Save changes.

You’ve successfully configured Raw Logs export to Amazon S3.

1. In the Gcore Customer Portal, navigate to CDN > Logs.

2. Enable the Receive Raw Logs toggle to open log settings.

3. Keep the Do not send empty logs option selected if you don’t want to receive empty logs. Otherwise, uncheck it.

4. If you use our Origin Shielding feature, you’ll see a checkbox Add logs from origin shielding. Select the checkbox to receive logs from both edge servers and pre-cache shielding servers.

5. Select to receive logs using S3.

6. Choose Amazon as the storage type.

8. Specify your secret access key. In your Amazon account, it’s called “AWS secret access key”. For details on how to find your access key, check the official Amazon guide.

10. Choose how to organize your storage: put logs of all CDN resources into one bucket or use separate buckets for each CDN resource.

12. Click Save changes.

You’ve successfully configured Raw Logs export to Amazon S3.

1. In the Gcore Customer Portal, navigate to CDN > Logs.

2. Enable the Receive Raw Logs toggle to open log settings.

3. Keep the Do not send empty logs option selected if you don’t want to receive empty logs. Otherwise, uncheck it.

4. If you use our Origin Shielding feature, you’ll see a checkbox Add logs from origin shielding. Select the checkbox to receive logs from both edge servers and pre-cache shielding servers.

5. Select to receive logs using S3.

6. Select Other as the storage type.

7. Specify a hostname—a name that’s assigned to a storage server within a network and is used instead of an IP address. If you use Gcore Object Storage, you can find its access key ID in your personal account in the Hostname field.

8. Specify your access key ID. Along with a secret access key, it’s required to configure log export to your storage. If you use Gcore Object Storage, you can find its access key ID in your personal account in the Access key field.

9. Specify your secret access key. If you use Gcore Object Storage, you can find its secret access key in your personal account in the Secret key field.

10. Specify a bucket hostname—a bucket ID that’s used by your Object storage system in the {bucket_name}.{hostname} format. It’s required to ensure that logs are exported to the correct bucket within a storage. A bucket hostname of the Gcore storage looks as follows: {bucket name}.{hostname from step 3}. For example: examplename.s-ed1.cloud.gcore.lu.

11. (Optional) Specify a regio—location ID of a server where your storage is hosted. For some storage types, the region is determined automatically, so you can leave the field empty.

Info

If you use Gcore Object Storage, you must specify the location ID. You can find it in the Details of the storage. Your location ID is a part of your hostname to the first dot.

12. Choose how to organize storage: put logs of all CDN resources into one bucket or use separate buckets for each CDN resource.

13. Specify buckets for log export. Make sure to indicate an existing bucket. Otherwise, your logs won’t be exported. If you want to export logs to a specific folder within a bucket, specify the folder name.

14. Click Save changes.

You’ve successfully configured Raw Logs export to S3 storage.

Export logs to FTP or SFTP storage

1. In the Gcore Customer Portal, navigate to CDN > Logs.

2. Enable the Receive Raw Logs toggle to open log settings.

3. Keep the Do not send empty logs option selected if you don’t want to receive empty logs. Otherwise, uncheck it.

4. If you use our Origin Shielding feature, you’ll see a checkbox Add logs from origin shielding. Select the checkbox to receive logs from both edge servers and pre-cache shielding servers.

5. Select to receive logs via SFTP.

6. Specify a hostname—a name that’s assigned to a storage server within a network and is used instead of an IP address. If you use Gcore SFTP Storage, you can find its hostname in the Details of the storage in the Hostname field. It looks as follows: ams.origin.gcdn.co.

Warning

You need to specify an FTP or SFTP storage port after the hostname. For example: ams.origin.gcore.co:2200. Otherwise, you’ll get the “Wrong connection settings error” and won’t be able to connect to the storage.

7. Specify a username you use to log in to the storage. If you use Gcore SFTP storage, you can find the username in the Details of your storage in the Storage/User name field.

8. Enter the password you use to authenticate to the storage.

9. Specify a folder for export. If you use Gcore SFTP storage, specify the root (home) folder where other folders originate from. You can find its name in the Details of your storage at the end of the Upload path field.

If you use an SFTP storage from another provider, clarify whether a root folder that includes other folders is created by default. If not, leave the field empty. If yes, specify a folder name.

Info

If you’re using the Gcore SFTP storage, you won’t be able to write to the home folder. It only receives logs and can’t be manually modified.

10. Choose how to organize the storage: put logs of all CDN resources into one folder or use separate folders for each CDN resource. Then, specify a folder name. If you specify a non-existent folder, logs will be exported to a root folder.

11. (Optional) Specify the folder name where logs will be stored. If you specify a non-existent folder, logs will be exported to a root folder. For Gcore SFTP storage, you can only specify the default htdocs folder.

12. Click Save changes.

You’ve successfully configured Raw Logs export to FTP or SFTP storage.

Deactivate raw logs

To stop Raw Logs delivery, disable the Receive Raw Logs toggle.

If you disable the logs, your settings will be saved. You can always activate the same configuration by enabling the Receive Raw Logs toggle again.

If you made any changes to the configuration, click Discard changes to cancel any changes made during the editing process.

Prefetch Logs uploader (near real-time)

On this page

Raw logs settings
Add logs from origin shielding
Raw logs statuses
Export time intervals
Log traffic calculation
Log path example
Log example
Log format
Log fields
Export logs to S3 storage
Export logs to FTP or SFTP storage
Deactivate raw logs

Account settings

CDN

FastEdge

Edge Cloud

Edge AI

Managed DNS

Hosting

Storage

Video streaming

DDoS protection

WAAP

Raw logs: export CDN resource logs to a storage

Raw logs settings

Add logs from origin shielding

Raw logs statuses

Export time intervals

Log traffic calculation

Log path example

Log example

Log format

Log fields

Export logs to S3 storage

Export logs to FTP or SFTP storage

Deactivate raw logs

Account settings

CDN

FastEdge

Edge Cloud

Edge AI

Managed DNS

Hosting

Storage

Video streaming

DDoS protection

WAAP

​Raw logs settings

​Add logs from origin shielding

​Raw logs statuses

​Export time intervals

​Log traffic calculation

​Log path example

​Log example

​Log format

​Log fields

​Export logs to S3 storage

​Export logs to FTP or SFTP storage

​Deactivate raw logs

Raw logs settings

Add logs from origin shielding

Raw logs statuses

Export time intervals

Log traffic calculation

Log path example

Log example

Log format

Log fields

Export logs to S3 storage

Export logs to FTP or SFTP storage

Deactivate raw logs