Frequently Asked Questions

The NearlyFreeSpeech.NET FAQ (*)

Log Files (*)

What is the easiest way to make sense of my log files?

We have made AWStats, a graphical site statistics package, available for our members. Setup instructions for AWStats have been moved to our Wiki.

Do my site's log files count as space usage for billing purposes?

Yes they do. If you use log files, you should regularly rotate them in our member interface.

I've enabled log files, now where are they?

They are located in the /home/logs directory, as seen from ssh and SFTP, or the /logs directory as seen from FTP.

Most FTP clients start in the /public directory, so you would need to first change to the root directory (/) before you will see this.

What are log files and how are they used?

There are three primary kinds of log files available at NearlyFreeSpeech.NET:

All three are (if enabled) found in the /home/logs directory of your site.

Access logs are the most common type. Each line in the access log represents one access to your website. It contains a variety of information, like what IP address accessed your site, what they accessed, what web browser they used, etc. This is the type of log file that is most commonly used by analytics programs like AWStats and analog.

Error logs contain information about problems with your site. Although they contain lots of generic information, like records of attempts to access pages that don't exist, they are most important to web developers and people with complex software loaded on their site. Whenever software running on a site has a problem, it should (but doesn't always) say something about the problem in the error log. Finding that message can save you fruitless hours of wondering what the heck is wrong with your site.

If you ask for help troubleshooting a problem, on our forum for example, the first thing people will typically ask is whether you have your site's error log enabled and, assuming you do, what it says. To save time, it's often helpful to make sure it's enabled, say so in your question, and include any relevant log entries from around the time the problem occurred.

The third type, rewrite logs, are a bit more special-purpose. Rewrite logs track internal changes made to URLs by Apache. They're generally only useful when debugging these changes if you've set up your own rewriting rules, because the time it takes to write rewrite logs to our disk can slow your site to a crawl if you use a lot of them. If you don't know what a rewrite is or why you'd want to log them, you should definitely leave this one turned off.

My log file is huge and I can't remove it. What do I do?

As long as the system is writing to a log file, you cannot modify or erase it. Once you rotate the log file and the system starts on a new one, you can delete the old one if you wish. You can read your log files even if they have not been rotated, you just can't delete them while a server is still using them.

You can access and delete your old log files via FTP by looking in the "/logs" directory of your site. Via ssh or SFTP, this is the /home/logs directory. (But you'll still have to rotate them before you can do anything but look.)

If you've disabled logging, the option to rotate them will not be available to you. Enable your logs, rotate them, disable them, and then delete them.

How does log rotation work?

If you have log files enabled, you can ask our system to rotate them from the Site Information page for your site in our member interface.

When you ask to rotate your logs, our system waits for an opportune moment and then does it. Your access_log file, for instance, becomes access_log.old and a new access_log is created. It may not be right away, but it usually takes just a few minutes.

If there was already an access_log.old, that file gets a name based on the timestamp of the logfile, such as access_log.20160828. At this point, any compression you have selected will be applied. However, as of August 2016, compression is generally not necessary as the filesystem already incorporates transparent LZ4 compression. If you do not want your log files to be doubly compressed, you can disable log compression on the Site Information panel.

How do I enable my log files?

Here are the steps for enabling log files:

  1. Visit the Sites panel.
  2. Choose the site's "Short Name" to load the Site Information page.
  3. Choose "Enabled" for the access or error logs (or both).
  4. Choose "Save Changes."
  5. Wait a few minutes for the change to take effect.

If you're planning to measure any kind of statistics, enable the access log. For best results, enable it a week or so before measuring your statistics, as data from a longer period of time will produce more informative reports.

To find and debug problems with your site, enable the error log. If you submit a support request about an error message received from your site, we will often ask what your error log says about it. If you do not have your error log enabled, we may not be able to help you diagnose some problems.

We also provide a rewrite log which can be enabled in the same way, but may be set to varying levels of verbosity. Do not enable the rewrite log unless you are having problems with RewriteRule directives in your .htaccess file, and disable it as soon as you fix them. Using the rewrite log will dramatically slow down your site.

Why aren't log files enabled by default?

Because they take up space.

The return code 304 shows up in my log files all the time. What is it?

304 is the HTTP status response for "not modified." It is used in our log files to indicate that the logged request was actually served by some other part of our network. Usually only simple files, like static graphics and files, are eligible for this type of handling. Dynamic content, such as PHP and CGI scripts, will typically be served (and logged) in its entirety unless your application generates appropriate cache-eligible headers and 304 responses.

This code can also appear for other reasons, such as if someone accesses your site through a web cache.

How come the bytes in the log file do not add up to what you charged me for?

This is largely due to requests filled by other parts of the network, which may appear in your access log as 304 responses with a size of 0 bytes, or if served entirely from memory, may not appear in the access log at all.

The number you see for bytes with 200 response codes should match the size of the named file, unless it is a PHP or CGI script. However, every object served regardless of origin has some overhead (HTTP headers) that is also not reflected in the access log.

How many old log files will be kept during log rotation?

There is no arbitrary limit on how many old log files there are. You will need to remove old log files manually when you get tired of them.

What are each of the fields in the access log?

Our log files are based on the Apache-standard "combined" log file format, which is a superset of the "common" format. This format is good for web log analyzing software. The fields, in order, are:

clientip - username [time] "request" status bytes "referer" "useragent"
1.2.3.4 - - [22/Nov/2013:22:32:01 -0700] "GET /index.html HTTP/1.0" 200 0 "-" "Wget/1.9.1"

The meaning of each field is shown below:

clientip
The IP address of the machine that requested the page. If you get a "-" in this field, it may mean that your site performs HTTP accesses on itself, or that we had to test your site with some internal application.
dash
This field used to be used, but isn't anymore. Lots of log analyzers still expect it to be there, so it always appears as a dash (-).
username
If you use Basic HTTP Authentication, the authenticated username will appear here. If you don't, it will be a dash (-).
time
The time in the format DD/Mon/Year:HH:MM:SS tzone
request
The request method, requested URI, and protocol, like GET /index.html HTTP/1.0.
status
The return code. The most common codes are 200 (ok), 304 (served elsewhere), 404 (file not found), and 500 (internal server error - a script error or a problem with our server).
bytes
The number of raw bytes of content served. This number does not include HTTP headers, transfer encoding, or TCP/IP protocol overhead.
referer
The page that contained the link to the requested resource. You can use this to see who is linking to you, and weblog analyzers use it to map your site. "referer" is not a typo, although it is spelled incorrectly. It's actually in the published standard that way.
useragent
The web browser sends this to identify itself. Lots of log analyzers can graph this.

More information about Apache log file formats is available on the Apache site.

I rotated my log files, but the access_log.old file was not compressed. Why not?

The .old log files are never automatically compressed. If you rotate again, it will be renamed based on its last-modified date and only then will it be automatically compressed.

This is for two reasons. First, log entries come from many different parts of our network so there needs to be a "cooling off" period after rotating to make sure all parts of the network have switched to the new log. Otherwise, stuff might still be logged while compression is happening, and it would get lost.

Also, many people like to run analysis against a static log file. That way, if you change an analyzer setting and run it again, or if you want to do more than one thing to the log, you're working with a consistent set of logs. After log files are compressed, they're a lot harder to analyze, so keeping the .old file around often makes log analyzing much easier.

You're free to remove the .old log file yourself if you like, although we recommend waiting until you see log entries appearing in the new log file. If you rename or compress it yourself, it will not be auto-rotated in the future.

What is the format of the error log?

The error log is a "stream of consciousness" output of all the various programs you run on your site. As such, it does not have a specific format, other than that the most recent errors are at the bottom.

What happens if I change log file compression techniques?

Nothing extraordinary. The change will only apply to newly compressed log files. Your old ones will be left as-is. If you want to update them to your newly chosen format, it is possible to do this from the ssh command line.