The 5 Most Important Logs An Application Should Write
A breakdown of the types of data used to drive intelligent businesses
The usefulness of logs is often underestimated. Most businesses rely on logs solely for the purpose of troubleshooting operational and availability problems.
What many people fail to realize is that proactive logging also enables improved business decisions. Business intelligence is directly fueled by data — which opens the possibilities for delivering better business performance.
Some organizations depend entirely on infrastructure logging such as network firewall and server OS logs, and neglect the use of application logging. This can be a big mistake because applications have access to a wide range of information that is not available to sources external to the application. These information should be collected as the primary event data source comes from the application itself. Application logging can provide greater insight to a business than relying solely on infrastructure logging.
Why Application Logs?
Application logs are not used just for averting operational problems, but also to provide business intelligence through data mining. Having deep knowledge of customer behavior can reveal insights which are otherwise difficult to ascertain.
In some online platforms, application event logs are used to perform operations such as machine learning to improve user search results and deliver accurate recommendations, to fuel product ranking algorithms for a shopping site, and to show trending articles for a news portal. These activities ultimately lead to better customer satisfaction and improved sales.
Applications have access to a wide range of information that is not available to sources external to the application.
Here are the 5 most important things your application should write to logs:
- Requests
- Audit Trail
- Availability
- Threats
- Events
Requests
Requests logs as the name suggests, record each request or invocation of service within the application. This includes service or API access, system and process start up, application access, authentication and authorization.
These logs typically consist of the following information: date and time, requester identity such as User ID and IP address or referral URL, and the actual request data. In the case of a Web Application or API, the requested endpoint URL and context header and body is logged.
Request logs reveal information on your application’s usage details like who is using it, when it is used and what part of the application is used.
Audit Trail
Audit trails are a record of the changes that have been made to your data. Any change to data including creating new data, updating or deleting data, and in some cases exporting data, must be recorded.
The need for such records has much to do with security, company policies and compliance. Audit trail is a necessary requirement in some industries such as the financial sector.
Audit trail logs usually include the identity of the user or system that changed the data, the date and time, what data was changed. If it was a person making that change, the system which the person used to change the data is recorded as well.
Having these information can aid in investigating data compromise in your system and support recovery by providing the means to undo damaged data.
Availability
Availability logs consist of faults and exceptions that can impact the availability and stability of the system. These include exceeded capacity limit or resource usage, system errors or bugs, connectivity issues, and slow response times.
Some of these logs can be captured outside the application via infrastructure logging such as network or firewall logs and operating system events. However, certain events which originate from within the application itself like errors can only be logged by the application.
Side note: It is important to have error logs be detailed enough to support quick troubleshooting and recovery. In essence, an error log should reveal the steps required in order to reproduce the error. In many cases, the log would include information on the full stack trace or program function calls and the input parameters to the faulty process.
Threats
Threat logs track suspicious activities or attempts at undermining the security of an application. Some common threat logs include unauthorized access to restricted processes or data, invalid parameters or input, failed authentication, failed security verification such as invalid API key and other warnings triggered by the application’s security mechanisms.
For more information on API security — see my previous article on Web API security techniques.
Events
Events are any activity that a user can make on your application. Event activity is the least collected information but is arguably one of the most valuable data a business can possess.
As the types of event logs vary from business to business, there is no fixed definition on what to record exactly. What event to log is entirely limited to the imagination and requirements of the business.
Some examples are search queries, product views, likes and favorites, shares, make or cancel an order, viewed a page, how much time a person spent looking at an image, and whether they scrolled to the bottom of a long page, etc.
Even though you may not use most of these data today, they may prove to be a gold mine in the future. You never know what data you can collect now that will be useful to you tomorrow. As such, it is ideal to store events in their raw form— having the most detailed information of an event activity. An example is to store actual GPS coordinates for user location instead of the City or Country.
You never know what data you can collect today that will be useful to you tomorrow.
Conclusion
Proactive logging leads to improved business decisions and drives better business performance. Application logs contain wide array of information that is not only useful for averting operational problems, but to also elevate the overall security of an application and to fuel business intelligence through strategic data analysis.
This is the kind of information that led to the discovery of the famous “Wal-Mart Beer and Diapers” story. There is a debate whether the story is real or not, but it nonetheless illustrates the potential opportunities of possessing detailed recordings on your business activities.
