Logging
Considering the different meetings and interviews with some technical leaders of the services, we were able to identify which were those fields that add value when it comes to monitoring the events generated by our services. With that knowledge, we created a standard based on OTeL Data Model, considering their base fields, but adapted to the specific needs of each team. For us it was very important in each meeting with the leaders, to identify which set of fields can provide a lot of information to generate business metrics that allow us to take action to positively impact the company and to generate alerts based on events reported by the services.
Structure
erDiagram
BaseAttributes ||--o{ Frontend : attributes
BaseAttributes ||--o{ Backend : attributes
BaseAttributes ||--o{ Mobile : attributes
BaseAttributes ||--o{ Job : attributes
BaseAttributes ||--o{ Consumer : attributes
BaseAttributes {
string time
string level
string service
string traceId
string eventType
string appVersion
string msg
string package
string method
string file
object attributes
}
Frontend {
string url
int statusCode
string httpMethod
string duration
string headers
string userId
string requestId
object customAttributes
}
Backend {
string url
int statusCode
string userAgent
string host
string httpMethod
string duration
string requestId
strung xAppId
object customAttributes
}
Mobile {
string device
string os
string screenView
string userId
string ip
object customAttributes
}
Job {
string schedule
string status
string duration
object customAttributes
}
Consumer {
string status
string duration
object customAttributes
}Examples
Attributes
Base attributes definition
Base attributes that all event types must have.
Note: - All properties of the
customAttributesattribute must be camelCased. - Any sensitive information added incustomAtributesmust be encrypted, obfuscated or masked.
time:Time when the event occurred measured by the origin clock, i.e. the time at the source. Format ISO 8601 YYYY-MM-DDTHH:MM:SS (2007-11-03T24:12:03) UTC-
level:This is the original string representation of the severity as it is known at the source (ref).- debug: A debugging event.
- info: An informational event. Indicates that an event happened.
- warning: A warning event. Not an error, but is likely more important than an informational event.
- error : An error event. Something went wrong, the owner team must be notified.
- emergency: Overall application or system failure that should be investigated immediately, DRI (Directly Responsible Individual) should be notified automatically. Alerts based on this event.
service:Name of the service that generates logs.appVersion:Service version.traceId:Set for logs that are part of event processing and have an assigned trace id related to other services.package:Library of the code that generates the log.method:Method or function of the code that generates the log.file:File name of the code that generates the log.msg:Message to be logged. It should be text.eventType:Type of the resource that generates the log:- backend: Development that ensures that all the logical part of an app (ims-backend, seller, property-catalogue, etc).
- frontend: Visible part of a website or application that users can interact with directly. Also the monolith (Sobreplanos, seller-frontend, ims-admin-frontend, etc).
- job: Scheduled or event-driven jobs, which are responsible for completing specific tasks.
- mobile: Mobile applications.
- consumer: Services that are designed to consume events such as kafka, sqs or eventbridge.
attributes:Object type field grouping attributes based of eventType (backend, frontend, mobile, job and consumer).
Attributes based on eventType definition
Backend
url:Request path (without domain).statusCode:An HTTP status codeuserAgent:User agent attribute used at request.host:Information of the server domain receiving the request.httpMethod:HTTP methods allow you to communicate to the server what you want to do with a request under a URL: POST, GET, PUT, DELETE, PATCH, and HEAD.duration:Time taken by the application to serve a particular event (in seconds).requestId:Identifies the HTTP request sent from the app to the service and enables the app to correlate requests and responses.xAppId:Identifier of the service from which the request originates.customAttributes:Field to add additional IMPORTANT information, for example; attributes to identify a user or custom metadata like SQL query.
Frontend
url:Request path (without domain).statusCode:An HTTP status code.ip:ip address sending the request.httpMethod:HTTP methods allow you to communicate to the server what you want to do with a request under a URL: POST, GET, PUT, DELETE, PATCH, and HEAD.duration:Time taken by the application to serve a particular event (in seconds).headers:Shares additional information between the clients and the server through the request and response header.userId:A unique customer identifier user visiting our website (email or anonymous).requestId:Identifies the HTTP request sent from the app to the service and enables the app to correlate requests and responses.customAttributes:Field to add additional IMPORTANT information, for example; attributes to identify a user or custom metadata like SQL query.
Mobile
device:Description of the device used by the customer.os:Operating system used by customers.screenView:Screen that generates the log report.userId:A unique customer identifier (final client) user visiting our website (email or anonymous).ip:Ip address sending the request.customAttributes:Field to add additional IMPORTANT information, for example; attributes to identify a user, advisor ID, or HTTP response.
Job
schedule:Execution schedule (cron expression or null).status:Status definition:success,skipped, orfailed.duration:Time taken by the application to process a particular event (in seconds).customAttributes:Field to add additional IMPORTANT information, for example; attributes to identify a user or custom metadata like SQL query.
Consumer
status:Status definition:processed,queued,failed.duration:Time taken by the application to process a particular event (in seconds).customAttributes:Field to add additional IMPORTANT information, for example; attributes to identify a user or custom metadata like SQL query.