Enable Verbose Logging

This guide explains how to enable detailed logging in the collector-manager for diagnosis. In short, adding --zap-devel to the spec.containers.args field in the collector-manager deployment enabled debug logs. This enhances log verbosity, offering valuable insights for troubleshooting complex problems.

Please note that running the collector-manager with --zap-devel or in debug mode is not recommended for regular operation. This mode is intended solely for issue identification and resolution purposes.

  1. Identify the collector-manager deployment:

    kubectl get deployment -n plerion-system

    Example output:

    NAME                                 READY   UP-TO-DATE   AVAILABLE   AGE
plerion-collector-manager   2/2     2            2           10m

  2. Edit the deployment configuration:

    kubectl edit deployment plerion-collector-manager -n plerion-system

  3. In your editor, add the spec.containers.args section for the collector-manager container and add the --zap-devel option like this:

    spec:
  containers:
    - name: collector-manager
      args:
        - --zap-devel

  4. Save and exit the your editor.

With the --zap-devel option enabled, you will receive more detailed logs, which can be especially helpful for pinpointing and resolving intricate issues within the collector-manager.

Example log of a pod with --zap-devel

2023-11-21T03:45:49Z	INFO	maxprocs: Updating GOMAXPROCS=2: determined from CPU quota
2023-11-21T03:45:49Z	INFO	setup	initializing controller
2023-11-21T03:45:49Z	INFO	setup	fetching tenant config
2023-11-21T03:45:49Z	DEBUG	k8scbclient	performing request	{"method": "POST", "url": "https://au.app.plerion.com/api****/extern****/kubern****/tenant****/7934cf****b34-4bda-adde-173a1fa30d9c/integr****ns/d09059****5cf-4e7f-b8da-18e2298a15a5/config****"}
2023-11-21T03:45:50Z	DEBUG	k8scbclient	request completed	{"request": "POST", "url": "https://au.app.plerion.com/api****/extern****/kubern****/tenant****/7934cf****b34-4bda-adde-173a1fa30d9c/integr****ns/d09059****5cf-4e7f-b8da-18e2298a15a5/config****", "status": 200}
2023-11-21T03:45:50Z	INFO	controller-runtime.metrics	Metrics server is starting to listen	{"addr": ":8080"}
2023-11-21T03:45:50Z	INFO	setup	starting manager
2023-11-21T03:45:50Z	INFO	setup	Starting server	{"kind": "health probe", "addr": "[::]:8081"}
2023-11-21T03:45:50Z	INFO	setup	starting server	{"path": "/metrics", "kind": "metrics", "addr": "[::]:8080"}
2023-11-21T03:45:50Z	INFO	attempting to acquire leader lease plerion-system/ecaf1259.collector.plerion.com...
2023-11-21T03:45:50.297049261Z
2023-11-21T03:46:06Z	INFO	successfully acquired lease plerion-system/ecaf1259.collector.plerion.com
2023-11-21T03:46:06Z	INFO	setup	Starting EventSource	{"controller": "resourcecollector", "source": "<redacted>"}
2023-11-21T03:46:06Z	INFO	setup	Starting Controller	{"controller": "resourcecollector"}
2023-11-21T03:46:06Z	INFO	setup	Starting workers	{"controller": "resourcecollector", "worker count": 1}
2023-11-21T03:46:06Z	DEBUG	setup.events	plerion-collector-manager-5898cbc55c-6cj9z_97def357-9cc4-45f3-9ffb-e42a717c1ebf became leader	{"type": "Normal", "object": {"kind":"Lease","namespace":"plerion-system","name":"ecaf1259.collector.plerion.com","uid":"2148bf34-4de6-4783-974a-c2be8b143f66","apiVersion":"coordination.k8s.io/v1","resourceVersion":"27347"}, "reason": "LeaderElection"}
  1. Log Analysis:

  • GOMAXPROCS Update: The system updates the GOMAXPROCS value based on CPU quota.

  • Controller Initialization: The controller initializes, fetching configuration information related to a specific tenant.

  • HTTP Request Debugging Information: Detailed debugging information about an HTTP request is logged, including the method, URL, and completion status. Any response code other than 200 or 204 should be a cause for concern.

  • Metrics Server Setup: The metrics server starts listening on port 8080.

  • Manager Start: The manager is initiated, starting servers for health probes and metrics.

  • Leader Lease Acquisition: An attempt is made to acquire a leader lease, indicating leadership status.

  • Leader Election Debugging: Debugging information is logged when the controller-manager becomes a leader.

  • Worker Initialization: After the system successfully acquires the leader lease, the Controller components start.

TroubleshootingEmail