Pod Probing

To ensure our deployed services are running properly Kubernetes allows for probing of our deployed services in pods. In our deployment, we make use of the liveness and the readiness probes.

Enabling Probing for a service

To enable probing for a service, changes to the deployment file of the service in infrastructure and the application settings (if the service is a spring boot application) need to be performed.

Enabling Probing in Spring Boot applications

Spring offers a technology-agnostic library (Spring Actuator) to expose operational information of a running application, such as health, info, and metrics. These can be exposed via HTTP endpoints or JMX beans. To add this library to a spring service following dependency needs to be added to the ‘build.gradle’ file.

dependencies {
...
implementation 'org.springframework.boot:spring-boot-starter-actuator'
}

By default spring offers the following HTTP endpoints:

http://localhost:PORTNUMBER/actuator

http://localhost:PORTNUMBER/actuator/health

http://localhost:PORTNUMBER/actuator/info

Calling either of those endpoints will return all enabled information under those endpoints. Under the first URL, all available underlying endpoints & URLs can be retrieved. By default, spring disables most information endpoints. This includes the endpoints for liveness and readiness. To enable these endpoints following lines need to be added to the ‘application.properties’ file:

management.endpoint.health.probes.enabled=true
management.health.livenessstate.enabled=true
management.health.readinessState.enabled=true

To locally test if these states can be accessed, deploy the application in a docker container. Then send an HTTP request to the following URLs:

http://localhost:PORTNUMBER/actuator/health/liveness

for liveness

http://localhost:PORTNUMBER/actuator/health/readiness

for readiness

If things are set up correctly the HTTP response code should be a ‘204’ and return a JSON in the response body. For more information, we recommend reading the official spring documentation.

Configure Probing in the deployment file

To manage all our configurations for the deployment of all our services we terraform. Therefore all of our files in our deployment repository are terraform files. To allow probing for a service the ‘container’ spec needs to be extended by the following example code block:

liveness_probe {
    http_get {
        path = "/actuator/health/liveness"
        port = 2001       
        }     

    initial_delay_seconds = 30         
    period_seconds        = 9         
}                           

readiness_probe {           
    http_get {         
        path = "/actuator/health/readiness"       
        port = 2001       
    }         

    initial_delay_seconds = 30         
    period_seconds        = 9         
}           

Note the port number has to match that of the app port number (see ‘server.port’ in ‘application.properties’)

Probing for Dapr Sidecar

With each service, we also inject/deploy a DAPR sidecar to handle incoming and outgoing DAPR messages. These already contain pre-defined probing endpoints:

livenessProbe:
      httpGet:
        path: v1.0/healthz
        port: 3500
      initialDelaySeconds: 5
      periodSeconds: 10
      timeoutSeconds : 5
      failureThreshold : 3
readinessProbe:
      httpGet:
        path: v1.0/healthz
        port: 3500
      initialDelaySeconds: 5
      periodSeconds: 10
      timeoutSeconds : 5
      failureThreshold: 3

More information on the responses and their meaning can be found here.