EnvoyPatchPolicy On Draining Listeners: A Configuration Guide

In this comprehensive guide, we will explore the application of EnvoyPatchPolicy to listeners specifically within the draining_state in Envoy proxy. This is crucial for ensuring seamless application functionality during Envoy worker draining mode. We'll delve into the problem, the configurations, and the solution, providing a detailed understanding for developers and system administrators.

Understanding EnvoyPatchPolicy

Let's start by understanding what EnvoyPatchPolicy is and why it matters. EnvoyPatchPolicy is a powerful feature within the Envoy proxy that allows you to dynamically modify Envoy's configuration without restarting the proxy. This is particularly useful for applying changes on the fly, such as security policies, routing rules, or other configurations, without causing disruption to the service. Envoy is a high-performance proxy designed for modern application architectures, and its ability to be dynamically configured is one of its key strengths. EnvoyPatchPolicy leverages this capability by providing a Kubernetes-native way to express configuration changes, making it easier to manage Envoy in a Kubernetes environment. One of the primary advantages of using EnvoyPatchPolicy is its flexibility. It supports various types of patches, including JSON patches, which allow you to make granular changes to the Envoy configuration. This means you can target specific parts of the configuration and modify them without affecting other parts. This fine-grained control is essential for complex deployments where different parts of the system may require different configurations. Furthermore, EnvoyPatchPolicy integrates seamlessly with Kubernetes, allowing you to define policies as Kubernetes resources. This integration simplifies the management and deployment of Envoy configurations, as you can use standard Kubernetes tools and practices to manage your Envoy policies. The policy definitions are stored in Kubernetes, providing a central source of truth for your Envoy configurations. In practice, EnvoyPatchPolicy works by intercepting the configuration updates that Envoy receives. When a new configuration is applied, EnvoyPatchPolicy checks if there are any defined policies that match the target resource. If there are, it applies the patches specified in the policy to the configuration before it is applied to Envoy. This process ensures that the desired changes are made to the configuration in a controlled and predictable manner. The status of EnvoyPatchPolicy is continuously monitored to ensure that the patches are successfully applied. The Programmed status, as seen in the example provided, indicates that the patches have been successfully applied. This feedback mechanism is crucial for maintaining the integrity of the configuration and quickly identifying any issues that may arise. In summary, EnvoyPatchPolicy is a powerful tool for dynamically managing Envoy configurations in a Kubernetes environment. Its flexibility, integration with Kubernetes, and continuous monitoring capabilities make it an essential component for modern application deployments. Understanding how to effectively use EnvoyPatchPolicy is key to leveraging the full potential of Envoy proxy.

The Draining State Challenge

The draining state in Envoy is a critical phase during which an Envoy worker is being removed or restarted. During this phase, Envoy stops accepting new connections but continues to process existing ones. This ensures that no requests are abruptly terminated, providing a graceful transition. However, the challenge arises when configurations applied via EnvoyPatchPolicy are not consistently applied to listeners in the draining_state. This inconsistency can lead to issues, especially when specific configurations are essential for maintaining application functionality during the draining process. For instance, if certain routing rules or security policies are not applied in the draining_state, existing connections may not be handled correctly, leading to errors or service disruptions. The draining_state is particularly important in a dynamic environment like Kubernetes, where pods may be frequently scaled up or down. When a pod is being terminated, Envoy enters the draining_state to ensure that all ongoing requests are completed before the pod is fully removed. If the Envoy configuration is not correctly applied during this state, users may experience intermittent issues, such as failed requests or unexpected behavior. This can be particularly problematic for applications that require long-lived connections, such as WebSocket or gRPC services, where maintaining connection integrity during the draining process is crucial. The issue of inconsistent configuration in the draining_state often stems from the way Envoy manages its listeners. Envoy maintains separate listeners for active and draining states, and configurations applied to the active listener may not automatically propagate to the draining listener. This separation is designed to ensure that new configurations do not interfere with existing connections that are being drained. However, it also means that any configuration changes, including those applied via EnvoyPatchPolicy, need to be explicitly applied to both the active and draining listeners to ensure consistency. In the provided scenario, the user observed that the EnvoyPatchPolicy was successfully applied to the socket in the active_state but not in the draining_state. This discrepancy highlights the importance of understanding how Envoy manages its listeners and the need for a mechanism to ensure that policies are consistently applied across all states. To address this challenge, it is necessary to either modify the Envoy configuration process to automatically propagate policies to the draining listener or to explicitly apply the policies to both listeners. The solution may involve changes to the Envoy configuration files, the EnvoyPatchPolicy definitions, or the deployment scripts used to manage the Envoy proxy. Ultimately, the goal is to ensure that the application functions correctly even when Envoy workers are in the draining_state, providing a seamless experience for users.

The Reported Issue: EPP Not Applied in Draining State

A user reported a specific issue where an EnvoyPatchPolicy (EPP) was not being applied to listeners in the draining_state. The user's configuration involved setting a path_with_escaped_slashes_action within an Envoy listener using a JSON patch. The EPP status indicated that the policy was accepted and patches were successfully applied. However, when the Envoy configuration was dumped, the policy was only applied to the socket in the active_state and not in the draining_state. This discrepancy poses a significant problem because the application's functionality is compromised when Envoy workers are placed in draining mode. Specifically, the application relies on the path_with_escaped_slashes_action setting to function correctly, and its absence in the draining_state listener leads to misbehavior during worker transitions. To better understand the issue, let's break down the key components involved. The EnvoyPatchPolicy resource is defined using the gateway.envoyproxy.io/v1alpha1 API version and kind. This resource specifies the type of patch (JSONPatch), the target reference (in this case, a Gateway resource), and the actual JSON patches to be applied. The target reference includes the group, kind, name, and namespace of the resource to be patched. In this scenario, the target is a Gateway resource named couchdb in the esrp namespace. The JSON patches array contains individual patch definitions, each with a name, type, operation, path, and value. The type specifies the Envoy configuration object being patched (in this case, envoy.config.listener.v3.Listener), the operation specifies the action to be taken (replace), the path specifies the location within the object to be modified, and the value specifies the new value to be set. The user's configuration aims to replace the path_with_escaped_slashes_action within the listener's filters with the value KEEP_UNCHANGED. This setting is crucial for handling paths with escaped slashes correctly. The EPP status provides valuable information about the policy's application. The ancestors section lists the resources to which the policy applies, and the conditions section provides details about the policy's status. In this case, the status indicates that the policy has been accepted and the patches have been successfully applied. However, as the user observed, this status only reflects the application of the policy to the active_state listener. The fact that the policy is not being applied to the draining_state listener suggests that there may be a gap in how EnvoyPatchPolicy handles listeners in different states. It is possible that the policy application logic only targets the active listener, or that there is a mechanism that prevents the policy from being applied to the draining listener. Addressing this issue requires a deeper understanding of how Envoy manages listeners in different states and how EnvoyPatchPolicy interacts with this mechanism. Potential solutions may involve modifying the EnvoyPatchPolicy controller to explicitly target both active and draining listeners, or adjusting the Envoy configuration process to ensure that policies are consistently applied across all listeners. Ultimately, the goal is to ensure that the application functions correctly regardless of the Envoy worker's state, providing a reliable and consistent user experience.

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyPatchPolicy
metadata:
  name: keep-escaped-slashes
  namespace: esrp
spec:
  type: JSONPatch
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: couchdb
  jsonPatches:
    - name: esrp/couchdb/http-couchdb
      type: type.googleapis.com/envoy.config.listener.v3.Listener
      operation:
        op: replace
        path: /default_filter_chain/filters/0/typed_config/path_with_escaped_slashes_action
        value: KEEP_UNCHANGED

The EPP status shows:

status:
  ancestors:
  - ancestorRef:
      group: gateway.networking.k8s.io
      kind: Gateway
      name: couchdb
      namespace: esrp
    conditions:
    - lastTransitionTime: "2025-12-01T18:07:46Z"
      message: Policy has been accepted.
      observedGeneration: 1
      reason: Accepted
      status: "True"
      type: Accepted
    - lastTransitionTime: "2025-12-01T18:07:46Z"
      message: Patches have been successfully applied.
      reason: Programmed
      status: "True"
      type: Programmed
  controllerName: gateway.envoyproxy.io/gatewayclass-controller

Analyzing the Configuration

The provided YAML configurations offer valuable insights into the problem. The EnvoyPatchPolicy is designed to modify the path_with_escaped_slashes_action within the Envoy listener. This setting is crucial for applications that need to handle escaped slashes in URLs correctly. The policy targets a Gateway resource named couchdb in the esrp namespace, indicating that it is intended to apply to the Envoy listener associated with this gateway. The JSON patch operation specifies that the path_with_escaped_slashes_action should be replaced with the value KEEP_UNCHANGED. This ensures that escaped slashes in the URL are preserved, which is essential for certain applications. The EPP status confirms that the policy has been accepted and the patches have been successfully applied. However, as the user noted, this status is misleading because the patches are only applied to the active listener and not the draining listener. This discrepancy highlights a critical gap in the policy application process. To fully analyze the configuration, it is important to understand how Envoy manages listeners in different states. Envoy maintains separate listeners for active and draining states, and configurations applied to the active listener may not automatically propagate to the draining listener. This separation is designed to ensure that new configurations do not interfere with existing connections that are being drained. However, it also means that any configuration changes, including those applied via EnvoyPatchPolicy, need to be explicitly applied to both the active and draining listeners to ensure consistency. The fact that the EPP status indicates that the patches have been successfully applied, even though they are not applied to the draining listener, suggests that the status reporting mechanism may not be fully aware of the listener's state. This can lead to confusion and make it difficult to diagnose issues related to policy application. To address this, it may be necessary to enhance the status reporting mechanism to provide more granular information about the policy's application to different listeners. This could involve adding a new condition to the EPP status that specifically indicates whether the policy has been applied to the draining listener. Additionally, it is important to consider the implications of applying policies to the draining listener. In some cases, it may be desirable to apply the same policies to both active and draining listeners to ensure consistency. However, in other cases, it may be necessary to apply different policies to the draining listener to ensure a graceful transition. For example, during draining, it may be necessary to redirect traffic to a different set of backend servers or to apply stricter security policies. The decision of whether to apply the same policies to both listeners depends on the specific requirements of the application and the desired behavior during draining. In summary, the provided configurations highlight a critical issue with EnvoyPatchPolicy: the lack of consistent application of policies to listeners in the draining state. Addressing this issue requires a deeper understanding of how Envoy manages listeners in different states, as well as enhancements to the policy application and status reporting mechanisms. By carefully analyzing the configuration and considering the specific requirements of the application, it is possible to develop a solution that ensures consistent and reliable policy application across all listeners.

Potential Solutions and Workarounds

Several potential solutions and workarounds can address the issue of EnvoyPatchPolicy not being applied in the draining_state:

1. Modify the EnvoyPatchPolicy Controller: The most direct solution involves modifying the EnvoyPatchPolicy controller to explicitly target both the active and draining listeners. This would ensure that any policy applied to the active listener is also applied to the draining listener. This approach requires a deeper understanding of the EnvoyPatchPolicy controller's implementation and may involve changes to the controller's code. The modification would need to identify the draining listener and apply the same patches that are applied to the active listener. This could involve adding a new logic to the controller that iterates through all listeners and applies the patches to each one. It is important to carefully test this modification to ensure that it does not introduce any unintended side effects. For example, it is possible that applying the same patches to both listeners could lead to conflicts or unexpected behavior in certain scenarios. Therefore, it is crucial to thoroughly validate the changes before deploying them to a production environment.

2. Use a Webhook to Apply Patches: An alternative approach is to use a webhook to intercept the listener creation or update events and apply the patches directly. This webhook could be configured to listen for events related to Envoy listeners and apply the necessary patches whenever a listener is created or updated. This approach provides more flexibility and control over the patching process. The webhook can be implemented as a separate service that runs alongside the Envoy proxy and the EnvoyPatchPolicy controller. When a listener is created or updated, the webhook receives a notification and applies the patches specified in the EnvoyPatchPolicy. This approach allows for more fine-grained control over the patching process, as the webhook can be configured to apply different patches based on the listener's state or other criteria. For example, the webhook could apply a different set of patches to the draining listener than to the active listener. This can be useful in scenarios where different configurations are required for different listener states. However, using a webhook also adds complexity to the system, as it introduces a new component that needs to be managed and maintained. It is important to carefully consider the trade-offs between flexibility and complexity when choosing this approach.

3. Explicitly Define Policies for Draining Listeners: Another workaround is to explicitly define separate EnvoyPatchPolicies that target the draining listeners. This approach involves creating a second policy that specifically targets the draining listener and applies the same patches as the original policy. This ensures that the desired configuration is applied to both listeners. This workaround is relatively simple to implement, as it only requires creating a new EnvoyPatchPolicy resource. However, it also has some drawbacks. For example, it requires duplicating the policy definition, which can make it more difficult to manage and maintain the policies. Additionally, it is possible that the two policies could conflict with each other, leading to unexpected behavior. Therefore, it is important to carefully consider the implications of this approach before implementing it.

4. Envoy Configuration Templates: Using Envoy configuration templates can provide a more robust solution. Templates allow you to define the desired configuration with placeholders that can be dynamically populated. This ensures consistency across all listeners, including those in the draining_state. This approach involves creating a template for the Envoy configuration that includes placeholders for the values that need to be patched. The template can then be used to generate the actual Envoy configuration by replacing the placeholders with the appropriate values. This approach provides a high degree of consistency and can simplify the management of Envoy configurations. However, it also requires a deeper understanding of Envoy configuration templates and may involve changes to the deployment process. It is important to carefully design the template to ensure that it meets the requirements of the application and that it is easy to maintain. Additionally, it is important to consider how the placeholders will be populated with the correct values. This may involve using a separate service or tool to generate the configuration from the template.

The choice of solution depends on the specific requirements and constraints of your environment. Modifying the EnvoyPatchPolicy controller offers a comprehensive fix but requires more development effort. Using a webhook provides flexibility but adds complexity. Explicitly defining policies for draining listeners is a simpler workaround but may lead to duplication. Envoy configuration templates offer a robust solution but require more initial setup.

Implementing a Webhook Solution (Example)

To illustrate a practical solution, let's consider how a webhook could be implemented to address the issue. A webhook can intercept listener creation or update events and apply patches directly, ensuring that both active and draining listeners receive the necessary configurations. The basic steps for implementing a webhook solution are as follows:

1. Develop the Webhook: Create a service that acts as the webhook. This service should listen for HTTP requests from the Kubernetes API server and be able to process admission review requests. The webhook service needs to be able to handle admission review requests, which are sent by the Kubernetes API server when a resource is created or updated. The admission review request contains information about the resource being created or updated, including its kind, name, namespace, and configuration. The webhook service needs to be able to parse this information and determine whether to apply any patches. The webhook service should also be able to apply the patches to the resource configuration. This typically involves modifying the resource's YAML or JSON representation and returning a modified version of the resource in the admission review response. The webhook service can be implemented using a variety of programming languages and frameworks. For example, it could be implemented using Go, Python, or Java, and it could use a framework such as Kubernetes client-go or Spring Boot. The choice of programming language and framework depends on the specific requirements and preferences of the development team.

2. Configure the Webhook: Deploy the webhook service to your Kubernetes cluster and configure it to listen for listener-related events. This involves creating a Kubernetes service and deployment for the webhook service and configuring a mutating webhook configuration to intercept the relevant events. The mutating webhook configuration tells the Kubernetes API server to send admission review requests to the webhook service when certain resources are created or updated. The configuration specifies the resources to be intercepted, the namespaces to be included or excluded, and the URL of the webhook service. It is important to carefully configure the webhook to ensure that it only intercepts the events that it needs to process. Intercepting too many events can lead to performance issues, while intercepting too few events can cause the webhook to miss important updates.

3. Apply Patches: Within the webhook, implement the logic to apply the necessary patches based on the EnvoyPatchPolicy definitions. This involves parsing the EnvoyPatchPolicy and applying the JSON patches to the listener configuration. The webhook service needs to be able to parse the EnvoyPatchPolicy and extract the patches to be applied. This typically involves unmarshaling the EnvoyPatchPolicy YAML or JSON representation into a data structure that can be easily processed. The webhook service then needs to apply the patches to the listener configuration. This can be done using a variety of techniques, such as JSON patching libraries or custom code that manipulates the configuration directly. It is important to carefully apply the patches to ensure that they are applied correctly and that they do not introduce any errors or inconsistencies. The webhook service should also handle any errors that may occur during the patching process and return an appropriate error response to the Kubernetes API server.

4. Test the Solution: Thoroughly test the webhook to ensure that it correctly applies patches to both active and draining listeners. This involves creating and updating listeners and verifying that the patches are applied as expected. Testing the webhook is crucial to ensure that it is working correctly and that it does not introduce any issues into the system. The tests should cover a variety of scenarios, such as creating new listeners, updating existing listeners, and deleting listeners. The tests should also verify that the patches are applied correctly and that they do not introduce any errors or inconsistencies. Automated testing can be used to ensure that the webhook is tested consistently and that any issues are quickly identified.

This approach ensures that the configurations are consistently applied across all listeners, including those in the draining_state, resolving the reported issue.

Best Practices for EnvoyPatchPolicy

To effectively use EnvoyPatchPolicy and avoid common pitfalls, consider the following best practices:

• Granular Policies: Create policies that are as granular as possible. This makes it easier to understand and manage the impact of each policy. Granular policies allow for more precise control over the Envoy configuration. This means that you can target specific parts of the configuration and modify them without affecting other parts. This fine-grained control is essential for complex deployments where different parts of the system may require different configurations. For example, you might create a separate policy for each listener or for each set of filters. This makes it easier to understand which policy is affecting which part of the configuration and to troubleshoot any issues that may arise. Granular policies also make it easier to manage the policies over time. As your application evolves, you may need to update the policies to reflect the changes. Granular policies make it easier to make these updates without affecting other parts of the configuration. Additionally, granular policies can improve the performance of the system. When a policy is applied, Envoy needs to evaluate the policy and determine whether it applies to the current configuration. Granular policies are typically simpler and faster to evaluate than more complex policies. This can reduce the overhead of policy evaluation and improve the overall performance of the system.

• Use Namespaces: Organize policies by namespace to improve clarity and prevent conflicts. Using namespaces helps to isolate policies and prevent them from interfering with each other. This is particularly important in multi-tenant environments where different teams or applications may be using the same Envoy proxy. By organizing policies by namespace, you can ensure that each team or application has its own set of policies and that these policies do not conflict with the policies of other teams or applications. Namespaces can also be used to control access to policies. You can grant different permissions to different users or groups based on their namespace. This allows you to restrict access to sensitive policies and prevent unauthorized users from making changes. Additionally, namespaces can improve the organization and manageability of the policies. By grouping policies by namespace, you can make it easier to find and manage the policies that are relevant to a particular team or application. This can simplify the process of troubleshooting issues and making updates to the policies.

• Monitor Policy Status: Regularly check the status of your EnvoyPatchPolicy resources to ensure they are being applied correctly. Monitoring the policy status is crucial for ensuring that the policies are being applied as expected and that there are no issues with the configuration. The status of the EnvoyPatchPolicy resource provides valuable information about the policy's application. The conditions section of the status indicates whether the policy has been accepted, whether the patches have been successfully applied, and whether there are any errors or warnings. By regularly checking the status, you can quickly identify any issues and take corrective action. Monitoring the policy status can also help to prevent issues from occurring in the first place. By proactively monitoring the status, you can identify potential problems before they impact the application. For example, if a policy is not being applied correctly, you can investigate the issue and fix it before it causes any service disruptions. There are a variety of tools and techniques that can be used to monitor the policy status. For example, you can use the Kubernetes command-line tool (kubectl) to check the status of the EnvoyPatchPolicy resources. You can also use monitoring tools such as Prometheus and Grafana to create dashboards and alerts that track the policy status over time. Additionally, you can configure the EnvoyPatchPolicy controller to send notifications when there are any changes to the policy status.

• Test Policies in a Staging Environment: Always test your policies in a staging environment before applying them to production. Testing policies in a staging environment is essential for ensuring that they are working correctly and that they do not introduce any unintended side effects. A staging environment is a replica of the production environment that can be used to test changes before they are deployed to production. This allows you to identify and fix any issues in a safe and controlled environment. When testing policies in a staging environment, it is important to simulate the production workload as closely as possible. This includes sending traffic to the staging environment, running the application in the same configuration as production, and monitoring the performance of the system. Testing policies in a staging environment can help to identify a variety of issues, such as configuration errors, performance bottlenecks, and security vulnerabilities. By identifying and fixing these issues before deploying to production, you can reduce the risk of service disruptions and improve the overall reliability of the application. Additionally, testing policies in a staging environment can help to build confidence in the changes and make it easier to deploy them to production. When you have successfully tested the policies in a staging environment, you can be more confident that they will work correctly in production.

• Use Version Control: Keep your EnvoyPatchPolicy definitions in version control to track changes and facilitate rollbacks. Using version control is a fundamental best practice for managing any type of configuration, including EnvoyPatchPolicy definitions. Version control systems such as Git allow you to track changes to files over time, making it easy to see who made what changes and when. This is particularly important for EnvoyPatchPolicy definitions, as they can have a significant impact on the behavior of the Envoy proxy. By keeping the definitions in version control, you can easily track changes, revert to previous versions if necessary, and collaborate with other team members on the definitions. Version control also facilitates rollbacks. If a new policy is deployed and it causes issues, you can easily revert to a previous version of the policy. This can help to minimize the impact of the issue and restore the system to a working state. Additionally, version control can improve the security of the policies. By tracking changes and controlling access to the repository, you can prevent unauthorized users from making changes to the policies. This can help to protect the system from security vulnerabilities.

By following these best practices, you can ensure that your EnvoyPatchPolicy configurations are robust, manageable, and effective.

Conclusion

Applying EnvoyPatchPolicy to listeners in the draining_state is crucial for maintaining application functionality during Envoy worker transitions. The reported issue highlights the importance of ensuring consistent policy application across all listener states. By understanding the nuances of Envoy's draining state and implementing appropriate solutions, such as modifying the EnvoyPatchPolicy controller or using webhooks, you can ensure a seamless experience for your users. Remember to follow best practices for policy management, including granular policies, namespaces, status monitoring, staging environments, and version control, to maintain a robust and manageable configuration.

For further information on Envoy Proxy and its configurations, please refer to the official Envoy documentation available on the Envoy Proxy Website.