HTTP proxies

When to configure HTTP proxies

You may want to configure your Agent Management instance to work with HTTP proxies when either of the following scenarios applies:

Note

This must not be confused with the ability to deploy Agent Management behind a reverse proxy. See Using reverse proxies for more details.

About WebSocket

Agent Management uses the WebSocket protocol for parts of its user interface. The WebSocket protocol may not be supported by all HTTP proxies.

Ensure any direct or reverse proxy configured between Agent Management and its users correctly supports WebSocket, and is configured accordingly.

At the time of writing, this includes:

See Websockets problems for related details and troubleshooting advice.

Configuring a global proxy to allow Agent Management to access external resources

Routing all requests initiated by the Agent Management backend through a proxy

To route all requests initiated by the Agent Management backend through the proxy, you’ll need to configure the HTTP Proxy settings in the Administration menu.

  • Go to the Administration menu, select the Settings tab, and then open the Misc. panel.

  • In HTTP proxy, supply the proxy host and port.

  • Optional: If the HTTP proxy requires authentication, supply the username and password.

../_images/http-proxy-settings.png

Adding the proxy settings here will route all requests initiated by the Agent Management backend through the proxy.

Configuring a proxy for external HTTP-based or FTP-based network resources (for remote datasets)

If Agent Management runs inside your private network, you may need to configure an outgoing proxy to allow Agent Management to access external HTTP- or FTP-based network resources.

This applies in particular to HTTP, HTTPS and FTP remote datasets, and Amazon S3, Google Cloud Storage, Azure Blob, Snowflake, BigQuery and Elasticsearch datasets.

You can define a global proxy configuration for Agent Management in the Setting tab of the Administration page. Choose Proxy, fill in the fields, and save your changes.

Every HTTP(S)- and FTP-based connection will now have an additional “Use global proxy” checkbox. Clear the checkbox if the connection should not go through the proxy (e.g., for services that are inside your private network). This also applies to Amazon S3, GCS, Azure Blob, Snowflake, BigQuery, and Elasticsearch connections.

Note

SOCKS proxies are not supported in Agent Management.

Warning

A note on FTP through HTTP Proxy

Connecting to a FTP server through an HTTP proxy requires passive mode, and requires the proxy to allow and support HTTP CONNECT method on ports 20, 21 and all unpriviledged ports (1024-65535).

Below is a sample Apache 2.4 configuration for this:

Listen 3128
<VirtualHost *:3128>
  ProxyRequests On
  ProxyVia On
  AllowConnect 20 21 443 1024-65535
  <Proxy *>
    Order deny,allow
    Deny from all
    # IP of internal network
    Allow from 1.2.3.4
  </Proxy>
</VirtualHost>

Configuring a proxy for Python and R processes

Configuring access to external HTTP-based or FTP-based network resources only applies to native connections made from the Agent Management backend.

If you need to go through a proxy for network connections done from Python or R code (from a recipe or a notebook), you should configure the proxy using standard configuration directives for these environments. This includes adding explicit proxy parameters to the network calls, e.g., for Python requests:

requests.get(URL, proxies={'http', 'http://MYPROXY:MYPROXYPORT'})

and/or globally configuring proxy directives through the standard http_proxy (https_proxy, ftp_proxy …) environment variables, e.g.:

# Add the following directive to DATADIR/bin/env-site.sh
# or to the session initialization file of the Agent Management Unix user (.profile or equivalent)
export http_proxy="http://MYPROXY:MYPROXYPORT"

Refer to Python or R reference manuals for details.

Note

This also applies to network accesses needed to download and install additional Python or R packages.

Configuring Agent Management to trust a certificate authority

Agent Management may reside in a network that uses a cloud-based proxy as a man in the middle (MITM) proxy. In this case, Agent Management will need to trust the Certificate Authority of that proxy, as it will sign the communication that comes back to Agent Management.

The most efficient way to do this is via the system’s global trust store:

  • As root, copy the certificates that need to be trusted to /etc/pki/ca-trust/source/anchors

  • Again as root, run the update-ca-trust command

  • Finally, as the user running Agent Management, restart the application using the DATA_DIR/bin/dss restart command

For Python or PIP processes that may need to trust the certificate of a MITM proxy, the REQUESTSCABUNDLE environment variable can be used.

# Add the following directive to DATADIR/bin/env-site.sh
# or to the session initialization file of the Agent Management Unix user (.profile or equivalent)
export REQUESTSCABUNDLE=/path/to/CA_to_TRUST