-
Notifications
You must be signed in to change notification settings - Fork 75
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add monitorType JMX #472
base: main
Are you sure you want to change the base?
Add monitorType JMX #472
Conversation
I am not very experienced with Java and Testing in Java. Please feel free to add tests. |
512ce06
to
82cf53d
Compare
82cf53d
to
40cb273
Compare
gateway-ha/src/main/java/io/trino/gateway/ha/config/ClusterStatsMonitorType.java
Show resolved
Hide resolved
40cb273
to
5389a0d
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just skimmed.
gateway-ha/src/main/java/io/trino/gateway/ha/clustermonitor/ClusterStatsJmxMonitor.java
Show resolved
Hide resolved
gateway-ha/src/main/java/io/trino/gateway/ha/handler/QueryIdCachingProxyHandler.java
Outdated
Show resolved
Hide resolved
gateway-ha/src/main/java/io/trino/gateway/ha/clustermonitor/ClusterStatsJmxMonitor.java
Outdated
Show resolved
Hide resolved
gateway-ha/src/main/java/io/trino/gateway/ha/clustermonitor/ClusterStatsJmxMonitor.java
Outdated
Show resolved
Hide resolved
gateway-ha/src/main/java/io/trino/gateway/ha/clustermonitor/ClusterStatsJmxMonitor.java
Outdated
Show resolved
Hide resolved
Thanks for the reviews. I was on vacation and didn't have time to implement the requested changes yet. I will take care of them in the next days. I might need some support to activate the /v1/jmx/mbean endpoint in the test environment for testing. |
@alaturqua please ping me when this is ready for another look |
fb4ac57
to
b333076
Compare
b333076
to
a11e5a4
Compare
I had to sync my fork to be able to change my PR. But it got automatically closed, when I did so. |
6aebdfb
to
995229a
Compare
gateway-ha/src/main/java/io/trino/gateway/ha/clustermonitor/ClusterStatsJmxMonitor.java
Outdated
Show resolved
Hide resolved
gateway-ha/src/main/java/io/trino/gateway/ha/clustermonitor/ClusterStatsJmxMonitor.java
Outdated
Show resolved
Hide resolved
a79acde
to
7671694
Compare
4fb7de5
to
847a945
Compare
I addressed all the points and made the changes. Tests are passing as well. Please review again. |
847a945
to
b51df33
Compare
@willmostly added documentation as well. |
145f0ca
to
39585df
Compare
39585df
to
0abf92c
Compare
This is using `v1/jmx/mbean` endpoint on trino clusters. | ||
To enable this, [JMX monitoring](https://trino.io/docs/current/admin/jmx.html) must be activated on all Trino clusters. | ||
|
||
Ensure that a username and password are configured by adding the `backendState` section to your configuration. The credentials must be consistent across all backend clusters and have `read` rights on the `system_information`. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
and have
read
rights on thesystem_information
.
Is there anywhere in the Trino documentation we could link to that gives an example of how to set this up? Or we could provide a snippet of FBAC configuration
Request preparedRequest = prepareGet() | ||
.setUri(uriBuilderFrom(URI.create(jmxUrl)) | ||
.appendPath(JMX_PATH) | ||
.appendPath(mbeanName) | ||
.build() | ||
).addHeader("X-Trino-User", username) | ||
.build(); | ||
|
||
boolean isHttps = preparedRequest.getUri().getScheme().equalsIgnoreCase("https"); | ||
|
||
if (isHttps) { | ||
HttpRequestFilter filter = new BasicAuthRequestFilter(username, password); | ||
request = filter.filterRequest(preparedRequest); | ||
} | ||
else { | ||
request = preparedRequest; | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add a method to create the basic auth header. I'm not sure if this is the intended use case for the HttpRequestFilter classes. Then you can just
Request preparedRequest = prepareGet()
.setUri(uriBuilderFrom(URI.create(jmxUrl))
.appendPath(JMX_PATH)
.appendPath(mbeanName)
.build())
.addHeader("X-Trino-User", username);
if (isHttps) {
preparedRequest.addHeader(createBasicAuthHeader(username, password));
}
preparedRequest.build();
and use preparedRequest
directly.
JmxStatProcessor processor, ClusterStats.Builder clusterStats) | ||
{ | ||
queryJmx(backend, mbeanName) | ||
.ifPresent(response -> processor.process(response, clusterStats)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
queryJmx
may return Optional.empty()
in the case of an error response. In this case ifPresent
will not trigger, and the backend stats will not be updated, so the backend will still appear healthy. You can avoid this by doing something like
.ifPresent(response -> processor.process(response, clusterStats)); | |
.ifPresentOrElse((response -> processor.process(response, clusterStats), () -> clusterStats.trinoStatus(TrinoStatus.UNHEALTHY)); |
implements ClusterStatsMonitor | ||
{ | ||
private static final Logger log = Logger.get(ClusterStatsJmxMonitor.class); | ||
private static final JsonResponseHandler<JsonNode> JMX_JSON_RESPONSE_HANDLER = |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
create a pair of record
classes for the JMX response and the attributes items, and use this instead of the generic JsonNode
here and in processDiscoveryNodeManagerStats
and processQueryManagerStats
@@ -61,6 +62,12 @@ void testJdbcMonitor() | |||
testClusterStatsMonitor(backendStateConfiguration -> new ClusterStatsJdbcMonitor(backendStateConfiguration, new MonitorConfiguration())); | |||
} | |||
|
|||
@Test |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
add tests for the failure cases, 400 and 500 response codes with non json and invalid json response bodies
Description
This pull request introduces a new
monitorType
for JMX-based monitoring in Trino Gateway. It includes documentation and configuration details for retrieving cluster information using specific JMX metrics.Details
Component: JMX Monitoring in Trino Gateway
New Feature: Added support for
monitorType: JMX
to enable cluster monitoring via JMX metrics.Metrics Used:
trino.execution:name=QueryManager
trino.metadata:name=DiscoveryNodeManager
Configuration Instructions:
backendState
.read
rights onsystem_information
.Example Configuration:
Additional context and related issues
Release notes
( ) This is not user-visible or is docs only, and no release notes are required.
(x) Release notes are required. Please propose a release note for me.
( ) Release notes are required, with the following suggested text:
*