Dapr Custom Application Metrics

Question

I have a question regarding dapr metrics. Today when I enable metrics on port 9090, I am able to see all the available standard dapr metrics. How do I emit custom metrics for my application so that my custom metrics is available at port 9090 as well ? Is there any code example, preferably python on how to emit custom metrics via dapr sidecar ?

My objective is to have dapr sidecar emitting all metrics and we can configure OTEL collector to receive from one port 9090. If this is not possible, am I right to say that we need to configure OTEL to receive 2 metrics from each respective port 9090 (sidecar) and 9091 (application) assuming our application is emitting custom metrics via port 9091 as describe below in Custom Application Metrics ?

Standard Dapr Metrics

When I curl port 9090, i am able to get all the metrics as described in https://github.com/dapr/dapr/blob/master/docs/development/dapr-metrics.md#dapr-common-metrics

Custom Application Metrics

Is there a way to emit the custom metrics to dapr ? OR I have to expose my application metrics with another port like 9091 and configure OTEL collector to scrap directly from my Python application and standard Dapr sidecar ?

Example Python App Custom Metrics

import time

from prometheus_client import start_http_server

from opentelemetry import metrics
from opentelemetry.exporter.prometheus import PrometheusMetricReader
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.sdk.resources import SERVICE_NAME, Resource


if __name__ == '__main__':
    # Service name is required for most backends
    resource = Resource(attributes={
        SERVICE_NAME: "otel-prom-test"
    })

    # Start Prometheus client
    start_http_server(port=9091, addr="0.0.0.0")
    # Initialize PrometheusMetricReader which pulls metrics from the SDK
    # on-demand to respond to scrape requests
    reader = PrometheusMetricReader()
    provider = MeterProvider(resource=resource, metric_readers=[reader])
    metrics.set_meter_provider(provider)

    meter = metrics.get_meter_provider().get_meter("getting-started", "0.1.2")

    # Counter
    print("Synchronous counter")
    counter = meter.create_counter("test_counter")
    counter.add(1)
    print("Sleeping for 3 secs.")
    time.sleep(10)
    counter.add(1)
    print("Finished counting.")

Example custom metrics output via port 9091

# HELP python_gc_objects_collected_total Objects collected during gc
# TYPE python_gc_objects_collected_total counter
python_gc_objects_collected_total{generation="0"} 361.0
python_gc_objects_collected_total{generation="1"} 163.0
python_gc_objects_collected_total{generation="2"} 0.0
# HELP python_gc_objects_uncollectable_total Uncollectable objects found during GC
# TYPE python_gc_objects_uncollectable_total counter
python_gc_objects_uncollectable_total{generation="0"} 0.0
python_gc_objects_uncollectable_total{generation="1"} 0.0
python_gc_objects_uncollectable_total{generation="2"} 0.0
# HELP python_gc_collections_total Number of times this generation was collected
# TYPE python_gc_collections_total counter
python_gc_collections_total{generation="0"} 55.0
python_gc_collections_total{generation="1"} 4.0
python_gc_collections_total{generation="2"} 0.0
# HELP python_info Python platform information
# TYPE python_info gauge
python_info{implementation="CPython",major="3",minor="11",patchlevel="5",version="3.11.5"} 1.0
# HELP test_counter_total
# TYPE test_counter_total counter
test_counter_total 1.0