Update: Based on App Metrics 1.0, latest documentation can be found here

One of my previous posts was about open source application performance management, and why we even consider open source. If we were to give an open source monitoring solution a shot for an ASP.NET Core application, what options are available? One option is an open source project I’ve been working on, App Metrics. It targets .NET Standard which of course makes it a cross platform solution, provides various extensions allowing metrics to be captured in process and persisted to several TSDBs, as well as providing application health checks.

Here we’ll walk through getting started on a Windows 10 developer machine using InfluxDB, Grafana and App Metrics to monitor in real-time, the performance of an ASP.NET Core Application.

Setup the ASP.NET Core Application

For those not yet faimilar with ASP.NET Core, see the getting started tutorial.

Next we’ll need to add the required App Metrics nuget packages, install App.Metrics.Extensions.Mvc and App.Metrics.Formatters.Json, then configure the required services using Microsoft’s dependency injection framework.

Replace the ConfigureServices method created by the project template with the following:

public void ConfigureServices(IServiceCollection services)
{
   services.AddMetrics()
      .AddHealthChecks()
      .AddJsonSerialization()
      .AddMetricsMiddleware(options => options.IgnoredHttpStatusCodes = new [] {404});

    services.AddMvc(options => options.AddMetricsResourceFilter());
}

We can then register the default middleware App Metrics provides via the ​IApplicationBuilder, replace the Configure method with the following:

public void Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory)
{
   loggerFactory.AddConsole(Configuration.GetSection("Logging"));
   app.UseMetrics();
   app.UseMvc();
}

That’s all the configuration we need to run with the defaults. Hit F5 and request the following urls:

  • /ping – simply returns pong, which is useful to use for load balancers to determine if there is connectivity to the application.
  • /health – executes all registered health checks and returns a JSON representation of the results.
  • /metrics – takes a snapshot of the application’s metrics and returns a JSON representation of the measurments.

Request a few different endpoints multiple times (or pages if your’ve created a website project) and then visit the metrics endpoint again, you’ll see something similar to the following for each endpoint

{
   name: "Transactions Per Endpoint|route:GET api/values/{id}",
   unit: "Requests",
   activeSessions: 0,
   count: 2,
   durationUnit: "ms",
   histogram: {
      lastValue: 25.579883,
      max: 69.527087,
      mean: 47.553484,
      median: 69.527087,
      min: 25.579883,
      percentile75: 69.527087,
      percentile95: 69.527087,
      percentile98: 69.527087,
      percentile99: 69.527087,
      percentile999: 69.527087,
      sampleSize: 2,
      stdDev: 21.973602,
      sum: 95.106969
   },
   rate: {
      fifteenMinuteRate: 24,
      fiveMinuteRate: 24,
      meanRate: 35.072894,
      oneMinuteRate: 24
   },
   rateUnit: "min",
   tags: {
      route: "GET api/values/{id}"
   }
}

The metrics endpoint doesn’t provide much value on it’s own in terms of visualization. In this tutorial we’ll push our application’s metrics to InfluxDB and use Grafana for real-time analysis. We’ll start by setting up InfluxDB.

Setup InfluxDB on Windows Subsystem for Linux

InfluxDB will run on Windows, I did find there were some issues with the CLI running on Windows, however it does run perfectly using “Ubuntu on Windows”. If not familiar with this have a quick read through the FAQ and follow the installation guide provided by Microsoft.

With Ubuntu now running on Windows, next we’ll install InfluxDB. Run a new bash shell with Administrator privileges and run the following commands to add the InfluxData repository:

$ curl -sL https://repos.influxdata.com/influxdb.key | sudo apt-key add -
$ source /etc/lsb-release
$ echo "deb https://repos.influxdata.com/${DISTRIB_ID,,} ${DISTRIB_CODENAME} stable" | sudo tee /etc/apt/sources.list.d/influxdb.list

Then run the following commands to install and start InfluxDB:

$ sudo apt-get update && sudo apt-get install influxdb
$ sudo influxd

That’s all there is to get up and running with InfluxDB. We do need to create a new InfluxDB database to store our metrics, lets do that. Open another bash shell, again with Administrator privileges, and run the following command to start the InfluxDB shell and then create a new database for our metrics:

$ sudo influx
> create database appmetricsdemo

Setup Grafana

App Metrics comes with a couple of generic Grafana dashboards built to work with metrics captured by the ASP.NET Core Middleware we added earlier using the App.Metrics.Extensions.Middleware package.This allows us to simply import these dashboards which provide a fairly comprehensive insight into our web application’s performance.

Grafana will run on Windows, Linux or Mac, you can follow the instructions here to download via command line or download manually.

Leaving the default configuration start Grafana by running the following command:

C:\> .\bin\grafana-server.exe

Grafana by default listens on port 3000, open a browser and navigate to http://localhost:3000/.

App Metric’s dashboards can be discovered at grafana.com. For this tutorial, import this Grafana dashboard by copying the dashboard ID and pasting into the Import Dashboard window.

The result will look as follows:

AppMetricsGenericWebGrafanaDashboard

Obvisouly at the moment we don’t see any data points, we’ll first need to have our web application push it’s metrics.

Configure InfluxDB Reporting

To configure reporting in App Metrics we just need a few more lines of start up configuration.

First we’ll need to install the App.Metrics.Extensions.Reporting.InfluxDB package to provide the reporting options.

Next, configure the required reporting services, once again replace the ConfigureServices with the following:

public void ConfigureServices(IServiceCollection services)
{
   var database = "appmetricsdemo";
   var uri = new Uri("http://127.0.0.1:8086");

   services.AddMetrics(options => 
      {
         options.WithGlobalTags((globalTags, info) => 
         { 
            globalTags.Add("app", info.EntryAssemblyName); 
            globalTags.Add("env", "stage");
         });
      })
      .AddHealthChecks()
      .AddJsonSerialization()
      .AddReporting(
         factory =>
         {
            factory.AddInfluxDb(
               new InfluxDBReporterSettings
               {                  
                  InfluxDbSettings = new InfluxDBSettings(database, uri),
                  ReportInterval = TimeSpan.FromSeconds(5)
               });
       })
      .AddMetricsMiddleware(options => options.IgnoredHttpStatusCodes = new [] {404});

    services.AddMvc(options => options.AddMetricsResourceFilter());
}

Lastly, we need to run the reporter once our web application has successfully started, to do so once again replace the Configure method with the following:

public void Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory, IApplicationLifetime lifetime)
{
   loggerFactory.AddConsole(Configuration.GetSection("Logging"));
   app.UseMetrics();
   app.UseMetricsReporting(lifetime);
   app.UseMvc();
   // Or if an ASP.NET Core MVC Web Application template was created rather than a Web API
   // app.UseMvc(routes => 
   // {
   // routes.MapRoute(
   // "default",
   // "{controller=Home}/{action=Index}/{id?}");
   // });
}

Finishing up

That’s all the setup and configuration done, hit F5, run some requests, then view the dashboard in Grafana to see something like the blow.

app metrics grafana

A single instance of the dashboard can be re-used for mulitple applications and environments where the application and environment variables at the top of the dashbaord can be adjusted to view each app independently across their environments.

Advertisements

Posted by Allan Hardy

11 Comments

  1. Hello, can you give an example how to save the metric informations to a sql database?

    Like

    Reply

    1. There are various reporters written to persist metrics recorded by App Metrics, see the list on the github repo’s readme (https://github.com/alhardy/AppMetrics/blob/dev/README.md#what-is-app-metrics). SQL databases aren’t really a good choice for timeseries, however if required you could use one of the existing reporters as a guide in implementing a custom SQL reporter.

      Like

      Reply

      1. Hello Allan,

        thanks for this really really fast answer. Two more questions:
        1. Where is the documentation for the Metrics?
        2. Are the metrics saved in an file?

        Thanks for your help and a really great articel!

        Like

  2. No problem and thank you.

    1. You can find the documentation at http://app-metrics.io
    2. You can use the Text File Reporter http://app-metrics.io/reporting/text-file.html, other than for debugging, not very useful though

    Like

    Reply

  3. Hey Allan,

    First of all thanks for the work on AppMetrics, it’s really nice to have a metrics collection tool nicely tailored for ASP.NET Core.

    I have a question: I tried using it in combination with Grafana + Prometheus, and I imported the dashboard https://grafana.com/dashboards/2204.
    I’m running my api in a containerized, auto-scaling infrastructure (Kubernetes), so at all times I have multiple instances of the same api running, with different server names.

    This seems to trip up the dashboard if I don’t specify a concrete server in the server dropdown, but just select All.

    The first problem was that the graph received no data at all, I managed to fix that by changing server=”$server” to server=~”$server”. (I’m not sure why this is fixing it, maybe if I select “All”, then Grafana is passing in “.*” as $server, and that’s why the regex matching is needed?)

    The second problem was that Grafana recognized the data coming from each server separately, so I had as many separate lines in the graph as many servers I had, which is pretty difficult to read as an api scales out. I addressed this for example on the “Throughput” graph by wrapping the query in a “sum()”, so the final query looks like this:
    sum(rate(application_httprequests_transactions_count{env=”$environment”,app=”$application”,server=~”$server”}[1m]))*60

    I similarly managed to fix the “Response Time” graph by using the aggregate “avg()”.

    I haven’t looked at the other graphs yet (none of them works for me if I don’t select a server), first I wanted to check with you: have you encountered this problem, or do you think this is the correct way to fix it?

    Cheers,
    Mark

    Like

    Reply

    1. Hey Mark,

      1) Re grafana’s “All” and Prometheus, are you using the 2.0 alpha version of Prometheus? I’m yet to test with this. This is working for me in the prior version, maybe throw that question over to https://github.com/grafana/grafana or log an issue at https://github.com/alhardy/AppMetrics.Extensions.Prometheus and ping @Rurouni, he’s using an App Metrics / Prometheus / Kubernetes setup, I’m on InfluxDB with my prod setup.
      2) App Metrics tags all metrics by server, Grafana is showing metrics per server because metrics are by default in the “quick start dashboards” grouped by server. If your preferred view is a sum, your modification is fine, or you could also just “stack” the metrics in the graphs settings.

      Hope this helps.

      Allan

      Like

      Reply

      1. Thanks for the info!

        Do you know what’s the best way to contact @Rurouni? I found a twitter account (@RurouniGhost), but it doesn’t seem to be active.

        Like

    1. Thanks for creating an issue, I’ll follow that.
      I have another question. (If this is not the preferred place to discuss, just tell and I can use email/github, etc., as you prefer.)
      So I managed to fix most of the dashboard, and it nicely aggregates all the metrics from all the servers. I still have some issues that I’m investigating, but there is one I’m stuck with: the Results panel in the Health row doesn’t show anything.
      This is its query: application_health_results{env=”stage”,app=”$application”,server=”$server”}
      I checked my /metrics-text endpoint, and I don’t have the application_health_results metric at all.
      I also cloned AppMetrics and AppMetrics.Extensions.Prometheus, but I can’t find it in the code where this is supposed to be reported. I have also added AddHealthChecks() and AddJsonHealthSerialization() in my Startup.
      Can you point me to the right direction where I can start investigating why this metric is not in my metrics stream?

      Like

      Reply

      1. You can ignore this one, finally I found it, but something is wrong with all the Table panels in my dashboard, so I’ll have to get familiar with that in Grafana to sort it out.

        Like

      2. Glad you’re making progress.

        In short, try requesting `/health` a few times and then view the health check section of the dashboard.

        The preferred place for questions like this would be GitHub.

        More than happy to assist with any other queries you have.

        ——————————

        The ‘/health’ endpoint will execute all your health checks and display the results. When health checks are executed, “health metrics” are also recorded which will then be flushed (in your case to Prometheus) providing visibility on current and historic results. Therefore, it is requried to periodically make a request to `/health’.

        A typical setup is to use a service like Pingdom which periodically makes a request to this endpoint to determine the health of your application, then if unhealthy, it will alert the fact.

        Alternatively, if you don’t want to pay for such a service, a “poor mans approach” could be to run a task within your application to request this endpoint and use Grafana’s alerting capability (http://docs.grafana.org/alerting/rules/) which supports a variety of notification types e.g. Email, Slack, Pager Duty etc.

        Other options could be to use Grafana’s World Ping plugin or to roll your own. The advantages of services like Pingdom and Grafana’s World Ping is that they run requests from multiple locations which may or may not be a requirement for you.

        Initially App Metrics was executing health checks as part of reporting metrics, however these are separate concerns and not required by all users, therefore the above approach was decided on instead.

        Like

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s