RRDTool

The Round Robin Database Tool (RRDTool) is the open source industry standard, high performance data logging and graphing system for time series data. This module stores values from devices which have graphing enabled and datasources defines.

DomotiGa displays the (simple) graphs in the screen named according to the groups that you enabled for the device, so if you add a graph for a temperature sensor value and make the device part of the Climate group, you will find the graph in the Climate screen.

Databases are stored in domotiga/rrd, graphics in domotiga/rrd/graphs.


Example:

Below you will find instructions to install rrdtool, how to add a graph for a device, how to set up a web-server for a Client-Server configuration and information on debug and maintenance.

For a possible alternative giving interactive charts, see DomotiCharts : Nice interactive charts.

Install

$ sudo apt-get install rrdtool

To check if it is (already) installed:

$ rrdtool
RRDtool 1.4.7  Copyright 1997-2012 by Tobias Oetiker <tobi@oetiker.ch>
               Compiled Aug  2 2012 08:05:17

Adding a graphs for a device value

First enable RRDTool support under Modules -> RRDTool:

For the first configuration it is also wise to enable Debug.

Simple versus extended graphs

Two methods to add graphs are supported:
  1. Simple graphing
  2. Extended graphing

Differences:

Simple graphing Extended graphing
Simple to set up More difficult to set up
One graph per device One graph can based on the data of multiple devices
Fixed colors, scale and line types Lots of configurable items
Enable using the device value window Configuration split over device value window and Graph Editor menu

Simple graphing

When you want to monitor for instance the temperature or the energy logged by a device/sensor:

[*] If you want to create your own graph size, color and layout go for extended graphing and don't enable simple graphing.
[**] See this ticket, DERIVE and COUNTER operates on integer numbers only.

If you, after pressing Save, get a pop-up saying:

it is because DomotiGa creates one .rrd file per device and DomotiGa is not capable of changing the format of the RRD data or adding a data source. If you do not select Delete then very likely the data collection will not work.

Next you want to click the Groups tab and put the device in a group you want to see the graphs under, for example, Climate, Security, Energy, HVAC.

Extended graphing

To create an extended graph first make sure that you enabled Log RRD Data did put a descriptive name in the DataSource Name field for the values you want to graph. See above, Simple graphing on how to do this.

Next open the Graph Editor using Edit -> Graphs

Enter a name in Name field and click Save.
Then re select the graph just created.
Define Units to use and change size if wanted.
Click Save.

Goto Graph Data tab, and select the device, then select the DataSource from the list, if the device is not there RRD log is not enabled for that device and/or DS.
Enter a Name, click Save next to ID.
Open tab Groups and add the graph to one or more groups to get them displayed on a page, for example Climate, Energy, Security.

Click Save and close window.
You should now see the extended graph on the page Climate, Energy or other group you added it too...
Wait for it to update.

Client-Server configuration

Domotiga creates the RRD database in the domotiga/rrd diorectory, the filename is <id of device>.rrd.

When graphs are needed, a rrdtool command is given to create the graphs. If you run DomotiGaServer as server and one or more DomotiGa's as client, the client sends a JSON-RPC command to let the server create the rrdtool graphs it needs.

GraphsUrl

Domotiga uses http fetch to download the RRD graphs it wants to display. The base URL for this is by default http;//localhost/graphs/ but should be changed if you run in a server-client configuration and if you changed setting so that the graphs are no longer on the default location.

Changing the GraphsUrl can be done in the GUI settings window: Setup -> +G+UI

Web server configuration

If you run in a server-client configuration you need to enable http access to the graphs.

For general instructions on how to set up a web server, see DomotiYii Installation.
Below you find instructions how to enable your webserver to access the graphs using http;//<ip address of server>/graphs/.

You should be able to surf to this address with a webbrowser and you should see a directory with the graph images from the DomotiGaServer.

Once done, change the base URL in GUI Settings, that can be accessed using Setup --> Setup --> GUI. Here you want to change the GraphsURL field. Be sure to restart Domotiga.

Apache

127.0.0.0 allows to access it via the same machine (localhost) , change 192.168.178.0 to your local subnet to allow other machines to access /graphs.

lighttpd

Add these lines to your config (sudo vi /etc/lighttpd/lighttpd.conf):

dir-listing.activate = "enable" 
alias.url += ( "/graphs/" => "/home/<user>/domotiga/rrd/graphs/" )

nginx

Add these lines to your config (sudo vi /etc/nginx/sites-enabled/default):

    # Also serve the rrd graphs
    location /graphs {
        root  /home/<user>/domotiga/rrd/graphs;
        autoindex on;
        rewrite ^/graphs(/.*\.png)$ $1 break;
        rewrite ^/graphs/?$ / break;
    }

Debug

A good start is to get Server Stats working.

For debugging, enable debug output by enabling it in the RDDTool Settings: Modules --> RRDTool. Check the Debug checkbox.
Next wait till the problem happened and open the debug log file that you can find as ~/domotiga/log/debug-YYYY-MM.log (replace YYYY-MM by the current year-month). Do here a search for [RRDTool] and see if there message reveal anything that is wrong.

If you do not see values logged to a particular rrd you can dump the value of the last 10 minutes using:

$ rrdtool fetch ~/domotiga/rrd/serverload.rrd AVERAGE -s -10min
                          load1               load5              load15

1411242720: 1,6859972267e-01 2,2536244800e-01 1,8178748267e-01
1411242780: 1,3161101550e-01 1,9483304650e-01 1,8000000000e-01
1411242840: 6,2866196000e-02 1,6482482350e-01 1,6321654900e-01
1411242900: 1,8422362667e-01 1,7677795333e-01 1,6838897667e-01
1411242960: 1,0092521383e-01 1,5482889550e-01 1,6160963183e-01
1411243020: 1,3018806600e-01 1,5836467767e-01 1,6000000000e-01
1411243080: 6,4499484500e-02 1,3483316150e-01 1,5161105383e-01
1411243140: 2,4825724500e-02 1,1321714967e-01 1,4160857483e-01
1411243200: 7,8739909667e-02 1,1000000000e-01 1,4000000000e-01
1411243260: 3,9648564000e-02 9,3216188000e-02 1,3160809400e-01
1411243320: nan nan nan

To decode the unix time stamps:

$ perl -e 'print scalar localtime(1411243320),"\n"'
Sat Sep 20 22:02:00 2014

If you see all 'nan' for the data, open the debug log file and search for the command that should update the value, for instance:

2014/09/20 21:56:09 [RDDTool] rrdtool update /home/emvee/domotiga/rrd/77.rrd N:0.591

Copy the command and try it at the command line and see if any error is reported.

Maintenance

Remove particular data

Suppose you have a graphs that collected bad data during a certain time and you want to correct it, then:

Domotiga update

When upgrading Domotiga, copy the ~/domotiga/rrd/*.rrd from the old version to the new version before starting the new version otherwise the graphs will be started from scratch and you will not see the history.

If you forgot to do this, it is also possible to merge the old data with the new data, see rrdmerger.pl and Merge multiple RRDs over time

Reading data from rrd files

To calculate things like a rolling 7 and 30 day sum, it can be handy to retrieve the relevant data from the rrd file and calculate these values, see also 7 / 30 day sum for counter.

To retrieve data from a rrd file, use rrdtool fetch:

$ rrdtool fetch 71.rrd AVERAGE --end $(date +%s) 
              buitentemperatuur

1420449900: 1,0000000000e+00
1420450200: 1,0000000000e+00
1420450500: 1,0000000000e+00
1420450800: 9,0111601233e-01
...
1420535100: 4,9889375667e-01
1420535400: 5,0000000000e-01
1420535700: 6,9782271467e-01
1420536000: 7,0000000000e-01
1420536300: nan

$(date +%s) is here giving the current date/time in seconds since epoch/1970.

The data is given with a interval/resolution of 300 seconds which corresponds to five minutes, the interval DomotiGa updates the rrd files. To get data that is averaged over an hour, try:

$ rrdtool fetch 71.rrd AVERAGE --end $(date +%s) --resolution 3600

Advanced

I could not find any rrd merge tool that works for me, but there is a nice Perl module to handle RRD data, RRD::Editor. To install:

$ perl -MCPAN -e shell
cpan[1]> install RRD::Editor
cpan[2]> exit

Test:


$ perl -e 'use RRD::Editor (); my $rrd = RRD::Editor->new(); $rrd->open("serverload.rrd"); print $rrd->info();'
rrd_version = 0003
encoding = native-double (native-double-mixed)
step = 60
last_update = 1411307538
ds[load1].index = 0
ds[load1].type = "GAUGE" 
ds[load1].minimal_heartbeat = 300
ds[load1].min = 0
ds[load1].max = nan
ds[load1].last_ds = "0.28" 
ds[load1].value = 5.3129638800e+00
ds[load1].unknown_sec = 0
...

Rename a data source:

$ perl -d -e "print" 
  DB<1> use RRD::Editor (); $rrd = RRD::Editor->new(); $rrd->open("xxx.rrd");
  DB<2> x $rrd->DS_names();
0  'not_a_good_name'
  DB<5> $rrd->rename_DS('not_a_good_name','perfect_name')
  DB<6> $rrd->save();
  DB<7> exit

Remove all negative values from a .rrd file:
See attached rrd-delete-negative.pl

Problems

Related Resources

Updated by: danfos, Updated over 4 years ago
Access count: 82479 since 2011-08-25