RRDTool

The Round Robin Database Tool (RRDTool) is the open source industry standard, high performance data logging and graphing system for time series data. This module stores values from devices which have graphing enabled and datasources defines.

DomotiGa displays the (simple) graphs in the screen named according to the groups that you enabled for the device, so if you add a graph for a temperature sensor value and make the device part of the Climate group, you will find the graph in the Climate screen.

Databases are stored in domotiga/rrd, graphics in domotiga/rrd/graphs.


Example:

Below you will find instructions to install rrdtool, how to add a graph for a device, how to set up a web-server for a Client-Server configuration and information on debug and maintenance.

For a possible alternative giving interactive charts, see DomotiCharts : Nice interactive charts.

Install

$ sudo apt-get install rrdtool

To check if it is (already) installed:

$ rrdtool
RRDtool 1.4.7  Copyright 1997-2012 by Tobias Oetiker <tobi@oetiker.ch>
               Compiled Aug  2 2012 08:05:17

Adding a graphs for a device value

First enable RRDTool support under Modules -> RRDTool:

For the first configuration it is also wise to enable Debug.

Simple versus extended graphs

Two methods to add graphs are supported:
  1. Simple graphing
  2. Extended graphing

Differences:

Simple graphing Extended graphing
Simple to set up More difficult to set up
One graph per device One graph can based on the data of multiple devices
Fixed colors, scale and line types Lots of configurable items
Enable using the device value window Configuration split over device value window and Graph Editor menu

Simple graphing

When you want to monitor for instance the temperature or the energy logged by a device/sensor:
  • Go to the Devices list (Devices -> List Devices or Alt+D), select the device
  • Right mouse click, Edit
  • Select the Values tab
  • Double-click the line with the Id you like to Log
  • Go to the Graphs tab of this Edit Device window
    • Enable Log RRD data to have the value data logged to an rrd file
    • Select _Enable simple graphing" to have Domotiga create graphs[*]
    • In the DataSource Name field enter a name for the graph. This names will be shown at top of the graph.
    • In the Type box select:
      • GAUGE to graph the actual value, for instance the temperature
      • COUNTER to graph the change for integer values[**] that are always increasing, like the ifInOctets counter in a router. Domotiga logs these values as integers.
      • DERIVE to graph the change of an integer value[**] that can up or down, for instance to see the change of the free space on a harddisk
      • ABSOLUTE to graph the change but the value itself is already giving the change. This can be a counter that is reset when read.
      • For more details on this look in "man rrdcreate"
    • Press Save

[*] If you want to create your own graph size, color and layout go for extended graphing and don't enable simple graphing.
[**] See this ticket, DERIVE and COUNTER operates on integer numbers only.

If you, after pressing Save, get a pop-up saying:

it is because DomotiGa creates one .rrd file per device and DomotiGa is not capable of changing the format of the RRD data or adding a data source. If you do not select Delete then very likely the data collection will not work.

Next you want to click the Groups tab and put the device in a group you want to see the graphs under, for example, Climate, Security, Energy, HVAC.

Extended graphing

To create an extended graph first make sure that you enabled Log RRD Data did put a descriptive name in the DataSource Name field for the values you want to graph. See above, Simple graphing on how to do this.

Next open the Graph Editor using Edit -> Graphs

Enter a name in Name field and click Save.
Then re select the graph just created.
Define Units to use and change size if wanted.
Click Save.

Goto Graph Data tab, and select the device, then select the DataSource from the list, if the device is not there RRD log is not enabled for that device and/or DS.
Enter a Name, click Save next to ID.
Open tab Groups and add the graph to one or more groups to get them displayed on a page, for example Climate, Energy, Security.

Click Save and close window.
You should now see the extended graph on the page Climate, Energy or other group you added it too...
Wait for it to update.

Client-Server configuration

Domotiga creates the RRD database in the domotiga/rrd diorectory, the filename is <id of device>.rrd.

When graphs are needed, a rrdtool command is given to create the graphs. If you run DomotiGaServer as server and one or more DomotiGa's as client, the client sends a JSON-RPC command to let the server create the rrdtool graphs it needs.

GraphsUrl

Domotiga uses http fetch to download the RRD graphs it wants to display. The base URL for this is by default http;//localhost/graphs/ but should be changed if you run in a server-client configuration and if you changed setting so that the graphs are no longer on the default location.

Changing the GraphsUrl can be done in the GUI settings window: Setup -> +G+UI

Web server configuration

If you run in a server-client configuration you need to enable http access to the graphs.

For general instructions on how to set up a web server, see DomotiYii Installation.
Below you find instructions how to enable your webserver to access the graphs using http;//<ip address of server>/graphs/.

You should be able to surf to this address with a webbrowser and you should see a directory with the graph images from the DomotiGaServer.

Once done, change the base URL in GUI Settings, that can be accessed using Setup --> Setup --> GUI. Here you want to change the GraphsURL field. Be sure to restart Domotiga.

Apache

  • Edit the config file called default, located at /etc/apache2/sites-enabled/
    $ sudo vi /etc/apache2/sites-enabled/000-default
    
  • Under line DocumentRoot insert these lines below
    Then under line DocumentRoot insert:
    Alias /graphs/ "/home/user/domotiga/rrd/graphs/" 
        <Directory "/home/user/domotiga/rrd/graphs/">
            Options Indexes MultiViews
            AllowOverride None
            Order deny,allow
            Deny from all
            Allow from 127.0.0.0/255.0.0.0
            Allow from 192.168.178.0/255.255.255.0
        </Directory>
    

    Where /home/user/domotiga points to your domotiga directory.
    • There are some entries pointing to not being able to view graphs in the standard remote GUI, like RRD graphs: can't access serverIP/Graphs
      This might be because you are using Apache 2.4 which has other/additional settings. To check and solve do the following:
      • Just for reading - Apache 2.4 additional information
      • Check version of your Apache version
        • From a linux prompt
          $ HEAD http://yourip
          200 OK
          Connection: close
          Date: Sun, 07 Dec 2014 18:07:12 GMT
          Server: Apache/2.4.7 (Ubuntu)
          Content-Type: text/html;charset=UTF-8
          ...
          
        • From a Windows prompt
          telnet your.webserver.com 80
          

          Then type (carefully, your characters won't be echoed back)
          HEAD / HTTP/1.0
          

          Press return twice and you'll see the server headers like in the linux example.
      • Change the last 2 lines (again, only for Apache 2.4 and up) of the above alias for graphs
                Allow from 192.168.178.0/255.255.255.0
            </Directory>
        

        to ...
                Allow from 192.168.178.0/255.255.255.0
                Require all granted
            </Directory>
        
  • Restart Apache:
    $ sudo /etc/init.d/apache2 restart
    

127.0.0.0 allows to access it via the same machine (localhost) , change 192.168.178.0 to your local subnet to allow other machines to access /graphs.

lighttpd

Add these lines to your config (sudo vi /etc/lighttpd/lighttpd.conf):

dir-listing.activate = "enable" 
alias.url += ( "/graphs/" => "/home/<user>/domotiga/rrd/graphs/" )

nginx

Add these lines to your config (sudo vi /etc/nginx/sites-enabled/default):

    # Also serve the rrd graphs
    location /graphs {
        root  /home/<user>/domotiga/rrd/graphs;
        autoindex on;
        rewrite ^/graphs(/.*\.png)$ $1 break;
        rewrite ^/graphs/?$ / break;
    }

Debug

A good start is to get Server Stats working.

For debugging, enable debug output by enabling it in the RDDTool Settings: Modules --> RRDTool. Check the Debug checkbox.
Next wait till the problem happened and open the debug log file that you can find as ~/domotiga/log/debug-YYYY-MM.log (replace YYYY-MM by the current year-month). Do here a search for [RRDTool] and see if there message reveal anything that is wrong.

If you do not see values logged to a particular rrd you can dump the value of the last 10 minutes using:

$ rrdtool fetch ~/domotiga/rrd/serverload.rrd AVERAGE -s -10min
                          load1               load5              load15

1411242720: 1,6859972267e-01 2,2536244800e-01 1,8178748267e-01
1411242780: 1,3161101550e-01 1,9483304650e-01 1,8000000000e-01
1411242840: 6,2866196000e-02 1,6482482350e-01 1,6321654900e-01
1411242900: 1,8422362667e-01 1,7677795333e-01 1,6838897667e-01
1411242960: 1,0092521383e-01 1,5482889550e-01 1,6160963183e-01
1411243020: 1,3018806600e-01 1,5836467767e-01 1,6000000000e-01
1411243080: 6,4499484500e-02 1,3483316150e-01 1,5161105383e-01
1411243140: 2,4825724500e-02 1,1321714967e-01 1,4160857483e-01
1411243200: 7,8739909667e-02 1,1000000000e-01 1,4000000000e-01
1411243260: 3,9648564000e-02 9,3216188000e-02 1,3160809400e-01
1411243320: nan nan nan

To decode the unix time stamps:

$ perl -e 'print scalar localtime(1411243320),"\n"'
Sat Sep 20 22:02:00 2014

If you see all 'nan' for the data, open the debug log file and search for the command that should update the value, for instance:

2014/09/20 21:56:09 [RDDTool] rrdtool update /home/emvee/domotiga/rrd/77.rrd N:0.591

Copy the command and try it at the command line and see if any error is reported.

Maintenance

Remove particular data

Suppose you have a graphs that collected bad data during a certain time and you want to correct it, then:
  • Use rrdtool dump xxx.rrd > xxx.xml to export the xxx RRD file to XML.
  • Open the XML file, find the incorrect data and correct it.
  • Restore the RRD file using rrdtool restore --force-overwrite xxx.xml xxx.rrd.

Domotiga update

When upgrading Domotiga, copy the ~/domotiga/rrd/*.rrd from the old version to the new version before starting the new version otherwise the graphs will be started from scratch and you will not see the history.

If you forgot to do this, it is also possible to merge the old data with the new data, see rrdmerger.pl and Merge multiple RRDs over time

Reading data from rrd files

To calculate things like a rolling 7 and 30 day sum, it can be handy to retrieve the relevant data from the rrd file and calculate these values, see also 7 / 30 day sum for counter.

To retrieve data from a rrd file, use rrdtool fetch:

$ rrdtool fetch 71.rrd AVERAGE --end $(date +%s) 
              buitentemperatuur

1420449900: 1,0000000000e+00
1420450200: 1,0000000000e+00
1420450500: 1,0000000000e+00
1420450800: 9,0111601233e-01
...
1420535100: 4,9889375667e-01
1420535400: 5,0000000000e-01
1420535700: 6,9782271467e-01
1420536000: 7,0000000000e-01
1420536300: nan

$(date +%s) is here giving the current date/time in seconds since epoch/1970.

The data is given with a interval/resolution of 300 seconds which corresponds to five minutes, the interval DomotiGa updates the rrd files. To get data that is averaged over an hour, try:

$ rrdtool fetch 71.rrd AVERAGE --end $(date +%s) --resolution 3600

Advanced

I could not find any rrd merge tool that works for me, but there is a nice Perl module to handle RRD data, RRD::Editor. To install:

$ perl -MCPAN -e shell
cpan[1]> install RRD::Editor
cpan[2]> exit

Test:


$ perl -e 'use RRD::Editor (); my $rrd = RRD::Editor->new(); $rrd->open("serverload.rrd"); print $rrd->info();'
rrd_version = 0003
encoding = native-double (native-double-mixed)
step = 60
last_update = 1411307538
ds[load1].index = 0
ds[load1].type = "GAUGE" 
ds[load1].minimal_heartbeat = 300
ds[load1].min = 0
ds[load1].max = nan
ds[load1].last_ds = "0.28" 
ds[load1].value = 5.3129638800e+00
ds[load1].unknown_sec = 0
...

Rename a data source:

$ perl -d -e "print" 
  DB<1> use RRD::Editor (); $rrd = RRD::Editor->new(); $rrd->open("xxx.rrd");
  DB<2> x $rrd->DS_names();
0  'not_a_good_name'
  DB<5> $rrd->rename_DS('not_a_good_name','perfect_name')
  DB<6> $rrd->save();
  DB<7> exit

Remove all negative values from a .rrd file:
See attached rrd-delete-negative.pl

Problems

  • It looks like Extended graphing is not working in Sep 2014, see #480 "not possible anymore to create new graphs".
  • Logging two values for one devices seems not to fully working. The rddtool update command is pushing two values but that triggers an error:
    $ rrdtool update /home/<user>/domotiga/rrd/70.rrd N:25.5:58
    ERROR: /home/<user>/domotiga/rrd/70.rrd: found extra data on update argument: 58
    

Related Resources

    Updated by: danfos, Updated over 2 years ago
    Access count: 74075 since 2011-08-25

    rrdtool-3dlogo.png (2.9 KB) rdnzl, 08/23/2011 02:20 PM

    System_Load_graph.png (20.2 KB) danfos, 07/20/2014 09:48 AM

    Value_Graphs.png (30 KB) danfos, 07/26/2014 12:56 PM

    Server_performance.png (26.4 KB) danfos, 07/27/2014 01:38 PM

    RRDTool_setup.png (20 KB) danfos, 08/02/2014 02:29 PM

    Graph_Editor.png (40.8 KB) danfos, 09/27/2014 04:42 PM

    RRD_has_to_be_deleted.png (30.2 KB) danfos, 09/27/2014 05:01 PM

    rrd-delete-negative.pl Magnifier (821 Bytes) danfos, 10/02/2014 09:57 PM

    Also available in: PDF HTML TXT