Category Archives: data logging

Data Logging with the Raspberry Pi pt. 2

Posted on by

Background

In the first article I described the basic system that recorded temperature and humidity, saved it to a local file and also sent the file to dropbox. This simple system worked well but had some limitations:

  • the data file will grow very large and repeated sending to Dropbox becomes very slow and potentially less reliable.
  • data is only available as a comma delimited file which has to be imported to some other app to be useful
  • the raspberry pi zero, whilst an excellent little device, does not lend itself to being a web server that is easily and securely accessed from outside of the local area network (LAN)

In this article I address these issues and add some bells and whistles, e.g. optional notifications by sms and an optional relay controlled heater interface .

Requirements:

  • A web server somewhere that you can use to run php scripts. It does not have to be  your own domain. There are many free offerings that allow use as sub-domains for free, e.g. https://www.000webhost.com/ (not tested and just a suggestion!)
  • Internet connection that allows access to the web server to edit the php scripts
  • For all the functions to work, a Dropbox account
  • Email address(es) if you want to receive notifications
  • The raspberry pi hardware as described in part 1.
  • An account with an sms provider if you want sms alarms sent e.g. smsbroadcast.co.uk. We use this provider and have found them to be relaible, and relatively cheap.
  • Optionally a relay interace for the pi ifyou want to control a heater

No modifications to the original hardware are necessary but SSH or SFTP access to the pi using something like Putty or FileZilla is necessary. We use Putty on a Windows PC with the built in Nano editor on the pi. This article also assumes that you have successfully completed the steps in part1.

Step 1

Modify the python code in file pithlog.py on the pi as follows:

#!/usr/bin/python3
#import libraries
import Adafruit_DHT,datetime,dropbox,urllib.request,urllib.parse
import RPi.GPIO as GPIO
import ssl,os
GPIO.setwarnings(False)

#set up I/O
sensor=Adafruit_DHT.DHT22
#relay control pin 27
relay=27
GPIO.setmode(GPIO.BCM)
GPIO.setup(relay,GPIO.OUT)
#set the pin for DHT22
gpio=17

#start reading
humidity, temperature=Adafruit_DHT.read_retry(sensor,gpio)

#we have a heater relay set to switch off at 20C, on at 18C
if temperature > 20:
GPIO.output(relay,GPIO.LOW)
if temperature < 18:
GPIO.output(relay,GPIO.HIGH)

#this block is optional and can be commented out if no sms alarms
#if the temperature is extremely high (>40C here), send an sms alarm message
#note that this will happen at EVERY reading interval for now
if temperature > 40 :
url=”https://api.smsbroadcast.co.uk/api-adv.php?username=YOURUSERNAME&password=YOURPASSWORD?&to=YOURPHONENUMBER&from=YOURSOURCE&message=YOURMESSAGE%20temperature%20alert%20>=40C&ref=00003&maxsplit=5&delay=0″
requ=urllib.request.Request(url)
response=urllib.request.urlopen(requ)
responseData=response.read()
#print (responseData)

#log the reading
logfile=”/home/pi/tdata1.csv”

#get the time of the reading
timestamp=datetime.datetime.now()

#if the file is over 500000 bytes, start a new one
newfile=str(timestamp.strftime(“%Y%m%d%H%Mtdata.csv”))
timestamp=str(timestamp.strftime(“%Y,%m,%d,%H,%M”))
filelen=os.path.getsize(logfile)
if (filelen>500000):
os.rename(logfile,newfile)

#optionally send the data to a server
#comment out this block if not used
url=”http://www.YOURDOMAINHERE.co.uk/mondata.php”
temp=str(“{0:0.1f}”.format(temperature))
hum=str(“{0:0.1f}”.format(humidity))
tdata=str(timestamp+”,”+temp+”,”+hum+”\n”)
#print (tdata)
readings={‘tstamp’ : timestamp, ‘temperature’ : temp, ‘humidity’ : hum}
data=urllib.parse.urlencode(readings)
data=data.encode(‘utf-8’)
requ=urllib.request.Request(url,data)
response=urllib.request.urlopen(requ)
responseData=response.read()
#print (responseData)

#save the data locally in a file
df=open(logfile,”a”)
df.write(tdata+”\n”)
df.close()

The above code will run according to your cron schedule as set up in part 1. Note that the dropbox backup is now added separately once per day as detailed below. All the files in /home/pi are saved.

Step 2

Install Dropbox-Uploader from Git by following the very clear “Getting Started” instructions in the README.MD file.
Using your editor creat a file called backup.sh and add the following:

#!/usr/bin/env bash
#
# backup files (and only files, not folders) in /home/pi to dropbox
# designed to be run from cron or when needed every now and again
# skips files already present and unchanged
#
# Uses Dropbox-Uploader by Andrea Fabrizi avaialable on Git
#
# backup pi home files
/home/pi/Dropbox-Uploader/dropbox_uploader.sh upload /home/pi/*.* /
#

Step 3

Now edit your crontab file (crontab -e) to add the line:

57  1  *  *  * /home/pi/backup.sh
If you are uploading the readings to a web server, you must now add a script called “mondata.php” in the root folder of the web server:
<?php
//uncomment following line if you want to see contents of POST when debugging
//print “$_POST[tstamp] $_POST[temperature] $_POST[humidity]”;
$dline=$_POST[tstamp].”,”.$_POST[‘temperature’].”,”. $_POST[‘humidity’].”\n”;
$dfile=fopen(“/home/johno/data/tdata.csv”,”a”);
fwrite($dfile,$dline);
fclose($dfile);
//print “OK”;
?>

This will create a csv file “tdata.csv” that will record each reading set as it is taken.

We now have a system that will allow us to control a relay according to temperature, optionally send alarm messages via sms as well as potentially being ready to display the data remotely. Th enext post will show how to display the logged data in a convenient graphical format.

Data Logging with the Raspberry Pi pt. 1

Posted on by

Background

This article was inspired by an excellent article in Linux Format (LXF 248, February 2019, P. 58) entitled “Remote sensors” or “Sharing data from remote sensors”. Essentially that mini-project used the Raspberry Pi in conjunction with a DHT11 temperature and humidity sensor to log data to a file stored on Dropbox. The project as described worked well but in my opinion had a few limitations:

  • As time went by, the file size uploaded to Dropbox increased and would eventually become rather slow and cumbersome. This also means that there is more likelihood of data loss caused by poor connections.
  • The project only logged temperature and ignored the humidity reading – a waste of the capabilities of the DHT11. Humidity data could also be important if you are looking to implement some form of weather station or monitor in a humidity sensitive area such as a greenhouse. If you only want temperature readings it is easier (and cheaper) to use a DS180B20 1-wire device.
  • No method of displaying the gathered data was presented except by loading the file into a Libre Office calc sheet each time. Again, there is nothing wrong with this, but some form of web based display might be more useful as it is accessible remotely.
  • The data was stored as a simple csv file. This is not unusual but recording to a full blown SQL database would make the system much more flexible.

With the above in mind, I decided to develop a system of my own that:

  • Minimises the uploading of data
  • Saves the data locally with dropbox backup as well as in a remote SQL database
  • Logs humidity as well as temperature
  • Adds alarm levels when certain temperature limits are exceeded.
  • Makes provision for controlling some form of heater switching.
  • Sends alarm emails and/or SMS messages if temperatures are out of preset ranges.

The output from my revised and extended system can be seen here (1 day’s data), or here (long term data, takes a while to load).

Hardware

The diagram below shows the basic datalogger connection diagram.

Datalogger

Datalogger basic connections.

The original article used a DHT11 sensor. My project uses a DHT22 4 pin sensor which is slightly more accurate and worked with the exact same code as the original.

DHT22 Pin 1 connects to Pi pin 1 (3v3)
DHT22 Pin 2 connects to Pi pin 11 (GPIO17)
DHT22 Pin 4 connects to Pi pin 39 (Ground/0v)

The (Python) Code for the Basic System

All the code in this project assumes that you are using a recent version of Raspbian. To install the necessary Python libraries, first make sure that you are up to date:
$ sudo apt-get update
$ sudo apt-get upgrade
Then install:
$ sudo pip3 install Adafruit DHT
$ sudo pip3 install dropbox
In order to use the dropbox facility it is necessary to set up a dropbox account, set up an app and get an access token. This can be obtained by going to your dropbox account and selecting developers –  www.dropbox.com/developers – the “Create your App” using the dropbox API, for access use “App Folder” and name your app, we will use “Pi log”. Create the app and go to the final screen where you will find you can generate an access token of the form “fU-2TIyyyfUBBBBBBBBCme2mCZpuu5GZZAVRm6soHH6fsYJ1CG6uyrvm-orxxe8Z”, make a note of this but make sure that you keep it secret – it is the key that opens the dropbox!

If you want to send email alert messages you need to install ssmtp and mailutils as well:
$ sudo apt-get install ssmtp
$ sudo apt-get install mailutils
You will need to edit the /etc/ssmtp/ssmtp.conf file to enter your own email details. An excellent ‘how to’ is here: https://raspberry-projects.com/pi/software_utilities/email/ssmtp-to-send-emails. Adjust the settings according to your email address, hostname etc. and test that it works using the command line as in the how to.

At this point you should test that all is well. Open a new file with your favourite python3 editor (I use idle3 as it allows you to test code as you develop it) and enter the following code:

#!/usr/bin/python3
import Adafruit_DHT,datetime,dropbox
#next import line will be used later
import urllib.request,urllib.parse
#note change of sensor type
sensor=Adafruit_DHT.DHT22
#set gpio pin for sensor
gpio=17
#read the values - note both temperature and humidity
humidity, temperature=Adafruit_DHT.read_retry(sensor,gpio)
timestamp=datetime.datetime.now()
#fix timestamp to something useful
timestamp=str(timestamp.strftime("%Y-%m-%d %H:%M"))
#fix the values read to truncate 1 decimal place and make into string
temp=str("{0:0.1f}".format(temperature))
hum=str("{0:0.1f}".format(humidity))
#make a log entry string
thdata=str(timestamp+","+temp+","+hum+"\n")
print (thdata)
#uncomment the next section if you want to "POST" values
#to a remote site
#readings={'tstamp' : timestamp, 'temperature' : temp, 'humidity' : hum}
#data=urllib.parse.urlencode(readings)
#data=data.encode('utf-8')
#requ=urllib.request.Request(url,data)
#response=urllib.request.urlopen(requ)
#responseData=response.read()
#print (responseData)
#end of POST to remote site
#Save the data to a local file
df=open("/home/pi//thdata.csv","a")
df.write(thdata+"\n")
df.close()
#now send to dropbox if you want to, substitute your token!
token="fU-2XXXXXXXXXXXXXXXXXXXXXZZZZZZZZZZZZZZZZZZ"
dbx=dropbox.Dropbox(token)
with open("/home/pi/thdata.csv","rb") as data:
    dbx.files_upload(data.read(),'/thdata.csv',mute=True,
    mode=dropbox.files.WriteMode.overwrite)

To test the code, save the file as pithlog.py and open a terminal window and make the file executable:
$ chmod +x pithlog.py

and run the script:
$ pithlog.py

You should see one line of output with the date, time and readings. Check your dropbox folder online to make sure that the data has arrived.
The final step in part 1 is to make the script run at regular intervals using cron. At a terminal prompt edit crontab:
$ crontab -e

and at the end of the file add the line:
*/5 * * * * /home/pi/pithlog.py

Make sure that there is a newline at the end of the file. You may also optionally add this line before the one above if you want email notifications if there are errors, useful if dropbox does not work:
MAILTO=youremailaddress@here

If you do not want to see any normal output, comment out any print statements used for debugging.
If you have successfully got this far Congratulations!

In the next part I will show how to POST the data to a remote site and display it in a chart. Coming soon.