Python + inotify = Pyinotify [ how to watch folders for file activity ]
Sometimes it just might be handy to be able to watch a folder on a hard disk for changes. For example: A client app might drop small files on a shared folder. A server app might be watching the folder for just such an event. Once the file is created, the server will kick into action and perform whatever tasks are required.
This all comes from my CD burning application. I am currently thinking that the client apps will drop small xml files containing information about what to burn onto a folder the webserver has access to and the cdburner service will be watching...
The linux kernel provides inotify. This from wikipedia:
This all comes from my CD burning application. I am currently thinking that the client apps will drop small xml files containing information about what to burn onto a folder the webserver has access to and the cdburner service will be watching...
The linux kernel provides inotify. This from wikipedia:
notify is a Linux kernel subsystem that provides file system event notification. It was written by John McCutchan with help from Robert Love and later Amy Griffis to replace dnotify. It was included in the mainline kernel from release 2.6.13 (2005-06-18), and could be compiled into 2.6.12 and possibly earlier releases by use of a patch. Its function is essentially an extension to filesystems to notice changes to the filesystem, and report those changes to applications.Pyinotify is a python module that exposes the inotify api in python. This from http://pyinotify.sourceforge.net/:
pyinotify is a Python module for watching filesystems changes. pyinotify can be used for various kind of fs monitoring. pyinotify relies on a recent Linux Kernel feature (merged in kernel 2.6.13) called inotify. inotify is an event-driven notifier, its notifications are exported from kernel space to user space through three system calls. pyinotify binds these system calls and provides an implementation on top of them offering a generic and abstract way to use inotify from Python. Pyinotify doesn't requires much detailed knowledge of inotify. Moreover, it only needs few statements for initializing, watching, handling (optionnaly trough a new separate thread), and processing events notifications through subclassing. The only things to know is the path of items to watch, the kind of events to monitor and the actions to execute on these notifications. Note: pyinotify requires Python 2.3 and above, and Linux 2.6.13 at least.I went ahead and gave it a go. I did find though, that on my fedora system ( fedora 9 x86_64 ) that the tutorial on the above wiki didn't quite work. Here is the tutorial code for the non threaded example that does work on my system.
Unfortunately inotify is a Linux Kernel technology that is not currently available on windows. I guess Windows has some other kind of API for filesystem event monitoring but if you are like me and want to keep things simple with python then I am afraid inotify on windows is not possible. If you feel that you would like to give this stuff a go then I suggest getting yourself a LiveCD of one of the Linux Distros. They can be bought here.
#!/usr/bin/python
import os
import pyinotify
wm = pyinotify.WatchManager()
mask = pyinotify.IN_DELETE | pyinotify.IN_CREATE
class PTmp(pyinotify.ProcessEvent):
def process_IN_CREATE(self, event):
print "Create: %s " % os.path.join(event.path, event.name)
def process_IN_DELETE(self, event):
print "Delete: %s " % os.path.join(event.path, event.name)
notifier = pyinotify.Notifier(wm, PTmp())
wdd = wm.add_watch('/home/dave/projects', mask, rec=True)
while True:
try:
notifier.process_events()
if notifier.check_events():
notifier.read_events()
except KeyboardInterrupt:
notifier.stop()
break
Comments
Is it possible to code this kind of thing so it is constantly running looking at a folder, and to do whatever when a file is dropped into it?
Thanks - Charlie
You can perform whatever functions you like in the IN_CREATE folder.
A specific application example would be:
An ftp service is setup where clients upload files to the ftp location and some automated process is required. This process then kicks off whenever a file is created on the ftp folder being watched. Maybe the incomming file has to be parsed and the data inserted into a database.
I suggest you try it out. Its pretty powerful stuff. Python provides a nice interface to this useful linux tool.
I'm not sure of the version differences, but for me (using pyinotify 0.7.1), the event codes were not in the pyinotify namespace. I had to go one level further into EventsCodes. Where you have pyinotify.IN_CREATE, I needed pyinotify.EventsCodes.IN_CREATE
***
Installed: python-inotify.noarch 0:0.8.0-3.r.fc9
***
Then I did a test:
***
Python 2.5.1 (r251:54863, Jun 15 2008, 18:24:56)
[GCC 4.3.0 20080428 (Red Hat 4.3.0-8)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyinotify
INFO: Maybe it could speed-up a little bit if you had psyco installed (not required).
>>> mask = pyinotify.IN_CREATE
>>> print mask
256
>>> mask = pyinotify.IN_DELETE
>>> print mask
512
***
So it is probably an issue with versions or simply the distribution.
But thanks anyway for the heads up.
im hoping you could help me.
i got this working well...
but i need it to on the increate step
run another python script in a folder.
eg the watched folder.
i have the python script that does what i need perfectly. and can execute it from cmd line but it would be better if the script could run auto from file change...
i really hope you can help...
thankyou - Adam
First of all the siplest way to deal with calling another script is to import and run one of its functions.
Here is my example:
notifier.py ( handles create and delete events. Calls main() on otherscript.py on create )
import os
import pyinotify
import otherscript
wm = pyinotify.WatchManager()
mask = pyinotify.IN_CREATE | pyinotify.IN_DELETE
class PTmp(pyinotify.ProcessEvent):
def process_IN_CREATE(self, event):
print "Create: %s" % os.path.join(event.path, event.name)
print "Executing otherscript.py"
otherscript.main(event.name)
def process_IN_DELETE(self, event):
print "Delete: %s" % os.path.join(event.path, event.name)
notifier = pyinotify.Notifier(wm, PTmp() )
wdd = wm.add_watch('/home/davidl/test', mask, rec = True)
while True:
try:
notifier.process_events()
if notifier.check_events():
notifier.read_events()
except KeyboardInterrupt:
notifier.stop()
break
otherscript.py ( defines a main() function that takes the filename argument and prints some text to the screen )
#!/usr/bin/python
def main(filename):
print "Other Python Script Executed!"
print "Otherscript received filename: %s " % filename
To use:
1. Create a folder called test
2. Install both of these scripts in the test folder
3. Open a shell and run "python notifier.py"
4. create a new file in the test folder.
5. watch your notifier script and otherscript do the job.
There is a way to call another application using the popen or popen2. Look in this blog for a script called pyVerify for examples.
The lines: notifier = pyinotify.Notifier... and wdd = wm.add.... are on new lines outside of the PTmp class.
thank you ALOT
Adam
once i got it running without indentation errors it gave a module main error and just sat there...
unfortunately im only learning, so
i googled the error and really got no answers as to why it didnt go...
again thankyou for trying...
Adam
I have tried to get the ThreadedNotifier to work the way you would like. Specifically, I have tried to get it to stop the thread on only the IN_CREATE event. In any case my attempts were largely futile. I suppose this would be because of a lack of thorough understanding about threads on my part.
I have learned the following:
The ThreadedNotifier inherits from threading.Thread so all of the threading.Thread methods should be available.
The ThreadedNotifier also inherits from Notifier but overrides the loop function. This is important because the loop() function in the Notifier Class provides a small callback handler.
Here is a code snippet to prove how callbacks work. The callback will be executed every time there is an event and will be passed the Notifier object.
Unfortunately the NotifierObject does not know exactly which event was fired by the kernel. At about this time, I started to loose interest in the problem. Maybe if I could learn more about the application... :)
#!/usr/bin/env python
import pyinotify
class PTmp(pyinotify.ProcessEvent):
"""Event handler"""
def process_IN_CREATE(self, event):
"""Capture events for the CREATE event."""
print "Create: ", event.pathname
def process_default(self,event):
"""Default handler for ALLOTHER events."""
print "Default handler hit: ", event.pathname
def myCallBack(notifierObj):
"""Call back function to show how it works."""
print "Callback received."
wm = pyinotify.WatchManager()
mask = pyinotify.IN_CREATE | pyinotify.IN_DELETE
notifier = pyinotify.Notifier(wm, PTmp())
path = '/home/dave'
wm.add_watch(path, mask, rec=True)
notifier.loop(callback=myCallBack)
http://tgolden.sc.sabren.com/python/win32_how_do_i/watch_directory_for_changes.html
http://tech-artists.org/wiki/Python_Recipes#Watch_a_Directory_for_Changes
Empty set (0.00 sec)
So let me see if I understand the question:
1. You want to copy files to a flash drive
2. and then have something tell you what you just copied?
3. And you want to be able to do this WITHOUT inserting the flash drive?
I will try to respond. (Not to your private email - that would deny all the other folks that seem to be following this post any chance at reading it)
1. Copy the files.
2. Remember what you copied.
3. Huh? How did you manage 1 and 2 without having FIRST inserted the flash drive?
You could write a python script that prompts for the folder to monitor and somehow checks to see if the flash drive is still mounted otherwise quit.
Then as files are copied to them, your py-inotify script will be watching the write events on the flash drive's filesystem. It could then write to a log file.
In the end when the flash drive is removed, it could email the contents of the log just written to you.
I can only assume you are trying to protect against staff removing sensitive data from your systems. How about an old fashioned security policy which all your staff have to sign...