windows监控本地文件夹变化，并将图像发送给多台ubuntu并行进行图像处理

背景

该方式是在离线环境下搭建的小型图像并行处理系统，主要是因为离线环境下去搭建消息队列等太过于复杂

• windows 7监控本地文件夹是否存入新的图像，无论是拷贝还是FTP传入该文件夹的新图像都将被监测
• 当出现新图像传入之后，win程序将图像间隔拷贝至本地N个文件夹（N的=ubuntu服务器的个数），每个文件夹都共享并挂载到一个ubuntu服务器
• ubuntu服务器监测本地一个文件夹变化，并将新增加的图片传入数组
• ubuntu上基于深度学习的图像处理线程判断数组是否有新的图像并进行处理，最终将处理后图片传送至ubuntu挂载的windows结果文件夹

一些依赖（离线配置用）

系统：ubuntu16.04.1、win7
其他小离线依赖：
链接
提取码：upue

ubuntu挂载windows共享文件夹

这个可以参照我之前的博客进行配置
需要共享的文件夹主要是N+1个，这里例如有两个ubuntu服务器，那么需要共享的文件夹是：win_ubuntu1, win_ubuntu2, win_output，这里需要解释的是，win_ubuntu1和win_ubuntu2都是ubuntu系统的文件夹然后挂载到win上的，其实步骤是ubuntu下samba共享文件夹，win可以共享访问，然后宣州共享文件夹将这两个共享文件夹设置在win上的盘符

windows端设置

首先是建立一个文件夹用于存储图像，并且用代码监测该文件夹变化，如果有新图像进入，则拷贝到win_ubuntu1, win_ubuntu2
（注意：这里用拷贝的方法而没有考虑用FTP直接发送到ubuntu，主要是因为FTP比拷贝慢的多）

import os
import sys
import time
import logging
import shutil import copyfile
from threading import Thread
from watchdog.observers import Observer
from watchdog.events import *

create_event = []
modify_event = []
index = 0
win_dirs = []
exts = ['png']

def multiprocess(f):
    def wrapper(*args, **kwargs):
    	thr = Thread(target=f, args=args, kwargs=kwargs)
    	thr.start()
    return wrapper
   
@multiprocess
def copy_file(input_file, out_file):
	copyfile(input_file, out_file)
	print ('copy_file: %s' % out_file)

class TestHandler(FileSystemEventHandler):
	def on_modified(self, event):
		global index
		if event.src_path in modify_event:
			ext = event.src_path.split('.')[-1]
			if event.src_path in create_event and ext in exts:
				create_event.remove(event.src_path)
				modify_event.remove(event.src_path)
				input_file = event.src_path
				out_file = os.path.join(win_dirs[index].strip('\n'), input_file.split('\\')[-1])
				index += 1
				if index == len(win_dirs):
					index = 0
			else:
				modify_event.remove(event.src_path)
		else:
			modify_event.append(event.src_path)
	
	def on_created(self, event):
		create_event.append(event.src_path)

	#def on_deleted(self, event):
	#def on_moved(self, event):

if __name__ == '__main__':
	f = open("config_path.txt", "r")
	win_dirs = f.readlines()
	f.close()
	logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(message)s', datefmt='%Y-%m-%d %H:%M:%S')

	path = sys.argv[1] if len(sys.argv) > 1 else '.'
	event_handler = TestHandler()
	observer = Observer()
	observer.schedule(event_handler, path, recursive=True)
	observer.start()
	try:
		while True:
			time.sleep(1)
	except KeyboardInterrupt:
		observer.stop()
	observer.join()

然后里面涉及到的config_path.txt内容就是ubuntu共享文件夹在win上的盘符地址，如：

Z:\
Y:\

ubuntu端

运行时候一定要用超级权限
sudo env “PATH=$PATH” python xx.py

import os
import datetime
import pyinotify
import logging
import threading

# 深度学习网络为例
from XX_model import Detector

paths = []
detector = Detector(各种参数)

def DeepLearningPipline(img_name, ext='.png', input_dir='/win_ubuntu1', output_dir='/home/win_share_output/'):
	name, _ = os.path.splitext(img_name)
	output_name = name + ext
	detector.detect_and_draw(各种参数)
	os.remove(os.path.join(input_dir, img_name))

def dl_process():
	while 1:
		try:
			DeepLearningPipline(img_name=paths[0].split('/')[-1])
		except:
			paths.append(paths[0])
			paths.pop(0)

class MyEventHandler(pyinotify.ProcessEvent):
	# logging.basicConfig(level=logging.INFO, filename='./monitor.log')
	# logging.info('Starting...')

	# def process_IN_ACCESS(self, event):
		# print ('ACCESS event: %s' % event.pathname)
		
	# def process_IN_ATTRIB(self, event):
		# print ('ATTRIB event: %s' % event.pathname)

	# def process_IN_CLOSE_NOWRITE(self, event):
		# print ('CLOSE_NOWRITE event: %s' % event.pathname)
		
	def process_IN_CLOSE_WRITE(self, event):
		# print ('CLOSE_WRITE event: %s' % event.pathname)
		if event.pathname in paths:
			pass
		else:
			paths.append(event.pathname)

	# def process_IN_CREATE(self, event):
		# print ('CREATE event: %s' % event.pathname)

	# def process_IN_DELETE(self, event):
		# print ('DELETE event: %s' % event.pathname)

	# def process_IN_MODIFY(self, event):
		# print ('MODIFY event: %s' % event.pathname)

	# def process_IN_OPEN(self, event):
		# print ('OPEN event: %s' % event.pathname)

def monitor():
	wm = pynotify.WatchManager()
	wm.add_watch('/win_ubuntu1', pynotify.ALL_EVENTS, rec=True)
	eh = MyEventHandler()
	notifier = pynotify.Notifier(wm, eh)
	notifier.loop()

if __name__ == '__main__':
	t_monitor_dir = threading.Thread(target=monitor)
	t_dl_process = threading.Thread(target=dl_process)

	t_monitor_dir.start()
	t_dl_process.start()