Solution to producer and consumer problem in python - part 2

The demo program

In the demo program, the task do some random work.

Sample - The Task
 class Task:
     def __init__(self,id):
         self.id = id
     def run(self):
         self.log('Started')
         self.doWork()
         self.log('Ended')
     def log(self,msg,*args):
         logLock.acquire(True)
         now = datetime.datetime.now()
         sys.stdout.write( '{1:>02}:{2:>02}:{3:>02} #{4:>04} | {0}\n'.format( msg.format(*args), now.hour, now.minute, now.second , self.id))
         logLock.release()
     def doWork(self):
         sum = 0
         for xx in range(0 , random.randint(1,100000) ):
             sum += xx
  • It prints a message at the start of the task and at its end. It will also print its unique id.
  • It do some random work - summing numbers from 1 to a random number.
  • It will also print the number of tasks left in the queue after the task has completed.
Sample - TaskWorkerPool
 twp = TaskWorkerPool(10)   # <== A1
 for ii in range(500):      # <== A2
     twp.add( Task(ii) )

 twp.wait()                 # <== A3
 print "Done with all tasks :)"

The producer thread is the main thread.

  • In A1 we define a TaskWorkerPool with 10 worker threads.
  • In A2 we add some tasks. Please note that the worker threads start executing the tasks while we are adding them.
  • In A3 we wait until all the tasks completed.

Running the program will produce the output similar to next one:

Sample - output
 10:00:00 #0000 | Started
 10:00:00 #0001 | Started
 ...
 10:00:00 #0009 | Started
 10:00:00 #0000 | Ended
 10:00:00 #0000 | Left 41 Tasks
 10:00:00 #0010 | Started
 10:00:00 #0010 | Ended
 ...
 10:00:09 #0493 | Left 3 Tasks
 10:00:09 #0491 | Left 3 Tasks
 10:00:09 #0486 | Left 0 Tasks
 ...
 10:00:09 #0499 | Ended
 10:00:09 #0499 | Left 0 Tasks
 Done with all tasks :)