Item 42. Variable Initialization is it?

本文解析了C++中四种不同变量初始化的方式:默认初始化、直接初始化、复制初始化及一个易混淆的函数声明案例。通过实例对比了不同初始化方式的区别,并提供了一条准则来帮助选择最佳实践。

This first problem highlights the importance of understanding what you write. Here we have four simple lines of code, no two of which mean the same thing, even though the syntax varies only slightly.

What is the difference, if any, between the following? (T stands for any class type.)

T t; 
T t();
T t(u);
T t = u;
 
I l@ve RuBoardPrevious Section Next Section

Solution

graphics/bulb_icon.gif

This puzzle demonstrates the difference between three kinds of initialization: default initialization, direct initialization, and copy initialization. It also contains one red herring that isn't initialization at all. Let's consider the cases one by one.

T t; 

This is default initialization. This code declares a variable named t, of type T, which is initialized using the default constructor T::T().

T t(); 

A red herring. At first glance, it may look like just another variable declaration. In reality, it's a function declaration for a function named t that takes no parameters and returns a T object by value. (If you can't see this at first, consider that the above code is no different from writing something like int f(); which is clearly a function declaration.)

Some people suggest writing "auto T t();" in an attempt to use the auto storage class to show that, yes, they really want a default-constructed variable named t of type T. Allow me license for a small rant here. For one thing, that won't work on a standard-conforming compiler; the compiler will still parse it as a function declaration, and then reject it because you can't specify an auto storage class for a return value. For another thing, even if it did work, it would be wrong-headed, because there's already a simpler way to do what's wanted. If you want a default-constructed variable t of type T, then just write "T t;" and quit trying to confuse the poor maintenance programmers with unnecessary subtlety. Always prefer simple solutions to cute solutions. Never write code that's any more subtle than necessary.

T t(u); 

Assuming u is not the name of a type, this is direct initialization. The variable t is initialized directly from the value of u by calling T::T(u). (If u is a type name, this is a declaration even if there is also a variable named u in scope; see above.)

T t = u; 

This is copy initialization. The variable t is always initialized using T's copy constructor, possibly after calling another function.

Common Mistake

graphics/commonmistake_icon.gif

This is always initialization; it is never assignment, and so it never calls T::operator=(). Yes, I know there's an "=" character in there, but don't let that throw you. That's just a syntax holdover from C, not an assignment operation.


Here are the semantics:

  • If u is of type T, this is the same as writing "T t(u);" and just calls T's copy constructor.

  • If u is of some other type, then this has the meaning "T t( T(u) );"梩hat is, u is first converted to a temporary T object, and then t is copy-constructed from that. Note, however, that in this case the compiler is allowed to optimize away the "extra" copy and convert it to the direct-initialization form (that is, make it the same as "T t(u);"). If your compiler does this, the copy constructor must still be accessible. But if the copy constructor has side effects, then you may not get the results you expect, because the copy constructor's side effects may or may not happen, depending on whether the compiler performs the optimization or not.

Guideline

graphics/guideline_icon.gif

Prefer using the form "T t(u);" instead of "T t = u;" where possible. The former usually works wherever the latter works, and has other advantages梖or example, it can take multiple parameters.


  •  
   
I l@ve RuBoardPrevious Section Next Section
 
Quickstart Note The data files used in the quickstart guide are updated from time to time, which means that the adjusted close changes and with it the close (and the other components). That means that the actual output may be different to what was put in the documentation at the time of writing. Using the platform Let’s run through a series of examples (from almost an empty one to a fully fledged strategy) but not without before roughly explaining 2 basic concepts when working with backtrader Lines Data Feeds, Indicators and Strategies have lines. A line is a succession of points that when joined together form this line. When talking about the markets, a Data Feed has usually the following set of points per day: Open, High, Low, Close, Volume, OpenInterest The series of “Open”s along time is a Line. And therefore a Data Feed has usually 6 lines. If we also consider “DateTime” (which is the actual reference for a single point), we could count 7 lines. Index 0 Approach When accessing the values in a line, the current value is accessed with index: 0 And the “last” output value is accessed with -1. This in line with Python conventions for iterables (and a line can be iterated and is therefore an iterable) where index -1 is used to access the “last” item of the iterable/array. In our case is the last output value what’s getting accessed. As such and being index 0 right after -1, it is used to access the current moment in line. With that in mind and if we imagine a Strategy featuring a Simple Moving average created during initialization: self.sma = SimpleMovingAverage(.....) The easiest and simplest way to access the current value of this moving average: av = self.sma[0] There is no need to know how many bars/minutes/days/months have been processed, because “0” uniquely identifies the current instant. Following pythonic tradition, the “last” output value is accessed using -1: previous_value = self.sma[-1] Of course earlier output values can be accessed with -2, -3, … From 0 to 100: the samples Basic Setup Let’s get running. from __future__ import (absolute_import, division, print_function, unicode_literals) import backtrader as bt if __name__ == '__main__': cerebro = bt.Cerebro() print('Starting Portfolio Value: %.2f' % cerebro.broker.getvalue()) cerebro.run() print('Final Portfolio Value: %.2f' % cerebro.broker.getvalue()) After the execution the output is: Starting Portfolio Value: 10000.00 Final Portfolio Value: 10000.00 In this example: backtrader was imported The Cerebro engine was instantiated The resulting cerebro instance was told to run (loop over data) And the resulting outcome was printed out Although it doesn’t seem much, let’s point out something explicitly shown: The Cerebro engine has created a broker instance in the background The instance already has some cash to start with This behind the scenes broker instantiation is a constant trait in the platform to simplify the life of the user. If no broker is set by the user, a default one is put in place. And 10K monetary units is a usual value with some brokers to begin with. Setting the Cash In the world of finance, for sure only “losers” start with 10k. Let’s change the cash and run the example again. from __future__ import (absolute_import, division, print_function, unicode_literals) import backtrader as bt if __name__ == '__main__': cerebro = bt.Cerebro() cerebro.broker.setcash(100000.0) print('Starting Portfolio Value: %.2f' % cerebro.broker.getvalue()) cerebro.run() print('Final Portfolio Value: %.2f' % cerebro.broker.getvalue()) After the execution the output is: Starting Portfolio Value: 1000000.00 Final Portfolio Value: 1000000.00 Mission accomplished. Let’s move to tempestuous waters. Adding a Data Feed Having cash is fun, but the purpose behind all this is to let an automated strategy multiply the cash without moving a finger by operating on an asset which we see as a Data Feed Ergo … No Data Feed -> No Fun. Let’s add one to the ever growing example. from __future__ import (absolute_import, division, print_function, unicode_literals) import datetime # For datetime objects import os.path # To manage paths import sys # To find out the script name (in argv[0]) # Import the backtrader platform import backtrader as bt if __name__ == '__main__': # Create a cerebro entity cerebro = bt.Cerebro() # Datas are in a subfolder of the samples. Need to find where the script is # because it could have been called from anywhere modpath = os.path.dirname(os.path.abspath(sys.argv[0])) datapath = os.path.join(modpath, '../../datas/orcl-1995-2014.txt') # Create a Data Feed data = bt.feeds.YahooFinanceCSVData( dataname=datapath, # Do not pass values before this date fromdate=datetime.datetime(2000, 1, 1), # Do not pass values after this date todate=datetime.datetime(2000, 12, 31), reverse=False) # Add the Data Feed to Cerebro cerebro.adddata(data) # Set our desired cash start cerebro.broker.setcash(100000.0) # Print out the starting conditions print('Starting Portfolio Value: %.2f' % cerebro.broker.getvalue()) # Run over everything cerebro.run() # Print out the final result print('Final Portfolio Value: %.2f' % cerebro.broker.getvalue()) After the execution the output is: Starting Portfolio Value: 1000000.00 Final Portfolio Value: 1000000.00 The amount of boilerplate has grown slightly, because we added: Finding out where our example script is to be able to locate the sample Data Feed file Having datetime objects to filter on which data from the Data Feed we will be operating Aside from that, the Data Feed is created and added to cerebro. The output has not changed and it would be a miracle if it had. Note Yahoo Online sends the CSV data in date descending order, which is not the standard convention. The reversed=True prameter takes into account that the CSV data in the file has already been reversed and has the standard expected date ascending order. Our First Strategy The cash is in the broker and the Data Feed is there. It seems like risky business is just around the corner. Let’s put a Strategy into the equation and print the “Close” price of each day (bar). DataSeries (the underlying class in Data Feeds) objects have aliases to access the well known OHLC (Open High Low Close) daily values. This should ease up the creation of our printing logic. from __future__ import (absolute_import, division, print_function, unicode_literals) import datetime # For datetime objects import os.path # To manage paths import sys # To find out the script name (in argv[0]) # Import the backtrader platform import backtrader as bt # Create a Stratey class TestStrategy(bt.Strategy): def log(self, txt, dt=None): ''' Logging function for this strategy''' dt = dt or self.datas[0].datetime.date(0) print('%s, %s' % (dt.isoformat(), txt)) def __init__(self): # Keep a reference to the "close" line in the data[0] dataseries self.dataclose = self.datas[0].close def next(self): # Simply log the closing price of the series from the reference self.log('Close, %.2f' % self.dataclose[0]) if __name__ == '__main__': # Create a cerebro entity cerebro = bt.Cerebro() # Add a strategy cerebro.addstrategy(TestStrategy) # Datas are in a subfolder of the samples. Need to find where the script is # because it could have been called from anywhere modpath = os.path.dirname(os.path.abspath(sys.argv[0])) datapath = os.path.join(modpath, '../../datas/orcl-1995-2014.txt') # Create a Data Feed data = bt.feeds.YahooFinanceCSVData( dataname=datapath, # Do not pass values before this date fromdate=datetime.datetime(2000, 1, 1), # Do not pass values before this date todate=datetime.datetime(2000, 12, 31), # Do not pass values after this date reverse=False) # Add the Data Feed to Cerebro cerebro.adddata(data) # Set our desired cash start cerebro.broker.setcash(100000.0) # Print out the starting conditions print('Starting Portfolio Value: %.2f' % cerebro.broker.getvalue()) # Run over everything cerebro.run() # Print out the final result print('Final Portfolio Value: %.2f' % cerebro.broker.getvalue()) After the execution the output is: Starting Portfolio Value: 100000.00 2000-01-03T00:00:00, Close, 27.85 2000-01-04T00:00:00, Close, 25.39 2000-01-05T00:00:00, Close, 24.05 ... ... ... 2000-12-26T00:00:00, Close, 29.17 2000-12-27T00:00:00, Close, 28.94 2000-12-28T00:00:00, Close, 29.29 2000-12-29T00:00:00, Close, 27.41 Final Portfolio Value: 100000.00 Someone said the stockmarket was risky business, but it doesn’t seem so. Let’s explain some of the magic: Upon init being called the strategy already has a list of datas that are present in the platform This is a standard Python list and datas can be accessed in the order they were inserted. The first data in the list self.datas[0] is the default data for trading operations and to keep all strategy elements synchronized (it’s the system clock) self.dataclose = self.datas[0].close keeps a reference to the close line. Only one level of indirection is later needed to access the close values. The strategy next method will be called on each bar of the system clock (self.datas[0]). This is true until other things come into play like indicators, which need some bars to start producing an output. More on that later. Adding some Logic to the Strategy Let’s try some crazy idea we had by looking at some charts If the price has been falling 3 sessions in a row … BUY BUY BUY!!! from __future__ import (absolute_import, division, print_function, unicode_literals) import datetime # For datetime objects import os.path # To manage paths import sys # To find out the script name (in argv[0]) # Import the backtrader platform import backtrader as bt # Create a Stratey class TestStrategy(bt.Strategy): def log(self, txt, dt=None): ''' Logging function fot this strategy''' dt = dt or self.datas[0].datetime.date(0) print('%s, %s' % (dt.isoformat(), txt)) def __init__(self): # Keep a reference to the "close" line in the data[0] dataseries self.dataclose = self.datas[0].close def next(self): # Simply log the closing price of the series from the reference self.log('Close, %.2f' % self.dataclose[0]) if self.dataclose[0] < self.dataclose[-1]: # current close less than previous close if self.dataclose[-1] < self.dataclose[-2]: # previous close less than the previous close # BUY, BUY, BUY!!! (with all possible default parameters) self.log('BUY CREATE, %.2f' % self.dataclose[0]) self.buy() if __name__ == '__main__': # Create a cerebro entity cerebro = bt.Cerebro() # Add a strategy cerebro.addstrategy(TestStrategy) # Datas are in a subfolder of the samples. Need to find where the script is # because it could have been called from anywhere modpath = os.path.dirname(os.path.abspath(sys.argv[0])) datapath = os.path.join(modpath, '../../datas/orcl-1995-2014.txt') # Create a Data Feed data = bt.feeds.YahooFinanceCSVData( dataname=datapath, # Do not pass values before this date fromdate=datetime.datetime(2000, 1, 1), # Do not pass values before this date todate=datetime.datetime(2000, 12, 31), # Do not pass values after this date reverse=False) # Add the Data Feed to Cerebro cerebro.adddata(data) # Set our desired cash start cerebro.broker.setcash(100000.0) # Print out the starting conditions print('Starting Portfolio Value: %.2f' % cerebro.broker.getvalue()) # Run over everything cerebro.run() # Print out the final result print('Final Portfolio Value: %.2f' % cerebro.broker.getvalue()) After the execution the output is: Starting Portfolio Value: 100000.00 2000-01-03, Close, 27.85 2000-01-04, Close, 25.39 2000-01-05, Close, 24.05 2000-01-05, BUY CREATE, 24.05 2000-01-06, Close, 22.63 2000-01-06, BUY CREATE, 22.63 2000-01-07, Close, 24.37 ... ... ... 2000-12-20, BUY CREATE, 26.88 2000-12-21, Close, 27.82 2000-12-22, Close, 30.06 2000-12-26, Close, 29.17 2000-12-27, Close, 28.94 2000-12-27, BUY CREATE, 28.94 2000-12-28, Close, 29.29 2000-12-29, Close, 27.41 Final Portfolio Value: 99725.08 Several “BUY” creation orders were issued, our porftolio value was decremented. A couple of important things are clearly missing. The order was created but it is unknown if it was executed, when and at what price. The next example will build upon that by listening to notifications of order status. The curious reader may ask how many shares are being bought, what asset is being bought and how are orders being executed. Where possible (and in this case it is) the platform fills in the gaps: self.datas[0] (the main data aka system clock) is the target asset if no other one is specified The stake is provided behind the scenes by a position sizer which uses a fixed stake, being the default “1”. It will be modified in a later example The order is executed “At Market”. The broker (shown in previous examples) executes this using the opening price of the next bar, because that’s the 1st tick after the current under examination bar. The order is executed so far without any commission (more on that later) Do not only buy … but SELL After knowing how to enter the market (long), an “exit concept” is needed and also understanding whether the strategy is in the market. Luckily a Strategy object offers access to a position attribute for the default data feed Methods buy and sell return the created (not yet executed) order Changes in orders’ status will be notified to the strategy via a notify method The “exit concept” will be an easy one: Exit after 5 bars (on the 6th bar) have elapsed for good or for worse Please notice that there is no “time” or “timeframe” implied: number of bars. The bars can represent 1 minute, 1 hour, 1 day, 1 week or any other time period. Although we know the data source is a daily one, the strategy makes no assumption about that. Additionally and to simplify: Do only allow a Buy order if not yet in the market Note The next method gets no “bar index” passed and therefore it seems obscure how to understand when 5 bars may have elapsed, but this has been modeled in pythonic way: call len on an object and it will tell you the length of its lines. Just write down (save in a variable) at which length in an operation took place and see if the current length is 5 bars away. from __future__ import (absolute_import, division, print_function, unicode_literals) import datetime # For datetime objects import os.path # To manage paths import sys # To find out the script name (in argv[0]) # Import the backtrader platform import backtrader as bt # Create a Stratey class TestStrategy(bt.Strategy): def log(self, txt, dt=None): ''' Logging function fot this strategy''' dt = dt or self.datas[0].datetime.date(0) print('%s, %s' % (dt.isoformat(), txt)) def __init__(self): # Keep a reference to the "close" line in the data[0] dataseries self.dataclose = self.datas[0].close # To keep track of pending orders self.order = None def notify_order(self, order): if order.status in [order.Submitted, order.Accepted]: # Buy/Sell order submitted/accepted to/by broker - Nothing to do return # Check if an order has been completed # Attention: broker could reject order if not enough cash if order.status in [order.Completed]: if order.isbuy(): self.log('BUY EXECUTED, %.2f' % order.executed.price) elif order.issell(): self.log('SELL EXECUTED, %.2f' % order.executed.price) self.bar_executed = len(self) elif order.status in [order.Canceled, order.Margin, order.Rejected]: self.log('Order Canceled/Margin/Rejected') # Write down: no pending order self.order = None def next(self): # Simply log the closing price of the series from the reference self.log('Close, %.2f' % self.dataclose[0]) # Check if an order is pending ... if yes, we cannot send a 2nd one if self.order: return # Check if we are in the market if not self.position: # Not yet ... we MIGHT BUY if ... if self.dataclose[0] < self.dataclose[-1]: # current close less than previous close if self.dataclose[-1] < self.dataclose[-2]: # previous close less than the previous close # BUY, BUY, BUY!!! (with default parameters) self.log('BUY CREATE, %.2f' % self.dataclose[0]) # Keep track of the created order to avoid a 2nd order self.order = self.buy() else: # Already in the market ... we might sell if len(self) >= (self.bar_executed + 5): # SELL, SELL, SELL!!! (with all possible default parameters) self.log('SELL CREATE, %.2f' % self.dataclose[0]) # Keep track of the created order to avoid a 2nd order self.order = self.sell() if __name__ == '__main__': # Create a cerebro entity cerebro = bt.Cerebro() # Add a strategy cerebro.addstrategy(TestStrategy) # Datas are in a subfolder of the samples. Need to find where the script is # because it could have been called from anywhere modpath = os.path.dirname(os.path.abspath(sys.argv[0])) datapath = os.path.join(modpath, '../../datas/orcl-1995-2014.txt') # Create a Data Feed data = bt.feeds.YahooFinanceCSVData( dataname=datapath, # Do not pass values before this date fromdate=datetime.datetime(2000, 1, 1), # Do not pass values before this date todate=datetime.datetime(2000, 12, 31), # Do not pass values after this date reverse=False) # Add the Data Feed to Cerebro cerebro.adddata(data) # Set our desired cash start cerebro.broker.setcash(100000.0) # Print out the starting conditions print('Starting Portfolio Value: %.2f' % cerebro.broker.getvalue()) # Run over everything cerebro.run() # Print out the final result print('Final Portfolio Value: %.2f' % cerebro.broker.getvalue()) After the execution the output is: Starting Portfolio Value: 100000.00 2000-01-03T00:00:00, Close, 27.85 2000-01-04T00:00:00, Close, 25.39 2000-01-05T00:00:00, Close, 24.05 2000-01-05T00:00:00, BUY CREATE, 24.05 2000-01-06T00:00:00, BUY EXECUTED, 23.61 2000-01-06T00:00:00, Close, 22.63 2000-01-07T00:00:00, Close, 24.37 2000-01-10T00:00:00, Close, 27.29 2000-01-11T00:00:00, Close, 26.49 2000-01-12T00:00:00, Close, 24.90 2000-01-13T00:00:00, Close, 24.77 2000-01-13T00:00:00, SELL CREATE, 24.77 2000-01-14T00:00:00, SELL EXECUTED, 25.70 2000-01-14T00:00:00, Close, 25.18 ... ... ... 2000-12-15T00:00:00, SELL CREATE, 26.93 2000-12-18T00:00:00, SELL EXECUTED, 28.29 2000-12-18T00:00:00, Close, 30.18 2000-12-19T00:00:00, Close, 28.88 2000-12-20T00:00:00, Close, 26.88 2000-12-20T00:00:00, BUY CREATE, 26.88 2000-12-21T00:00:00, BUY EXECUTED, 26.23 2000-12-21T00:00:00, Close, 27.82 2000-12-22T00:00:00, Close, 30.06 2000-12-26T00:00:00, Close, 29.17 2000-12-27T00:00:00, Close, 28.94 2000-12-28T00:00:00, Close, 29.29 2000-12-29T00:00:00, Close, 27.41 2000-12-29T00:00:00, SELL CREATE, 27.41 Final Portfolio Value: 100018.53 Blistering Barnacles!!! The system made money … something must be wrong The broker says: Show me the money! And the money is called “commission”. Let’s add a reasonable 0.1% commision rate per operation (both for buying and selling … yes the broker is avid …) A single line will suffice for it: # 0.1% ... divide by 100 to remove the % cerebro.broker.setcommission(commission=0.001) Being experienced with the platform we want to see the profit or loss after a buy/sell cycle, with and without commission. from __future__ import (absolute_import, division, print_function, unicode_literals) import datetime # For datetime objects import os.path # To manage paths import sys # To find out the script name (in argv[0]) # Import the backtrader platform import backtrader as bt # Create a Stratey class TestStrategy(bt.Strategy): def log(self, txt, dt=None): ''' Logging function fot this strategy''' dt = dt or self.datas[0].datetime.date(0) print('%s, %s' % (dt.isoformat(), txt)) def __init__(self): # Keep a reference to the "close" line in the data[0] dataseries self.dataclose = self.datas[0].close # To keep track of pending orders and buy price/commission self.order = None self.buyprice = None self.buycomm = None def notify_order(self, order): if order.status in [order.Submitted, order.Accepted]: # Buy/Sell order submitted/accepted to/by broker - Nothing to do return # Check if an order has been completed # Attention: broker could reject order if not enough cash if order.status in [order.Completed]: if order.isbuy(): self.log( 'BUY EXECUTED, Price: %.2f, Cost: %.2f, Comm %.2f' % (order.executed.price, order.executed.value, order.executed.comm)) self.buyprice = order.executed.price self.buycomm = order.executed.comm else: # Sell self.log('SELL EXECUTED, Price: %.2f, Cost: %.2f, Comm %.2f' % (order.executed.price, order.executed.value, order.executed.comm)) self.bar_executed = len(self) elif order.status in [order.Canceled, order.Margin, order.Rejected]: self.log('Order Canceled/Margin/Rejected') self.order = None def notify_trade(self, trade): if not trade.isclosed: return self.log('OPERATION PROFIT, GROSS %.2f, NET %.2f' % (trade.pnl, trade.pnlcomm)) def next(self): # Simply log the closing price of the series from the reference self.log('Close, %.2f' % self.dataclose[0]) # Check if an order is pending ... if yes, we cannot send a 2nd one if self.order: return # Check if we are in the market if not self.position: # Not yet ... we MIGHT BUY if ... if self.dataclose[0] < self.dataclose[-1]: # current close less than previous close if self.dataclose[-1] < self.dataclose[-2]: # previous close less than the previous close # BUY, BUY, BUY!!! (with default parameters) self.log('BUY CREATE, %.2f' % self.dataclose[0]) # Keep track of the created order to avoid a 2nd order self.order = self.buy() else: # Already in the market ... we might sell if len(self) >= (self.bar_executed + 5): # SELL, SELL, SELL!!! (with all possible default parameters) self.log('SELL CREATE, %.2f' % self.dataclose[0]) # Keep track of the created order to avoid a 2nd order self.order = self.sell() if __name__ == '__main__': # Create a cerebro entity cerebro = bt.Cerebro() # Add a strategy cerebro.addstrategy(TestStrategy) # Datas are in a subfolder of the samples. Need to find where the script is # because it could have been called from anywhere modpath = os.path.dirname(os.path.abspath(sys.argv[0])) datapath = os.path.join(modpath, '../../datas/orcl-1995-2014.txt') # Create a Data Feed data = bt.feeds.YahooFinanceCSVData( dataname=datapath, # Do not pass values before this date fromdate=datetime.datetime(2000, 1, 1), # Do not pass values before this date todate=datetime.datetime(2000, 12, 31), # Do not pass values after this date reverse=False) # Add the Data Feed to Cerebro cerebro.adddata(data) # Set our desired cash start cerebro.broker.setcash(100000.0) # Set the commission - 0.1% ... divide by 100 to remove the % cerebro.broker.setcommission(commission=0.001) # Print out the starting conditions print('Starting Portfolio Value: %.2f' % cerebro.broker.getvalue()) # Run over everything cerebro.run() # Print out the final result print('Final Portfolio Value: %.2f' % cerebro.broker.getvalue()) After the execution the output is: Starting Portfolio Value: 100000.00 2000-01-03T00:00:00, Close, 27.85 2000-01-04T00:00:00, Close, 25.39 2000-01-05T00:00:00, Close, 24.05 2000-01-05T00:00:00, BUY CREATE, 24.05 2000-01-06T00:00:00, BUY EXECUTED, Price: 23.61, Cost: 23.61, Commission 0.02 2000-01-06T00:00:00, Close, 22.63 2000-01-07T00:00:00, Close, 24.37 2000-01-10T00:00:00, Close, 27.29 2000-01-11T00:00:00, Close, 26.49 2000-01-12T00:00:00, Close, 24.90 2000-01-13T00:00:00, Close, 24.77 2000-01-13T00:00:00, SELL CREATE, 24.77 2000-01-14T00:00:00, SELL EXECUTED, Price: 25.70, Cost: 25.70, Commission 0.03 2000-01-14T00:00:00, OPERATION PROFIT, GROSS 2.09, NET 2.04 2000-01-14T00:00:00, Close, 25.18 ... ... ... 2000-12-15T00:00:00, SELL CREATE, 26.93 2000-12-18T00:00:00, SELL EXECUTED, Price: 28.29, Cost: 28.29, Commission 0.03 2000-12-18T00:00:00, OPERATION PROFIT, GROSS -0.06, NET -0.12 2000-12-18T00:00:00, Close, 30.18 2000-12-19T00:00:00, Close, 28.88 2000-12-20T00:00:00, Close, 26.88 2000-12-20T00:00:00, BUY CREATE, 26.88 2000-12-21T00:00:00, BUY EXECUTED, Price: 26.23, Cost: 26.23, Commission 0.03 2000-12-21T00:00:00, Close, 27.82 2000-12-22T00:00:00, Close, 30.06 2000-12-26T00:00:00, Close, 29.17 2000-12-27T00:00:00, Close, 28.94 2000-12-28T00:00:00, Close, 29.29 2000-12-29T00:00:00, Close, 27.41 2000-12-29T00:00:00, SELL CREATE, 27.41 Final Portfolio Value: 100016.98 God Save the Queen!!! The system still made money. Before moving on, let’s notice something by filtering the “OPERATION PROFIT” lines: 2000-01-14T00:00:00, OPERATION PROFIT, GROSS 2.09, NET 2.04 2000-02-07T00:00:00, OPERATION PROFIT, GROSS 3.68, NET 3.63 2000-02-28T00:00:00, OPERATION PROFIT, GROSS 4.48, NET 4.42 2000-03-13T00:00:00, OPERATION PROFIT, GROSS 3.48, NET 3.41 2000-03-22T00:00:00, OPERATION PROFIT, GROSS -0.41, NET -0.49 2000-04-07T00:00:00, OPERATION PROFIT, GROSS 2.45, NET 2.37 2000-04-20T00:00:00, OPERATION PROFIT, GROSS -1.95, NET -2.02 2000-05-02T00:00:00, OPERATION PROFIT, GROSS 5.46, NET 5.39 2000-05-11T00:00:00, OPERATION PROFIT, GROSS -3.74, NET -3.81 2000-05-30T00:00:00, OPERATION PROFIT, GROSS -1.46, NET -1.53 2000-07-05T00:00:00, OPERATION PROFIT, GROSS -1.62, NET -1.69 2000-07-14T00:00:00, OPERATION PROFIT, GROSS 2.08, NET 2.01 2000-07-28T00:00:00, OPERATION PROFIT, GROSS 0.14, NET 0.07 2000-08-08T00:00:00, OPERATION PROFIT, GROSS 4.36, NET 4.29 2000-08-21T00:00:00, OPERATION PROFIT, GROSS 1.03, NET 0.95 2000-09-15T00:00:00, OPERATION PROFIT, GROSS -4.26, NET -4.34 2000-09-27T00:00:00, OPERATION PROFIT, GROSS 1.29, NET 1.22 2000-10-13T00:00:00, OPERATION PROFIT, GROSS -2.98, NET -3.04 2000-10-26T00:00:00, OPERATION PROFIT, GROSS 3.01, NET 2.95 2000-11-06T00:00:00, OPERATION PROFIT, GROSS -3.59, NET -3.65 2000-11-16T00:00:00, OPERATION PROFIT, GROSS 1.28, NET 1.23 2000-12-01T00:00:00, OPERATION PROFIT, GROSS 2.59, NET 2.54 2000-12-18T00:00:00, OPERATION PROFIT, GROSS -0.06, NET -0.12 Adding up the “NET” profits the final figure is: 15.83 But the system said the following at the end: 2000-12-29T00:00:00, SELL CREATE, 27.41 Final Portfolio Value: 100016.98 And obviously 15.83 is not 16.98. There is no error whatsoever. The “NET” profit of 15.83 is already cash in the bag. Unfortunately (or fortunately to better understand the platform) there is an open position on the last day of the Data Feed. Even if a SELL operation has been sent … IT HAS NOT YET BEEN EXECUTED. The “Final Portfolio Value” calculated by the broker takes into account the “Close” price on 2000-12-29. The actual execution price would have been set on the next trading day which happened to be 2001-01-02. Extending the Data Feed” to take into account this day the output is: 2001-01-02T00:00:00, SELL EXECUTED, Price: 27.87, Cost: 27.87, Commission 0.03 2001-01-02T00:00:00, OPERATION PROFIT, GROSS 1.64, NET 1.59 2001-01-02T00:00:00, Close, 24.87 2001-01-02T00:00:00, BUY CREATE, 24.87 Final Portfolio Value: 100017.41 Now adding the previous NET profit to the completed operation’s net profit: 15.83 + 1.59 = 17.42 Which (discarding rounding errors in the “print” statements) is the extra Portfolio above the initial 100000 monetary units the strategy started with. Customizing the Strategy: Parameters It would a bit unpractical to hardcode some of the values in the strategy and have no chance to change them easily. Parameters come in handy to help. Definition of parameters is easy and looks like: params = (('myparam', 27), ('exitbars', 5),) Being this a standard Python tuple with some tuples inside it, the following may look more appealling to some: params = ( ('myparam', 27), ('exitbars', 5), ) With either formatting parametrization of the strategy is allowed when adding the strategy to the Cerebro engine: # Add a strategy cerebro.addstrategy(TestStrategy, myparam=20, exitbars=7) Note The setsizing method below is deprecated. This content is kept here for anyone looking at old samples of the sources. The sources have been update to use: cerebro.addsizer(bt.sizers.FixedSize, stake=10)`` Please read the section about sizers Using the parameters in the strategy is easy, as they are stored in a “params” attribute. If we for example want to set the stake fix, we can pass the stake parameter to the position sizer like this durint init: # Set the sizer stake from the params self.sizer.setsizing(self.params.stake) We could have also called buy and sell with a stake parameter and self.params.stake as the value. The logic to exit gets modified: # Already in the market ... we might sell if len(self) >= (self.bar_executed + self.params.exitbars): With all this in mind the example evolves to look like: from __future__ import (absolute_import, division, print_function, unicode_literals) import datetime # For datetime objects import os.path # To manage paths import sys # To find out the script name (in argv[0]) # Import the backtrader platform import backtrader as bt # Create a Stratey class TestStrategy(bt.Strategy): params = ( ('exitbars', 5), ) def log(self, txt, dt=None): ''' Logging function fot this strategy''' dt = dt or self.datas[0].datetime.date(0) print('%s, %s' % (dt.isoformat(), txt)) def __init__(self): # Keep a reference to the "close" line in the data[0] dataseries self.dataclose = self.datas[0].close # To keep track of pending orders and buy price/commission self.order = None self.buyprice = None self.buycomm = None def notify_order(self, order): if order.status in [order.Submitted, order.Accepted]: # Buy/Sell order submitted/accepted to/by broker - Nothing to do return # Check if an order has been completed # Attention: broker could reject order if not enough cash if order.status in [order.Completed]: if order.isbuy(): self.log( 'BUY EXECUTED, Price: %.2f, Cost: %.2f, Comm %.2f' % (order.executed.price, order.executed.value, order.executed.comm)) self.buyprice = order.executed.price self.buycomm = order.executed.comm else: # Sell self.log('SELL EXECUTED, Price: %.2f, Cost: %.2f, Comm %.2f' % (order.executed.price, order.executed.value, order.executed.comm)) self.bar_executed = len(self) elif order.status in [order.Canceled, order.Margin, order.Rejected]: self.log('Order Canceled/Margin/Rejected') self.order = None def notify_trade(self, trade): if not trade.isclosed: return self.log('OPERATION PROFIT, GROSS %.2f, NET %.2f' % (trade.pnl, trade.pnlcomm)) def next(self): # Simply log the closing price of the series from the reference self.log('Close, %.2f' % self.dataclose[0]) # Check if an order is pending ... if yes, we cannot send a 2nd one if self.order: return # Check if we are in the market if not self.position: # Not yet ... we MIGHT BUY if ... if self.dataclose[0] < self.dataclose[-1]: # current close less than previous close if self.dataclose[-1] < self.dataclose[-2]: # previous close less than the previous close # BUY, BUY, BUY!!! (with default parameters) self.log('BUY CREATE, %.2f' % self.dataclose[0]) # Keep track of the created order to avoid a 2nd order self.order = self.buy() else: # Already in the market ... we might sell if len(self) >= (self.bar_executed + self.params.exitbars): # SELL, SELL, SELL!!! (with all possible default parameters) self.log('SELL CREATE, %.2f' % self.dataclose[0]) # Keep track of the created order to avoid a 2nd order self.order = self.sell() if __name__ == '__main__': # Create a cerebro entity cerebro = bt.Cerebro() # Add a strategy cerebro.addstrategy(TestStrategy) # Datas are in a subfolder of the samples. Need to find where the script is # because it could have been called from anywhere modpath = os.path.dirname(os.path.abspath(sys.argv[0])) datapath = os.path.join(modpath, '../../datas/orcl-1995-2014.txt') # Create a Data Feed data = bt.feeds.YahooFinanceCSVData( dataname=datapath, # Do not pass values before this date fromdate=datetime.datetime(2000, 1, 1), # Do not pass values before this date todate=datetime.datetime(2000, 12, 31), # Do not pass values after this date reverse=False) # Add the Data Feed to Cerebro cerebro.adddata(data) # Set our desired cash start cerebro.broker.setcash(100000.0) # Add a FixedSize sizer according to the stake cerebro.addsizer(bt.sizers.FixedSize, stake=10) # Set the commission - 0.1% ... divide by 100 to remove the % cerebro.broker.setcommission(commission=0.001) # Print out the starting conditions print('Starting Portfolio Value: %.2f' % cerebro.broker.getvalue()) # Run over everything cerebro.run() # Print out the final result print('Final Portfolio Value: %.2f' % cerebro.broker.getvalue()) After the execution the output is: Starting Portfolio Value: 100000.00 2000-01-03T00:00:00, Close, 27.85 2000-01-04T00:00:00, Close, 25.39 2000-01-05T00:00:00, Close, 24.05 2000-01-05T00:00:00, BUY CREATE, 24.05 2000-01-06T00:00:00, BUY EXECUTED, Size 10, Price: 23.61, Cost: 236.10, Commission 0.24 2000-01-06T00:00:00, Close, 22.63 ... ... ... 2000-12-20T00:00:00, BUY CREATE, 26.88 2000-12-21T00:00:00, BUY EXECUTED, Size 10, Price: 26.23, Cost: 262.30, Commission 0.26 2000-12-21T00:00:00, Close, 27.82 2000-12-22T00:00:00, Close, 30.06 2000-12-26T00:00:00, Close, 29.17 2000-12-27T00:00:00, Close, 28.94 2000-12-28T00:00:00, Close, 29.29 2000-12-29T00:00:00, Close, 27.41 2000-12-29T00:00:00, SELL CREATE, 27.41 Final Portfolio Value: 100169.80 In order to see the difference, the print outputs have also been extended to show the execution size. Having multiplied the stake by 10, the obvious has happened: the profit and loss has been multiplied by 10. Instead of 16.98, the surplus is now 169.80 Adding an indicator Having heard of indicators, the next thing anyone would add to the strategy is one of them. For sure they must be much better than a simple “3 lower closes” strategy. Inspired in one of the examples from PyAlgoTrade a strategy using a Simple Moving Average. Buy “AtMarket” if the close is greater than the Average If in the market, sell if the close is smaller than the Average Only 1 active operation is allowed in the market Most of the existing code can be kept in place. Let’s add the average during init and keep a reference to it: self.sma = bt.indicators.MovingAverageSimple(self.datas[0], period=self.params.maperiod) And of course the logic to enter and exit the market will rely on the Average values. Look in the code for the logic. Note The starting cash will be 1000 monetary units to be in line with the PyAlgoTrade example and no commission will be applied from __future__ import (absolute_import, division, print_function, unicode_literals) import datetime # For datetime objects import os.path # To manage paths import sys # To find out the script name (in argv[0]) # Import the backtrader platform import backtrader as bt # Create a Stratey class TestStrategy(bt.Strategy): params = ( ('maperiod', 15), ) def log(self, txt, dt=None): ''' Logging function fot this strategy''' dt = dt or self.datas[0].datetime.date(0) print('%s, %s' % (dt.isoformat(), txt)) def __init__(self): # Keep a reference to the "close" line in the data[0] dataseries self.dataclose = self.datas[0].close # To keep track of pending orders and buy price/commission self.order = None self.buyprice = None self.buycomm = None # Add a MovingAverageSimple indicator self.sma = bt.indicators.SimpleMovingAverage( self.datas[0], period=self.params.maperiod) def notify_order(self, order): if order.status in [order.Submitted, order.Accepted]: # Buy/Sell order submitted/accepted to/by broker - Nothing to do return # Check if an order has been completed # Attention: broker could reject order if not enough cash if order.status in [order.Completed]: if order.isbuy(): self.log( 'BUY EXECUTED, Price: %.2f, Cost: %.2f, Comm %.2f' % (order.executed.price, order.executed.value, order.executed.comm)) self.buyprice = order.executed.price self.buycomm = order.executed.comm else: # Sell self.log('SELL EXECUTED, Price: %.2f, Cost: %.2f, Comm %.2f' % (order.executed.price, order.executed.value, order.executed.comm)) self.bar_executed = len(self) elif order.status in [order.Canceled, order.Margin, order.Rejected]: self.log('Order Canceled/Margin/Rejected') self.order = None def notify_trade(self, trade): if not trade.isclosed: return self.log('OPERATION PROFIT, GROSS %.2f, NET %.2f' % (trade.pnl, trade.pnlcomm)) def next(self): # Simply log the closing price of the series from the reference self.log('Close, %.2f' % self.dataclose[0]) # Check if an order is pending ... if yes, we cannot send a 2nd one if self.order: return # Check if we are in the market if not self.position: # Not yet ... we MIGHT BUY if ... if self.dataclose[0] > self.sma[0]: # BUY, BUY, BUY!!! (with all possible default parameters) self.log('BUY CREATE, %.2f' % self.dataclose[0]) # Keep track of the created order to avoid a 2nd order self.order = self.buy() else: if self.dataclose[0] < self.sma[0]: # SELL, SELL, SELL!!! (with all possible default parameters) self.log('SELL CREATE, %.2f' % self.dataclose[0]) # Keep track of the created order to avoid a 2nd order self.order = self.sell() if __name__ == '__main__': # Create a cerebro entity cerebro = bt.Cerebro() # Add a strategy cerebro.addstrategy(TestStrategy) # Datas are in a subfolder of the samples. Need to find where the script is # because it could have been called from anywhere modpath = os.path.dirname(os.path.abspath(sys.argv[0])) datapath = os.path.join(modpath, '../../datas/orcl-1995-2014.txt') # Create a Data Feed data = bt.feeds.YahooFinanceCSVData( dataname=datapath, # Do not pass values before this date fromdate=datetime.datetime(2000, 1, 1), # Do not pass values before this date todate=datetime.datetime(2000, 12, 31), # Do not pass values after this date reverse=False) # Add the Data Feed to Cerebro cerebro.adddata(data) # Set our desired cash start cerebro.broker.setcash(1000.0) # Add a FixedSize sizer according to the stake cerebro.addsizer(bt.sizers.FixedSize, stake=10) # Set the commission cerebro.broker.setcommission(commission=0.0) # Print out the starting conditions print('Starting Portfolio Value: %.2f' % cerebro.broker.getvalue()) # Run over everything cerebro.run() # Print out the final result print('Final Portfolio Value: %.2f' % cerebro.broker.getvalue()) Now, before skipping to the next section LOOK CAREFULLY to the first date which is shown in the log: It’ no longer 2000-01-03, the first trading day in the year 2K. It’s 2000-01-24 … Who has stolen my cheese? The missing days are not missing. The platform has adapted to the new circumstances: An indicator (SimpleMovingAverage) has been added to the Strategy. This indicator needs X bars to produce an output: in the example: 15 2000-01-24 is the day in which the 15th bar occurs The backtrader platform assumes that the Strategy has the indicator in place for a good reason, to use it in the decision making process. And it makes no sense to try to make decisions if the indicator is not yet ready and producing values. next will be 1st called when all indicators have already reached the minimum needed period to produce a value In the example there is a single indicator, but the strategy could have any number of them. After the execution the output is: Starting Portfolio Value: 1000.00 2000-01-24T00:00:00, Close, 25.55 2000-01-25T00:00:00, Close, 26.61 2000-01-25T00:00:00, BUY CREATE, 26.61 2000-01-26T00:00:00, BUY EXECUTED, Size 10, Price: 26.76, Cost: 267.60, Commission 0.00 2000-01-26T00:00:00, Close, 25.96 2000-01-27T00:00:00, Close, 24.43 2000-01-27T00:00:00, SELL CREATE, 24.43 2000-01-28T00:00:00, SELL EXECUTED, Size 10, Price: 24.28, Cost: 242.80, Commission 0.00 2000-01-28T00:00:00, OPERATION PROFIT, GROSS -24.80, NET -24.80 2000-01-28T00:00:00, Close, 22.34 2000-01-31T00:00:00, Close, 23.55 2000-02-01T00:00:00, Close, 25.46 2000-02-02T00:00:00, Close, 25.61 2000-02-02T00:00:00, BUY CREATE, 25.61 2000-02-03T00:00:00, BUY EXECUTED, Size 10, Price: 26.11, Cost: 261.10, Commission 0.00 ... ... ... 2000-12-20T00:00:00, SELL CREATE, 26.88 2000-12-21T00:00:00, SELL EXECUTED, Size 10, Price: 26.23, Cost: 262.30, Commission 0.00 2000-12-21T00:00:00, OPERATION PROFIT, GROSS -20.60, NET -20.60 2000-12-21T00:00:00, Close, 27.82 2000-12-21T00:00:00, BUY CREATE, 27.82 2000-12-22T00:00:00, BUY EXECUTED, Size 10, Price: 28.65, Cost: 286.50, Commission 0.00 2000-12-22T00:00:00, Close, 30.06 2000-12-26T00:00:00, Close, 29.17 2000-12-27T00:00:00, Close, 28.94 2000-12-28T00:00:00, Close, 29.29 2000-12-29T00:00:00, Close, 27.41 2000-12-29T00:00:00, SELL CREATE, 27.41 Final Portfolio Value: 973.90 In the name of the King!!! A winning system turned into a losing one … and that with no commission. It may well be that simply adding an indicator is not the universal panacea. Note The same logic and data with PyAlgoTrade yields a slightly different result (slightly off). Looking at the entire printout reveals that some operations are not exactly the same. Being the culprit again the usual suspect: rounding. PyAlgoTrade does not round the datafeed values when applying the divided “adjusted close” to the data feed values. The Yahoo Data Feed provided by backtrader rounds the values down to 2 decimals after applying the adjusted close. Upon printing the values everything seems the same, but it’s obvious that sometimes that 5th place decimal plays a role. Rounding down to 2 decimals seems more realistic, because Market Exchanges do only allow a number of decimals per asset (being that 2 decimals usually for stocks) Note The Yahoo Data Feed (starting with version 1.8.11.99 allows to specify if rounding has to happen and how many decimals) Visual Inspection: Plotting A printout or log of the actual whereabouts of the system at each bar-instant is good but humans tend to be visual and therefore it seems right to offer a view of the same whereabouts as chart. Note To plot you need to have matplotlib installed Once again defaults for plotting are there to assist the platform user. Plotting is incredibly a 1 line operation: cerebro.plot() Being the location for sure after cerebro.run() has been called. In order to display the automatic plotting capabilities and a couple of easy customizations, the following will be done: A 2nd MovingAverage (Exponential) will be added. The defaults will plot it (just like the 1st) with the data. A 3rd MovingAverage (Weighted) will be added. Customized to plot in an own plot (even if not sensible) A Stochastic (Slow) will be added. No change to the defaults. A MACD will be added. No change to the defaults. A RSI will be added. No change to the defaults. A MovingAverage (Simple) will be applied to the RSI. No change to the defaults (it will be plotted with the RSI) An AverageTrueRange will be added. Changed defaults to avoid it being plotted. The entire set of additions to the init method of the Strategy: # Indicators for the plotting show bt.indicators.ExponentialMovingAverage(self.datas[0], period=25) bt.indicators.WeightedMovingAverage(self.datas[0], period=25).subplot = True bt.indicators.StochasticSlow(self.datas[0]) bt.indicators.MACDHisto(self.datas[0]) rsi = bt.indicators.RSI(self.datas[0]) bt.indicators.SmoothedMovingAverage(rsi, period=10) bt.indicators.ATR(self.datas[0]).plot = False Note Even if indicators are not explicitly added to a member variable of the strategy (like self.sma = MovingAverageSimple…), they will autoregister with the strategy and will influence the minimum period for next and will be part of the plotting. In the example only RSI is added to a temporary variable rsi with the only intention to create a MovingAverageSmoothed on it. The example now: from __future__ import (absolute_import, division, print_function, unicode_literals) import datetime # For datetime objects import os.path # To manage paths import sys # To find out the script name (in argv[0]) # Import the backtrader platform import backtrader as bt # Create a Stratey class TestStrategy(bt.Strategy): params = ( ('maperiod', 15), ) def log(self, txt, dt=None): ''' Logging function fot this strategy''' dt = dt or self.datas[0].datetime.date(0) print('%s, %s' % (dt.isoformat(), txt)) def __init__(self): # Keep a reference to the "close" line in the data[0] dataseries self.dataclose = self.datas[0].close # To keep track of pending orders and buy price/commission self.order = None self.buyprice = None self.buycomm = None # Add a MovingAverageSimple indicator self.sma = bt.indicators.SimpleMovingAverage( self.datas[0], period=self.params.maperiod) # Indicators for the plotting show bt.indicators.ExponentialMovingAverage(self.datas[0], period=25) bt.indicators.WeightedMovingAverage(self.datas[0], period=25, subplot=True) bt.indicators.StochasticSlow(self.datas[0]) bt.indicators.MACDHisto(self.datas[0]) rsi = bt.indicators.RSI(self.datas[0]) bt.indicators.SmoothedMovingAverage(rsi, period=10) bt.indicators.ATR(self.datas[0], plot=False) def notify_order(self, order): if order.status in [order.Submitted, order.Accepted]: # Buy/Sell order submitted/accepted to/by broker - Nothing to do return # Check if an order has been completed # Attention: broker could reject order if not enough cash if order.status in [order.Completed]: if order.isbuy(): self.log( 'BUY EXECUTED, Price: %.2f, Cost: %.2f, Comm %.2f' % (order.executed.price, order.executed.value, order.executed.comm)) self.buyprice = order.executed.price self.buycomm = order.executed.comm else: # Sell self.log('SELL EXECUTED, Price: %.2f, Cost: %.2f, Comm %.2f' % (order.executed.price, order.executed.value, order.executed.comm)) self.bar_executed = len(self) elif order.status in [order.Canceled, order.Margin, order.Rejected]: self.log('Order Canceled/Margin/Rejected') # Write down: no pending order self.order = None def notify_trade(self, trade): if not trade.isclosed: return self.log('OPERATION PROFIT, GROSS %.2f, NET %.2f' % (trade.pnl, trade.pnlcomm)) def next(self): # Simply log the closing price of the series from the reference self.log('Close, %.2f' % self.dataclose[0]) # Check if an order is pending ... if yes, we cannot send a 2nd one if self.order: return # Check if we are in the market if not self.position: # Not yet ... we MIGHT BUY if ... if self.dataclose[0] > self.sma[0]: # BUY, BUY, BUY!!! (with all possible default parameters) self.log('BUY CREATE, %.2f' % self.dataclose[0]) # Keep track of the created order to avoid a 2nd order self.order = self.buy() else: if self.dataclose[0] < self.sma[0]: # SELL, SELL, SELL!!! (with all possible default parameters) self.log('SELL CREATE, %.2f' % self.dataclose[0]) # Keep track of the created order to avoid a 2nd order self.order = self.sell() if __name__ == '__main__': # Create a cerebro entity cerebro = bt.Cerebro() # Add a strategy cerebro.addstrategy(TestStrategy) # Datas are in a subfolder of the samples. Need to find where the script is # because it could have been called from anywhere modpath = os.path.dirname(os.path.abspath(sys.argv[0])) datapath = os.path.join(modpath, '../../datas/orcl-1995-2014.txt') # Create a Data Feed data = bt.feeds.YahooFinanceCSVData( dataname=datapath, # Do not pass values before this date fromdate=datetime.datetime(2000, 1, 1), # Do not pass values before this date todate=datetime.datetime(2000, 12, 31), # Do not pass values after this date reverse=False) # Add the Data Feed to Cerebro cerebro.adddata(data) # Set our desired cash start cerebro.broker.setcash(1000.0) # Add a FixedSize sizer according to the stake cerebro.addsizer(bt.sizers.FixedSize, stake=10) # Set the commission cerebro.broker.setcommission(commission=0.0) # Print out the starting conditions print('Starting Portfolio Value: %.2f' % cerebro.broker.getvalue()) # Run over everything cerebro.run() # Print out the final result print('Final Portfolio Value: %.2f' % cerebro.broker.getvalue()) # Plot the result cerebro.plot() After the execution the output is: Starting Portfolio Value: 1000.00 2000-02-18T00:00:00, Close, 27.61 2000-02-22T00:00:00, Close, 27.97 2000-02-22T00:00:00, BUY CREATE, 27.97 2000-02-23T00:00:00, BUY EXECUTED, Size 10, Price: 28.38, Cost: 283.80, Commission 0.00 2000-02-23T00:00:00, Close, 29.73 ... ... ... 2000-12-21T00:00:00, BUY CREATE, 27.82 2000-12-22T00:00:00, BUY EXECUTED, Size 10, Price: 28.65, Cost: 286.50, Commission 0.00 2000-12-22T00:00:00, Close, 30.06 2000-12-26T00:00:00, Close, 29.17 2000-12-27T00:00:00, Close, 28.94 2000-12-28T00:00:00, Close, 29.29 2000-12-29T00:00:00, Close, 27.41 2000-12-29T00:00:00, SELL CREATE, 27.41 Final Portfolio Value: 981.00 The final result has changed even if the logic hasn’t. This is true but the logic has not been applied to the same number of bars. Note As explained before, the platform will first call next when all indicators are ready to produce a value. In this plotting example (very clear in the chart) the MACD is the last indicator to be fully ready (all 3 lines producing an output). The 1st BUY order is no longer scheduled during Jan 2000 but close to the end of Feb 2000. The chart: image Let’s Optimize Many trading books say each market and each traded stock (or commodity or ..) have different rythms. That there is no such thing as a one size fits all. Before the plotting sample, when the strategy started using an indicator the period default value was 15 bars. It’s a strategy parameter and this can be used in an optimization to change the value of the parameter and see which one better fits the market. Note There is plenty of literature about Optimization and associated pros and cons. But the advice will always point in the same direction: do not overoptimize. If a trading idea is not sound, optimizing may end producing a positive result which is only valid for the backtested dataset. The sample is modified to optimize the period of the Simple Moving Average. For the sake of clarity any output with regards to Buy/Sell orders has been removed The example now: from __future__ import (absolute_import, division, print_function, unicode_literals) import datetime # For datetime objects import os.path # To manage paths import sys # To find out the script name (in argv[0]) # Import the backtrader platform import backtrader as bt # Create a Stratey class TestStrategy(bt.Strategy): params = ( ('maperiod', 15), ('printlog', False), ) def log(self, txt, dt=None, doprint=False): ''' Logging function fot this strategy''' if self.params.printlog or doprint: dt = dt or self.datas[0].datetime.date(0) print('%s, %s' % (dt.isoformat(), txt)) def __init__(self): # Keep a reference to the "close" line in the data[0] dataseries self.dataclose = self.datas[0].close # To keep track of pending orders and buy price/commission self.order = None self.buyprice = None self.buycomm = None # Add a MovingAverageSimple indicator self.sma = bt.indicators.SimpleMovingAverage( self.datas[0], period=self.params.maperiod) def notify_order(self, order): if order.status in [order.Submitted, order.Accepted]: # Buy/Sell order submitted/accepted to/by broker - Nothing to do return # Check if an order has been completed # Attention: broker could reject order if not enough cash if order.status in [order.Completed]: if order.isbuy(): self.log( 'BUY EXECUTED, Price: %.2f, Cost: %.2f, Comm %.2f' % (order.executed.price, order.executed.value, order.executed.comm)) self.buyprice = order.executed.price self.buycomm = order.executed.comm else: # Sell self.log('SELL EXECUTED, Price: %.2f, Cost: %.2f, Comm %.2f' % (order.executed.price, order.executed.value, order.executed.comm)) self.bar_executed = len(self) elif order.status in [order.Canceled, order.Margin, order.Rejected]: self.log('Order Canceled/Margin/Rejected') # Write down: no pending order self.order = None def notify_trade(self, trade): if not trade.isclosed: return self.log('OPERATION PROFIT, GROSS %.2f, NET %.2f' % (trade.pnl, trade.pnlcomm)) def next(self): # Simply log the closing price of the series from the reference self.log('Close, %.2f' % self.dataclose[0]) # Check if an order is pending ... if yes, we cannot send a 2nd one if self.order: return # Check if we are in the market if not self.position: # Not yet ... we MIGHT BUY if ... if self.dataclose[0] > self.sma[0]: # BUY, BUY, BUY!!! (with all possible default parameters) self.log('BUY CREATE, %.2f' % self.dataclose[0]) # Keep track of the created order to avoid a 2nd order self.order = self.buy() else: if self.dataclose[0] < self.sma[0]: # SELL, SELL, SELL!!! (with all possible default parameters) self.log('SELL CREATE, %.2f' % self.dataclose[0]) # Keep track of the created order to avoid a 2nd order self.order = self.sell() def stop(self): self.log('(MA Period %2d) Ending Value %.2f' % (self.params.maperiod, self.broker.getvalue()), doprint=True) if __name__ == '__main__': # Create a cerebro entity cerebro = bt.Cerebro() # Add a strategy strats = cerebro.optstrategy( TestStrategy, maperiod=range(10, 31)) # Datas are in a subfolder of the samples. Need to find where the script is # because it could have been called from anywhere modpath = os.path.dirname(os.path.abspath(sys.argv[0])) datapath = os.path.join(modpath, '../../datas/orcl-1995-2014.txt') # Create a Data Feed data = bt.feeds.YahooFinanceCSVData( dataname=datapath, # Do not pass values before this date fromdate=datetime.datetime(2000, 1, 1), # Do not pass values before this date todate=datetime.datetime(2000, 12, 31), # Do not pass values after this date reverse=False) # Add the Data Feed to Cerebro cerebro.adddata(data) # Set our desired cash start cerebro.broker.setcash(1000.0) # Add a FixedSize sizer according to the stake cerebro.addsizer(bt.sizers.FixedSize, stake=10) # Set the commission cerebro.broker.setcommission(commission=0.0) # Run over everything cerebro.run(maxcpus=1) Instead of calling addstrategy to add a stratey class to Cerebro, the call is made to optstrategy. And instead of passing a value a range of values is passed. One of the “Strategy” hooks is added, the stop method, which will be called when the data has been exhausted and backtesting is over. It’s used to print the final net value of the portfolio in the broker (it was done in Cerebro previously) The system will execute the strategy for each value of the range. The following will be output: 2000-12-29, (MA Period 10) Ending Value 880.30 2000-12-29, (MA Period 11) Ending Value 880.00 2000-12-29, (MA Period 12) Ending Value 830.30 2000-12-29, (MA Period 13) Ending Value 893.90 2000-12-29, (MA Period 14) Ending Value 896.90 2000-12-29, (MA Period 15) Ending Value 973.90 2000-12-29, (MA Period 16) Ending Value 959.40 2000-12-29, (MA Period 17) Ending Value 949.80 2000-12-29, (MA Period 18) Ending Value 1011.90 2000-12-29, (MA Period 19) Ending Value 1041.90 2000-12-29, (MA Period 20) Ending Value 1078.00 2000-12-29, (MA Period 21) Ending Value 1058.80 2000-12-29, (MA Period 22) Ending Value 1061.50 2000-12-29, (MA Period 23) Ending Value 1023.00 2000-12-29, (MA Period 24) Ending Value 1020.10 2000-12-29, (MA Period 25) Ending Value 1013.30 2000-12-29, (MA Period 26) Ending Value 998.30 2000-12-29, (MA Period 27) Ending Value 982.20 2000-12-29, (MA Period 28) Ending Value 975.70 2000-12-29, (MA Period 29) Ending Value 983.30 2000-12-29, (MA Period 30) Ending Value 979.80 Results: For periods below 18 the strategy (commissionless) loses money. For periods between 18 and 26 (both included) the strategy makes money. Above 26 money is lost again. And the winning period for this strategy and the given data set is: 20 bars, which wins 78.00 units over 1000 $/€ (a 7.8%) Note The extra indicators from the plotting example have been removed and the start of operations is only influenced by the Simple Moving Average which is being optimized. Hence the slightly different results for period 15 Conclusion The incremental samples have shown how to go from a barebones script to a fully working trading system which even plots the results and can be optimized. A lot more can be done to try to improve the chances of winning: Self defined Indicators Creating an indicator is easy (and even plotting them is easy) Sizers Money Management is for many the key to success Order Types (limit, stop, stoplimit) Some others To ensure all the above items can be fully utilized the documentation provides an insight into them (and other topics) Look in the table of contents and keep on reading … and developing. Best of luck
07-08
from data import COCODetection, get_label_map, MEANS, COLORS from yolact import Yolact from utils.augmentations import BaseTransform, FastBaseTransform, Resize from utils.functions import MovingAverage, ProgressBar from layers.box_utils import jaccard, center_size, mask_iou from utils import timer from utils.functions import SavePath from layers.output_utils import postprocess, undo_image_transformation import pycocotools from data import cfg, set_cfg, set_dataset import numpy as np import torch import torch.backends.cudnn as cudnn from torch.autograd import Variable import argparse import time import random import cProfile import pickle import json import os from collections import defaultdict from pathlib import Path from collections import OrderedDict from PIL import Image import matplotlib.pyplot as plt import cv2 def str2bool(v): if v.lower() in ('yes', 'true', 't', 'y', '1'): return True elif v.lower() in ('no', 'false', 'f', 'n', '0'): return False else: raise argparse.ArgumentTypeError('Boolean value expected.') def parse_args(argv=None): parser = argparse.ArgumentParser( description='YOLACT COCO Evaluation') parser.add_argument('--trained_model', default='weights/yolact_base_105_101798_interrupt.pth', type=str, help='Trained state_dict file path to open. If "interrupt", this will open the interrupt file.') parser.add_argument('--top_k', default=5, type=int, help='Further restrict the number of predictions to parse') parser.add_argument('--cuda', default=True, type=str2bool, help='Use cuda to evaulate model') parser.add_argument('--fast_nms', default=True, type=str2bool, help='Whether to use a faster, but not entirely correct version of NMS.') parser.add_argument('--cross_class_nms', default=False, type=str2bool, help='Whether compute NMS cross-class or per-class.') parser.add_argument('--display_masks', default=True, type=str2bool, help='Whether or not to display masks over bounding boxes') parser.add_argument('--display_bboxes', default=True, type=str2bool, help='Whether or not to display bboxes around masks') parser.add_argument('--display_text', default=True, type=str2bool, help='Whether or not to display text (class [score])') parser.add_argument('--display_scores', default=True, type=str2bool, help='Whether or not to display scores in addition to classes') parser.add_argument('--display', dest='display', action='store_true', help='Display qualitative results instead of quantitative ones.') parser.add_argument('--shuffle', dest='shuffle', action='store_true', help='Shuffles the images when displaying them. Doesn\'t have much of an effect when display is off though.') parser.add_argument('--ap_data_file', default='results/ap_data.pkl', type=str, help='In quantitative mode, the file to save detections before calculating mAP.') parser.add_argument('--resume', dest='resume', action='store_true', help='If display not set, this resumes mAP calculations from the ap_data_file.') parser.add_argument('--max_images', default=-1, type=int, help='The maximum number of images from the dataset to consider. Use -1 for all.') parser.add_argument('--output_coco_json', dest='output_coco_json', action='store_true', help='If display is not set, instead of processing IoU values, this just dumps detections into the coco json file.') parser.add_argument('--bbox_det_file', default='results/bbox_detections.json', type=str, help='The output file for coco bbox results if --coco_results is set.') parser.add_argument('--mask_det_file', default='results/mask_detections.json', type=str, help='The output file for coco mask results if --coco_results is set.') parser.add_argument('--config', default=None, help='The config object to use.') parser.add_argument('--output_web_json', dest='output_web_json', action='store_true', help='If display is not set, instead of processing IoU values, this dumps detections for usage with the detections viewer web thingy.') parser.add_argument('--web_det_path', default='web/dets/', type=str, help='If output_web_json is set, this is the path to dump detections into.') parser.add_argument('--no_bar', dest='no_bar', action='store_true', help='Do not output the status bar. This is useful for when piping to a file.') parser.add_argument('--display_lincomb', default=False, type=str2bool, help='If the config uses lincomb masks, output a visualization of how those masks are created.') parser.add_argument('--benchmark', default=False, dest='benchmark', action='store_true', help='Equivalent to running display mode but without displaying an image.') parser.add_argument('--no_sort', default=False, dest='no_sort', action='store_true', help='Do not sort images by hashed image ID.') parser.add_argument('--seed', default=None, type=int, help='The seed to pass into random.seed. Note: this is only really for the shuffle and does not (I think) affect cuda stuff.') parser.add_argument('--mask_proto_debug', default=False, dest='mask_proto_debug', action='store_true', help='Outputs stuff for scripts/compute_mask.py.') parser.add_argument('--no_crop', default=False, dest='crop', action='store_false', help='Do not crop output masks with the predicted bounding box.') parser.add_argument('--image', default=None, type=str, help='A path to an image to use for display.') parser.add_argument('--images', default='E:/yolact-master/coco/images/train2017', type=str, help='Input and output paths separated by a colon.') parser.add_argument('--video', default=None, type=str, help='A path to a video to evaluate on. Passing in a number will use that index webcam.') parser.add_argument('--video_multiframe', default=1, type=int, help='The number of frames to evaluate in parallel to make videos play at higher fps.') parser.add_argument('--score_threshold', default=0.15, type=float, help='Detections with a score under this threshold will not be considered. This currently only works in display mode.') parser.add_argument('--dataset', default=None, type=str, help='If specified, override the dataset specified in the config with this one (example: coco2017_dataset).') parser.add_argument('--detect', default=False, dest='detect', action='store_true', help='Don\'t evauluate the mask branch at all and only do object detection. This only works for --display and --benchmark.') parser.add_argument('--display_fps', default=False, dest='display_fps', action='store_true', help='When displaying / saving video, draw the FPS on the frame') parser.add_argument('--emulate_playback', default=False, dest='emulate_playback', action='store_true', help='When saving a video, emulate the framerate that you\'d get running in real-time mode.') parser.set_defaults(no_bar=False, display=False, resume=False, output_coco_json=False, output_web_json=False, shuffle=False, benchmark=False, no_sort=False, no_hash=False, mask_proto_debug=False, crop=True, detect=False, display_fps=False, emulate_playback=False) global args args = parser.parse_args(argv) if args.output_web_json: args.output_coco_json = True if args.seed is not None: random.seed(args.seed) iou_thresholds = [x / 100 for x in range(50, 100, 5)] coco_cats = {} # Call prep_coco_cats to fill this coco_cats_inv = {} color_cache = defaultdict(lambda: {}) def prep_display(dets_out, img, h, w, undo_transform=True, class_color=False, mask_alpha=0.45, fps_str=''): """ Note: If undo_transform=False then im_h and im_w are allowed to be None. """ if undo_transform: img_numpy = undo_image_transformation(img, w, h) img_gpu = torch.Tensor(img_numpy).cuda() else: img_gpu = img / 255.0 h, w, _ = img.shape with timer.env('Postprocess'): save = cfg.rescore_bbox cfg.rescore_bbox = True t = postprocess(dets_out, w, h, visualize_lincomb = args.display_lincomb, crop_masks = args.crop, score_threshold = args.score_threshold) cfg.rescore_bbox = save with timer.env('Copy'): idx = t[1].argsort(0, descending=True)[:args.top_k] if cfg.eval_mask_branch: # Masks are drawn on the GPU, so don't copy masks = t[3][idx] classes, scores, boxes = [x[idx].cpu().numpy() for x in t[:3]] num_dets_to_consider = min(args.top_k, classes.shape[0]) for j in range(num_dets_to_consider): if scores[j] < args.score_threshold: num_dets_to_consider = j break # Quick and dirty lambda for selecting the color for a particular index # Also keeps track of a per-gpu color cache for maximum speed def get_color(j, on_gpu=None): global color_cache color_idx = (classes[j] * 5 if class_color else j * 5) % len(COLORS) if on_gpu is not None and color_idx in color_cache[on_gpu]: return color_cache[on_gpu][color_idx] else: color = COLORS[color_idx] if not undo_transform: # The image might come in as RGB or BRG, depending color = (color[2], color[1], color[0]) if on_gpu is not None: color = torch.Tensor(color).to(on_gpu).float() / 255. color_cache[on_gpu][color_idx] = color return color # First, draw the masks on the GPU where we can do it really fast # Beware: very fast but possibly unintelligible mask-drawing code ahead # I wish I had access to OpenGL or Vulkan but alas, I guess Pytorch tensor operations will have to suffice if args.display_masks and cfg.eval_mask_branch and num_dets_to_consider > 0: # After this, mask is of size [num_dets, h, w, 1] masks = masks[:num_dets_to_consider, :, :, None] # Prepare the RGB images for each mask given their color (size [num_dets, h, w, 1]) colors = torch.cat([get_color(j, on_gpu=img_gpu.device.index).view(1, 1, 1, 3) for j in range(num_dets_to_consider)], dim=0) masks_color = masks.repeat(1, 1, 1, 3) * colors * mask_alpha # This is 1 everywhere except for 1-mask_alpha where the mask is inv_alph_masks = masks * (-mask_alpha) + 1 # I did the math for this on pen and paper. This whole block should be equivalent to: # for j in range(num_dets_to_consider): # img_gpu = img_gpu * inv_alph_masks[j] + masks_color[j] masks_color_summand = masks_color[0] if num_dets_to_consider > 1: inv_alph_cumul = inv_alph_masks[:(num_dets_to_consider-1)].cumprod(dim=0) masks_color_cumul = masks_color[1:] * inv_alph_cumul masks_color_summand += masks_color_cumul.sum(dim=0) img_gpu = img_gpu * inv_alph_masks.prod(dim=0) + masks_color_summand if args.display_fps: # Draw the box for the fps on the GPU font_face = cv2.FONT_HERSHEY_DUPLEX font_scale = 0.6 font_thickness = 1 text_w, text_h = cv2.getTextSize(fps_str, font_face, font_scale, font_thickness)[0] img_gpu[0:text_h+8, 0:text_w+8] *= 0.6 # 1 - Box alpha # Then draw the stuff that needs to be done on the cpu # Note, make sure this is a uint8 tensor or opencv will not anti alias text for whatever reason img_numpy = (img_gpu * 255).byte().cpu().numpy() if args.display_fps: # Draw the text on the CPU text_pt = (4, text_h + 2) text_color = [255, 255, 255] cv2.putText(img_numpy, fps_str, text_pt, font_face, font_scale, text_color, font_thickness, cv2.LINE_AA) if num_dets_to_consider == 0: return img_numpy if args.display_text or args.display_bboxes: for j in reversed(range(num_dets_to_consider)): x1, y1, x2, y2 = boxes[j, :] color = get_color(j) score = scores[j] if args.display_bboxes: cv2.rectangle(img_numpy, (x1, y1), (x2, y2), color, 1) if args.display_text: _class = cfg.dataset.class_names[classes[j]] text_str = '%s: %.2f' % (_class, score) if args.display_scores else _class font_face = cv2.FONT_HERSHEY_DUPLEX font_scale = 0.6 font_thickness = 1 text_w, text_h = cv2.getTextSize(text_str, font_face, font_scale, font_thickness)[0] text_pt = (x1, y1 - 3) text_color = [255, 255, 255] cv2.rectangle(img_numpy, (x1, y1), (x1 + text_w, y1 - text_h - 4), color, -1) cv2.putText(img_numpy, text_str, text_pt, font_face, font_scale, text_color, font_thickness, cv2.LINE_AA) return img_numpy def prep_benchmark(dets_out, h, w): with timer.env('Postprocess'): t = postprocess(dets_out, w, h, crop_masks=args.crop, score_threshold=args.score_threshold) with timer.env('Copy'): classes, scores, boxes, masks = [x[:args.top_k] for x in t] if isinstance(scores, list): box_scores = scores[0].cpu().numpy() mask_scores = scores[1].cpu().numpy() else: scores = scores.cpu().numpy() classes = classes.cpu().numpy() boxes = boxes.cpu().numpy() masks = masks.cpu().numpy() with timer.env('Sync'): # Just in case torch.cuda.synchronize() def prep_coco_cats(): """ Prepare inverted table for category id lookup given a coco cats object. """ for coco_cat_id, transformed_cat_id_p1 in get_label_map().items(): transformed_cat_id = transformed_cat_id_p1 - 1 coco_cats[transformed_cat_id] = coco_cat_id coco_cats_inv[coco_cat_id] = transformed_cat_id def get_coco_cat(transformed_cat_id): """ transformed_cat_id is [0,80) as indices in cfg.dataset.class_names """ return coco_cats[transformed_cat_id] def get_transformed_cat(coco_cat_id): """ transformed_cat_id is [0,80) as indices in cfg.dataset.class_names """ return coco_cats_inv[coco_cat_id] class Detections: def __init__(self): self.bbox_data = [] self.mask_data = [] def add_bbox(self, image_id:int, category_id:int, bbox:list, score:float): """ Note that bbox should be a list or tuple of (x1, y1, x2, y2) """ bbox = [bbox[0], bbox[1], bbox[2]-bbox[0], bbox[3]-bbox[1]] # Round to the nearest 10th to avoid huge file sizes, as COCO suggests bbox = [round(float(x)*10)/10 for x in bbox] self.bbox_data.append({ 'image_id': int(image_id), 'category_id': get_coco_cat(int(category_id)), 'bbox': bbox, 'score': float(score) }) def add_mask(self, image_id:int, category_id:int, segmentation:np.ndarray, score:float): """ The segmentation should be the full mask, the size of the image and with size [h, w]. """ rle = pycocotools.mask.encode(np.asfortranarray(segmentation.astype(np.uint8))) rle['counts'] = rle['counts'].decode('ascii') # json.dump doesn't like bytes strings self.mask_data.append({ 'image_id': int(image_id), 'category_id': get_coco_cat(int(category_id)), 'segmentation': rle, 'score': float(score) }) def dump(self): dump_arguments = [ (self.bbox_data, args.bbox_det_file), (self.mask_data, args.mask_det_file) ] for data, path in dump_arguments: with open(path, 'w') as f: json.dump(data, f) def dump_web(self): """ Dumps it in the format for my web app. Warning: bad code ahead! """ config_outs = ['preserve_aspect_ratio', 'use_prediction_module', 'use_yolo_regressors', 'use_prediction_matching', 'train_masks'] output = { 'info' : { 'Config': {key: getattr(cfg, key) for key in config_outs}, } } image_ids = list(set([x['image_id'] for x in self.bbox_data])) image_ids.sort() image_lookup = {_id: idx for idx, _id in enumerate(image_ids)} output['images'] = [{'image_id': image_id, 'dets': []} for image_id in image_ids] # These should already be sorted by score with the way prep_metrics works. for bbox, mask in zip(self.bbox_data, self.mask_data): image_obj = output['images'][image_lookup[bbox['image_id']]] image_obj['dets'].append({ 'score': bbox['score'], 'bbox': bbox['bbox'], 'category': cfg.dataset.class_names[get_transformed_cat(bbox['category_id'])], 'mask': mask['segmentation'], }) with open(os.path.join(args.web_det_path, '%s.json' % cfg.name), 'w') as f: json.dump(output, f) def _mask_iou(mask1, mask2, iscrowd=False): with timer.env('Mask IoU'): ret = mask_iou(mask1, mask2, iscrowd) return ret.cpu() def _bbox_iou(bbox1, bbox2, iscrowd=False): with timer.env('BBox IoU'): ret = jaccard(bbox1, bbox2, iscrowd) return ret.cpu() def prep_metrics(ap_data, dets, img, gt, gt_masks, h, w, num_crowd, image_id, detections:Detections=None): """ Returns a list of APs for this image, with each element being for a class """ if not args.output_coco_json: with timer.env('Prepare gt'): gt_boxes = torch.Tensor(gt[:, :4]) gt_boxes[:, [0, 2]] *= w gt_boxes[:, [1, 3]] *= h gt_classes = list(gt[:, 4].astype(int)) gt_masks = torch.Tensor(gt_masks).view(-1, h*w) if num_crowd > 0: split = lambda x: (x[-num_crowd:], x[:-num_crowd]) crowd_boxes , gt_boxes = split(gt_boxes) crowd_masks , gt_masks = split(gt_masks) crowd_classes, gt_classes = split(gt_classes) with timer.env('Postprocess'): classes, scores, boxes, masks = postprocess(dets, w, h, crop_masks=args.crop, score_threshold=args.score_threshold) if classes.size(0) == 0: return classes = list(classes.cpu().numpy().astype(int)) if isinstance(scores, list): box_scores = list(scores[0].cpu().numpy().astype(float)) mask_scores = list(scores[1].cpu().numpy().astype(float)) else: scores = list(scores.cpu().numpy().astype(float)) box_scores = scores mask_scores = scores masks = masks.view(-1, h*w).cuda() boxes = boxes.cuda() if args.output_coco_json: with timer.env('JSON Output'): boxes = boxes.cpu().numpy() masks = masks.view(-1, h, w).cpu().numpy() for i in range(masks.shape[0]): # Make sure that the bounding box actually makes sense and a mask was produced if (boxes[i, 3] - boxes[i, 1]) * (boxes[i, 2] - boxes[i, 0]) > 0: detections.add_bbox(image_id, classes[i], boxes[i,:], box_scores[i]) detections.add_mask(image_id, classes[i], masks[i,:,:], mask_scores[i]) return with timer.env('Eval Setup'): num_pred = len(classes) num_gt = len(gt_classes) mask_iou_cache = _mask_iou(masks, gt_masks) bbox_iou_cache = _bbox_iou(boxes.float(), gt_boxes.float()) if num_crowd > 0: crowd_mask_iou_cache = _mask_iou(masks, crowd_masks, iscrowd=True) crowd_bbox_iou_cache = _bbox_iou(boxes.float(), crowd_boxes.float(), iscrowd=True) else: crowd_mask_iou_cache = None crowd_bbox_iou_cache = None box_indices = sorted(range(num_pred), key=lambda i: -box_scores[i]) mask_indices = sorted(box_indices, key=lambda i: -mask_scores[i]) iou_types = [ ('box', lambda i,j: bbox_iou_cache[i, j].item(), lambda i,j: crowd_bbox_iou_cache[i,j].item(), lambda i: box_scores[i], box_indices), ('mask', lambda i,j: mask_iou_cache[i, j].item(), lambda i,j: crowd_mask_iou_cache[i,j].item(), lambda i: mask_scores[i], mask_indices) ] timer.start('Main loop') for _class in set(classes + gt_classes): ap_per_iou = [] num_gt_for_class = sum([1 for x in gt_classes if x == _class]) for iouIdx in range(len(iou_thresholds)): iou_threshold = iou_thresholds[iouIdx] for iou_type, iou_func, crowd_func, score_func, indices in iou_types: gt_used = [False] * len(gt_classes) ap_obj = ap_data[iou_type][iouIdx][_class] ap_obj.add_gt_positives(num_gt_for_class) for i in indices: if classes[i] != _class: continue max_iou_found = iou_threshold max_match_idx = -1 for j in range(num_gt): if gt_used[j] or gt_classes[j] != _class: continue iou = iou_func(i, j) if iou > max_iou_found: max_iou_found = iou max_match_idx = j if max_match_idx >= 0: gt_used[max_match_idx] = True ap_obj.push(score_func(i), True) else: # If the detection matches a crowd, we can just ignore it matched_crowd = False if num_crowd > 0: for j in range(len(crowd_classes)): if crowd_classes[j] != _class: continue iou = crowd_func(i, j) if iou > iou_threshold: matched_crowd = True break # All this crowd code so that we can make sure that our eval code gives the # same result as COCOEval. There aren't even that many crowd annotations to # begin with, but accuracy is of the utmost importance. if not matched_crowd: ap_obj.push(score_func(i), False) timer.stop('Main loop') class APDataObject: """ Stores all the information necessary to calculate the AP for one IoU and one class. Note: I type annotated this because why not. """ def __init__(self): self.data_points = [] self.num_gt_positives = 0 def push(self, score:float, is_true:bool): self.data_points.append((score, is_true)) def add_gt_positives(self, num_positives:int): """ Call this once per image. """ self.num_gt_positives += num_positives def is_empty(self) -> bool: return len(self.data_points) == 0 and self.num_gt_positives == 0 def get_ap(self) -> float: """ Warning: result not cached. """ if self.num_gt_positives == 0: return 0 # Sort descending by score self.data_points.sort(key=lambda x: -x[0]) precisions = [] recalls = [] num_true = 0 num_false = 0 # Compute the precision-recall curve. The x axis is recalls and the y axis precisions. for datum in self.data_points: # datum[1] is whether the detection a true or false positive if datum[1]: num_true += 1 else: num_false += 1 precision = num_true / (num_true + num_false) recall = num_true / self.num_gt_positives precisions.append(precision) recalls.append(recall) # Smooth the curve by computing [max(precisions[i:]) for i in range(len(precisions))] # Basically, remove any temporary dips from the curve. # At least that's what I think, idk. COCOEval did it so I do too. for i in range(len(precisions)-1, 0, -1): if precisions[i] > precisions[i-1]: precisions[i-1] = precisions[i] # Compute the integral of precision(recall) d_recall from recall=0->1 using fixed-length riemann summation with 101 bars. y_range = [0] * 101 # idx 0 is recall == 0.0 and idx 100 is recall == 1.00 x_range = np.array([x / 100 for x in range(101)]) recalls = np.array(recalls) # I realize this is weird, but all it does is find the nearest precision(x) for a given x in x_range. # Basically, if the closest recall we have to 0.01 is 0.009 this sets precision(0.01) = precision(0.009). # I approximate the integral this way, because that's how COCOEval does it. indices = np.searchsorted(recalls, x_range, side='left') for bar_idx, precision_idx in enumerate(indices): if precision_idx < len(precisions): y_range[bar_idx] = precisions[precision_idx] # Finally compute the riemann sum to get our integral. # avg([precision(x) for x in 0:0.01:1]) return sum(y_range) / len(y_range) def badhash(x): """ Just a quick and dirty hash function for doing a deterministic shuffle based on image_id. Source: https://stackoverflow.com/questions/664014/what-integer-hash-function-are-good-that-accepts-an-integer-hash-key """ x = (((x >> 16) ^ x) * 0x045d9f3b) & 0xFFFFFFFF x = (((x >> 16) ^ x) * 0x045d9f3b) & 0xFFFFFFFF x = ((x >> 16) ^ x) & 0xFFFFFFFF return x def evalimage(net:Yolact, path:str, save_path:str=None): frame = torch.from_numpy(cv2.imread(path)).cuda().float() batch = FastBaseTransform()(frame.unsqueeze(0)) preds = net(batch) img_numpy = prep_display(preds, frame, None, None, undo_transform=False) if save_path is None: img_numpy = img_numpy[:, :, (2, 1, 0)] if save_path is None: plt.imshow(img_numpy) plt.title(path) plt.show() else: cv2.imwrite(save_path, img_numpy) def evalimages(net:Yolact, input_folder:str, output_folder:str): if not os.path.exists(output_folder): os.mkdir(output_folder) print() for p in Path(input_folder).glob('*'): path = str(p) name = os.path.basename(path) name = '.'.join(name.split('.')[:-1]) + '.png' out_path = os.path.join(output_folder, name) evalimage(net, path, out_path) print(path + ' -> ' + out_path) print('Done.') from multiprocessing.pool import ThreadPool from queue import Queue class CustomDataParallel(torch.nn.DataParallel): """ A Custom Data Parallel class that properly gathers lists of dictionaries. """ def gather(self, outputs, output_device): # Note that I don't actually want to convert everything to the output_device return sum(outputs, []) def evalvideo(net:Yolact, path:str, out_path:str=None): # If the path is a digit, parse it as a webcam index is_webcam = path.isdigit() # If the input image size is constant, this make things faster (hence why we can use it in a video setting). cudnn.benchmark = True if is_webcam: vid = cv2.VideoCapture(int(path)) else: vid = cv2.VideoCapture(path) if not vid.isOpened(): print('Could not open video "%s"' % path) exit(-1) target_fps = round(vid.get(cv2.CAP_PROP_FPS)) frame_width = round(vid.get(cv2.CAP_PROP_FRAME_WIDTH)) frame_height = round(vid.get(cv2.CAP_PROP_FRAME_HEIGHT)) if is_webcam: num_frames = float('inf') else: num_frames = round(vid.get(cv2.CAP_PROP_FRAME_COUNT)) net = CustomDataParallel(net).cuda() transform = torch.nn.DataParallel(FastBaseTransform()).cuda() frame_times = MovingAverage(100) fps = 0 frame_time_target = 1 / target_fps running = True fps_str = '' vid_done = False frames_displayed = 0 if out_path is not None: out = cv2.VideoWriter(out_path, cv2.VideoWriter_fourcc(*"mp4v"), target_fps, (frame_width, frame_height)) def cleanup_and_exit(): print() pool.terminate() vid.release() if out_path is not None: out.release() cv2.destroyAllWindows() exit() def get_next_frame(vid): frames = [] for idx in range(args.video_multiframe): frame = vid.read()[1] if frame is None: return frames frames.append(frame) return frames def transform_frame(frames): with torch.no_grad(): frames = [torch.from_numpy(frame).cuda().float() for frame in frames] return frames, transform(torch.stack(frames, 0)) def eval_network(inp): with torch.no_grad(): frames, imgs = inp num_extra = 0 while imgs.size(0) < args.video_multiframe: imgs = torch.cat([imgs, imgs[0].unsqueeze(0)], dim=0) num_extra += 1 out = net(imgs) if num_extra > 0: out = out[:-num_extra] return frames, out def prep_frame(inp, fps_str): with torch.no_grad(): frame, preds = inp return prep_display(preds, frame, None, None, undo_transform=False, class_color=True, fps_str=fps_str) frame_buffer = Queue() video_fps = 0 # All this timing code to make sure that def play_video(): try: nonlocal frame_buffer, running, video_fps, is_webcam, num_frames, frames_displayed, vid_done video_frame_times = MovingAverage(100) frame_time_stabilizer = frame_time_target last_time = None stabilizer_step = 0.0005 progress_bar = ProgressBar(30, num_frames) while running: frame_time_start = time.time() if not frame_buffer.empty(): next_time = time.time() if last_time is not None: video_frame_times.add(next_time - last_time) video_fps = 1 / video_frame_times.get_avg() if out_path is None: cv2.imshow(path, frame_buffer.get()) else: out.write(frame_buffer.get()) frames_displayed += 1 last_time = next_time if out_path is not None: if video_frame_times.get_avg() == 0: fps = 0 else: fps = 1 / video_frame_times.get_avg() progress = frames_displayed / num_frames * 100 progress_bar.set_val(frames_displayed) print('\rProcessing Frames %s %6d / %6d (%5.2f%%) %5.2f fps ' % (repr(progress_bar), frames_displayed, num_frames, progress, fps), end='') # This is split because you don't want savevideo to require cv2 display functionality (see #197) if out_path is None and cv2.waitKey(1) == 27: # Press Escape to close running = False if not (frames_displayed < num_frames): running = False if not vid_done: buffer_size = frame_buffer.qsize() if buffer_size < args.video_multiframe: frame_time_stabilizer += stabilizer_step elif buffer_size > args.video_multiframe: frame_time_stabilizer -= stabilizer_step if frame_time_stabilizer < 0: frame_time_stabilizer = 0 new_target = frame_time_stabilizer if is_webcam else max(frame_time_stabilizer, frame_time_target) else: new_target = frame_time_target next_frame_target = max(2 * new_target - video_frame_times.get_avg(), 0) target_time = frame_time_start + next_frame_target - 0.001 # Let's just subtract a millisecond to be safe if out_path is None or args.emulate_playback: # This gives more accurate timing than if sleeping the whole amount at once while time.time() < target_time: time.sleep(0.001) else: # Let's not starve the main thread, now time.sleep(0.001) except: # See issue #197 for why this is necessary import traceback traceback.print_exc() extract_frame = lambda x, i: (x[0][i] if x[1][i]['detection'] is None else x[0][i].to(x[1][i]['detection']['box'].device), [x[1][i]]) # Prime the network on the first frame because I do some thread unsafe things otherwise print('Initializing model... ', end='') first_batch = eval_network(transform_frame(get_next_frame(vid))) print('Done.') # For each frame the sequence of functions it needs to go through to be processed (in reversed order) sequence = [prep_frame, eval_network, transform_frame] pool = ThreadPool(processes=len(sequence) + args.video_multiframe + 2) pool.apply_async(play_video) active_frames = [{'value': extract_frame(first_batch, i), 'idx': 0} for i in range(len(first_batch[0]))] print() if out_path is None: print('Press Escape to close.') try: while vid.isOpened() and running: # Hard limit on frames in buffer so we don't run out of memory >.> while frame_buffer.qsize() > 100: time.sleep(0.001) start_time = time.time() # Start loading the next frames from the disk if not vid_done: next_frames = pool.apply_async(get_next_frame, args=(vid,)) else: next_frames = None if not (vid_done and len(active_frames) == 0): # For each frame in our active processing queue, dispatch a job # for that frame using the current function in the sequence for frame in active_frames: _args = [frame['value']] if frame['idx'] == 0: _args.append(fps_str) frame['value'] = pool.apply_async(sequence[frame['idx']], args=_args) # For each frame whose job was the last in the sequence (i.e. for all final outputs) for frame in active_frames: if frame['idx'] == 0: frame_buffer.put(frame['value'].get()) # Remove the finished frames from the processing queue active_frames = [x for x in active_frames if x['idx'] > 0] # Finish evaluating every frame in the processing queue and advanced their position in the sequence for frame in list(reversed(active_frames)): frame['value'] = frame['value'].get() frame['idx'] -= 1 if frame['idx'] == 0: # Split this up into individual threads for prep_frame since it doesn't support batch size active_frames += [{'value': extract_frame(frame['value'], i), 'idx': 0} for i in range(1, len(frame['value'][0]))] frame['value'] = extract_frame(frame['value'], 0) # Finish loading in the next frames and add them to the processing queue if next_frames is not None: frames = next_frames.get() if len(frames) == 0: vid_done = True else: active_frames.append({'value': frames, 'idx': len(sequence)-1}) # Compute FPS frame_times.add(time.time() - start_time) fps = args.video_multiframe / frame_times.get_avg() else: fps = 0 fps_str = 'Processing FPS: %.2f | Video Playback FPS: %.2f | Frames in Buffer: %d' % (fps, video_fps, frame_buffer.qsize()) if not args.display_fps: print('\r' + fps_str + ' ', end='') except KeyboardInterrupt: print('\nStopping...') cleanup_and_exit() def evaluate(net:Yolact, dataset, train_mode=False): net.detect.use_fast_nms = args.fast_nms net.detect.use_cross_class_nms = args.cross_class_nms cfg.mask_proto_debug = args.mask_proto_debug # TODO Currently we do not support Fast Mask Re-scroing in evalimage, evalimages, and evalvideo if args.image is not None: if ':' in args.image: inp, out = args.image.split(':') evalimage(net, inp, out) else: evalimage(net, args.image) return elif args.images is not None: inp, out = args.images.split('E:/yolact-master/coco/images/train2017: E:/yolact-master/results/output') evalimages(net, inp, out) return elif args.video is not None: if ':' in args.video: inp, out = args.video.split(':') evalvideo(net, inp, out) else: evalvideo(net, args.video) return frame_times = MovingAverage() dataset_size = len(dataset) if args.max_images < 0 else min(args.max_images, len(dataset)) progress_bar = ProgressBar(30, dataset_size) print() if not args.display and not args.benchmark: # For each class and iou, stores tuples (score, isPositive) # Index ap_data[type][iouIdx][classIdx] ap_data = { 'box' : [[APDataObject() for _ in cfg.dataset.class_names] for _ in iou_thresholds], 'mask': [[APDataObject() for _ in cfg.dataset.class_names] for _ in iou_thresholds] } detections = Detections() else: timer.disable('Load Data') dataset_indices = list(range(len(dataset))) if args.shuffle: random.shuffle(dataset_indices) elif not args.no_sort: # Do a deterministic shuffle based on the image ids # # I do this because on python 3.5 dictionary key order is *random*, while in 3.6 it's # the order of insertion. That means on python 3.6, the images come in the order they are in # in the annotations file. For some reason, the first images in the annotations file are # the hardest. To combat this, I use a hard-coded hash function based on the image ids # to shuffle the indices we use. That way, no matter what python version or how pycocotools # handles the data, we get the same result every time. hashed = [badhash(x) for x in dataset.ids] dataset_indices.sort(key=lambda x: hashed[x]) dataset_indices = dataset_indices[:dataset_size] try: # Main eval loop for it, image_idx in enumerate(dataset_indices): timer.reset() with timer.env('Load Data'): img, gt, gt_masks, h, w, num_crowd = dataset.pull_item(image_idx) # Test flag, do not upvote if cfg.mask_proto_debug: with open('scripts/info.txt', 'w') as f: f.write(str(dataset.ids[image_idx])) np.save('scripts/gt.npy', gt_masks) batch = Variable(img.unsqueeze(0)) if args.cuda: batch = batch.cuda() with timer.env('Network Extra'): preds = net(batch) # Perform the meat of the operation here depending on our mode. if args.display: img_numpy = prep_display(preds, img, h, w) elif args.benchmark: prep_benchmark(preds, h, w) else: prep_metrics(ap_data, preds, img, gt, gt_masks, h, w, num_crowd, dataset.ids[image_idx], detections) # First couple of images take longer because we're constructing the graph. # Since that's technically initialization, don't include those in the FPS calculations. if it > 1: frame_times.add(timer.total_time()) if args.display: if it > 1: print('Avg FPS: %.4f' % (1 / frame_times.get_avg())) plt.imshow(img_numpy) plt.title(str(dataset.ids[image_idx])) plt.show() elif not args.no_bar: if it > 1: fps = 1 / frame_times.get_avg() else: fps = 0 progress = (it+1) / dataset_size * 100 progress_bar.set_val(it+1) print('\rProcessing Images %s %6d / %6d (%5.2f%%) %5.2f fps ' % (repr(progress_bar), it+1, dataset_size, progress, fps), end='') if not args.display and not args.benchmark: print() if args.output_coco_json: print('Dumping detections...') if args.output_web_json: detections.dump_web() else: detections.dump() else: if not train_mode: print('Saving data...') with open(args.ap_data_file, 'wb') as f: pickle.dump(ap_data, f) return calc_map(ap_data) elif args.benchmark: print() print() print('Stats for the last frame:') timer.print_stats() avg_seconds = frame_times.get_avg() print('Average: %5.2f fps, %5.2f ms' % (1 / frame_times.get_avg(), 1000*avg_seconds)) except KeyboardInterrupt: print('Stopping...') def calc_map(ap_data): print('Calculating mAP...') aps = [{'box': [], 'mask': []} for _ in iou_thresholds] for _class in range(len(cfg.dataset.class_names)): for iou_idx in range(len(iou_thresholds)): for iou_type in ('box', 'mask'): ap_obj = ap_data[iou_type][iou_idx][_class] if not ap_obj.is_empty(): aps[iou_idx][iou_type].append(ap_obj.get_ap()) all_maps = {'box': OrderedDict(), 'mask': OrderedDict()} # Looking back at it, this code is really hard to read :/ for iou_type in ('box', 'mask'): all_maps[iou_type]['all'] = 0 # Make this first in the ordereddict for i, threshold in enumerate(iou_thresholds): mAP = sum(aps[i][iou_type]) / len(aps[i][iou_type]) * 100 if len(aps[i][iou_type]) > 0 else 0 all_maps[iou_type][int(threshold*100)] = mAP all_maps[iou_type]['all'] = (sum(all_maps[iou_type].values()) / (len(all_maps[iou_type].values())-1)) print_maps(all_maps) # Put in a prettier format so we can serialize it to json during training all_maps = {k: {j: round(u, 2) for j, u in v.items()} for k, v in all_maps.items()} return all_maps def print_maps(all_maps): # Warning: hacky make_row = lambda vals: (' %5s |' * len(vals)) % tuple(vals) make_sep = lambda n: ('-------+' * n) print() print(make_row([''] + [('.%d ' % x if isinstance(x, int) else x + ' ') for x in all_maps['box'].keys()])) print(make_sep(len(all_maps['box']) + 1)) for iou_type in ('box', 'mask'): print(make_row([iou_type] + ['%.2f' % x if x < 100 else '%.1f' % x for x in all_maps[iou_type].values()])) print(make_sep(len(all_maps['box']) + 1)) print() if __name__ == '__main__': parse_args() if args.config is not None: set_cfg(args.config) if args.trained_model == 'interrupt': args.trained_model = SavePath.get_interrupt('weights/') elif args.trained_model == 'latest': args.trained_model = SavePath.get_latest('weights/', cfg.name) if args.config is None: model_path = SavePath.from_str(args.trained_model) # TODO: Bad practice? Probably want to do a name lookup instead. args.config = model_path.model_name + '_config' print('Config not specified. Parsed %s from the file name.\n' % args.config) set_cfg(args.config) if args.detect: cfg.eval_mask_branch = False if args.dataset is not None: set_dataset(args.dataset) with torch.no_grad(): if not os.path.exists('results'): os.makedirs('results') if args.cuda: cudnn.fastest = True torch.set_default_tensor_type('torch.cuda.FloatTensor') else: torch.set_default_tensor_type('torch.FloatTensor') if args.resume and not args.display: with open(args.ap_data_file, 'rb') as f: ap_data = pickle.load(f) calc_map(ap_data) exit() if args.image is None and args.video is None and args.images is None: dataset = COCODetection(cfg.dataset.valid_images, cfg.dataset.valid_info, transform=BaseTransform(), has_gt=cfg.dataset.has_gt) prep_coco_cats() else: dataset = None print('Loading model...', end='') net = Yolact() net.load_weights(args.trained_model) net.eval() print(' Done.') if args.cuda: net = net.cuda() evaluate(net, dataset) Traceback (most recent call last): File "eval.py", line 1105, in <module> evaluate(net, dataset) File "eval.py", line 884, in evaluate inp, out = args.images.split('E:/yolact-master/coco/images/train2017: E:/yolact-master/results/output') ValueError: not enough values to unpack (expected 2, got 1)
06-18
"""Append module search paths for third-party packages to sys.path. **************************************************************** * This module is automatically imported during initialization. * **************************************************************** This will append site-specific paths to the module search path. On Unix (including Mac OSX), it starts with sys.prefix and sys.exec_prefix (if different) and appends lib/python<version>/site-packages. On other platforms (such as Windows), it tries each of the prefixes directly, as well as with lib/site-packages appended. The resulting directories, if they exist, are appended to sys.path, and also inspected for path configuration files. If a file named "pyvenv.cfg" exists one directory above sys.executable, sys.prefix and sys.exec_prefix are set to that directory and it is also checked for site-packages (sys.base_prefix and sys.base_exec_prefix will always be the "real" prefixes of the Python installation). If "pyvenv.cfg" (a bootstrap configuration file) contains the key "include-system-site-packages" set to anything other than "false" (case-insensitive), the system-level prefixes will still also be searched for site-packages; otherwise they won't. All of the resulting site-specific directories, if they exist, are appended to sys.path, and also inspected for path configuration files. A path configuration file is a file whose name has the form <package>.pth; its contents are additional directories (one per line) to be added to sys.path. Non-existing directories (or non-directories) are never added to sys.path; no directory is added to sys.path more than once. Blank lines and lines beginning with '#' are skipped. Lines starting with 'import' are executed. For example, suppose sys.prefix and sys.exec_prefix are set to /usr/local and there is a directory /usr/local/lib/python2.5/site-packages with three subdirectories, foo, bar and spam, and two path configuration files, foo.pth and bar.pth. Assume foo.pth contains the following: # foo package configuration foo bar bletch and bar.pth contains: # bar package configuration bar Then the following directories are added to sys.path, in this order: /usr/local/lib/python2.5/site-packages/bar /usr/local/lib/python2.5/site-packages/foo Note that bletch is omitted because it doesn't exist; bar precedes foo because bar.pth comes alphabetically before foo.pth; and spam is omitted because it is not mentioned in either path configuration file. The readline module is also automatically configured to enable completion for systems that support it. This can be overridden in sitecustomize, usercustomize or PYTHONSTARTUP. Starting Python in isolated mode (-I) disables automatic readline configuration. After these operations, an attempt is made to import a module named sitecustomize, which can perform arbitrary additional site-specific customizations. If this import fails with an ImportError exception, it is silently ignored. """ import sys import os import builtins import _sitebuiltins import io import stat import errno # Prefixes for site-packages; add additional prefixes like /usr/local here PREFIXES = [sys.prefix, sys.exec_prefix] # Enable per user site-packages directory # set it to False to disable the feature or True to force the feature ENABLE_USER_SITE = None # for distutils.commands.install # These values are initialized by the getuserbase() and getusersitepackages() # functions, through the main() function when Python starts. USER_SITE = None USER_BASE = None def _trace(message): if sys.flags.verbose: print(message, file=sys.stderr) def makepath(*paths): dir = os.path.join(*paths) try: dir = os.path.abspath(dir) except OSError: pass return dir, os.path.normcase(dir) def abs_paths(): """Set all module __file__ and __cached__ attributes to an absolute path""" for m in set(sys.modules.values()): loader_module = None try: loader_module = m.__loader__.__module__ except AttributeError: try: loader_module = m.__spec__.loader.__module__ except AttributeError: pass if loader_module not in {'_frozen_importlib', '_frozen_importlib_external'}: continue # don't mess with a PEP 302-supplied __file__ try: m.__file__ = os.path.abspath(m.__file__) except (AttributeError, OSError, TypeError): pass try: m.__cached__ = os.path.abspath(m.__cached__) except (AttributeError, OSError, TypeError): pass def removeduppaths(): """ Remove duplicate entries from sys.path along with making them absolute""" # This ensures that the initial path provided by the interpreter contains # only absolute pathnames, even if we're running from the build directory. L = [] known_paths = set() for dir in sys.path: # Filter out duplicate paths (on case-insensitive file systems also # if they only differ in case); turn relative paths into absolute # paths. dir, dircase = makepath(dir) if dircase not in known_paths: L.append(dir) known_paths.add(dircase) sys.path[:] = L return known_paths def _init_pathinfo(): """Return a set containing all existing file system items from sys.path.""" d = set() for item in sys.path: try: if os.path.exists(item): _, itemcase = makepath(item) d.add(itemcase) except TypeError: continue return d def addpackage(sitedir, name, known_paths): """Process a .pth file within the site-packages directory: For each line in the file, either combine it with sitedir to a path and add that to known_paths, or execute it if it starts with 'import '. """ if known_paths is None: known_paths = _init_pathinfo() reset = True else: reset = False fullname = os.path.join(sitedir, name) try: st = os.lstat(fullname) except OSError: return if ((getattr(st, 'st_flags', 0) & stat.UF_HIDDEN) or (getattr(st, 'st_file_attributes', 0) & stat.FILE_ATTRIBUTE_HIDDEN)): _trace(f"Skipping hidden .pth file: {fullname!r}") return _trace(f"Processing .pth file: {fullname!r}") try: with io.open_code(fullname) as f: pth_content = f.read() except OSError: return try: # Accept BOM markers in .pth files as we do in source files # (Windows PowerShell 5.1 makes it hard to emit UTF-8 files without a BOM) pth_content = pth_content.decode("utf-8-sig") except UnicodeDecodeError: # Fallback to locale encoding for backward compatibility. # We will deprecate this fallback in the future. import locale pth_content = pth_content.decode(locale.getencoding()) _trace(f"Cannot read {fullname!r} as UTF-8. " f"Using fallback encoding {locale.getencoding()!r}") for n, line in enumerate(pth_content.splitlines(), 1): if line.startswith("#"): continue if line.strip() == "": continue try: if line.startswith(("import ", "import\t")): exec(line) continue line = line.rstrip() dir, dircase = makepath(sitedir, line) if dircase not in known_paths and os.path.exists(dir): sys.path.append(dir) known_paths.add(dircase) except Exception as exc: print(f"Error processing line {n:d} of {fullname}:\n", file=sys.stderr) import traceback for record in traceback.format_exception(exc): for line in record.splitlines(): print(' '+line, file=sys.stderr) print("\nRemainder of file ignored", file=sys.stderr) break if reset: known_paths = None return known_paths def addsitedir(sitedir, known_paths=None): """Add 'sitedir' argument to sys.path if missing and handle .pth files in 'sitedir'""" _trace(f"Adding directory: {sitedir!r}") if known_paths is None: known_paths = _init_pathinfo() reset = True else: reset = False sitedir, sitedircase = makepath(sitedir) if not sitedircase in known_paths: sys.path.append(sitedir) # Add path component known_paths.add(sitedircase) try: names = os.listdir(sitedir) except OSError: return names = [name for name in names if name.endswith(".pth") and not name.startswith(".")] for name in sorted(names): addpackage(sitedir, name, known_paths) if reset: known_paths = None return known_paths def check_enableusersite(): """Check if user site directory is safe for inclusion The function tests for the command line flag (including environment var), process uid/gid equal to effective uid/gid. None: Disabled for security reasons False: Disabled by user (command line option) True: Safe and enabled """ if sys.flags.no_user_site: return False if hasattr(os, "getuid") and hasattr(os, "geteuid"): # check process uid == effective uid if os.geteuid() != os.getuid(): return None if hasattr(os, "getgid") and hasattr(os, "getegid"): # check process gid == effective gid if os.getegid() != os.getgid(): return None return True # NOTE: sysconfig and it's dependencies are relatively large but site module # needs very limited part of them. # To speedup startup time, we have copy of them. # # See https://bugs.python.org/issue29585 # Copy of sysconfig._get_implementation() def _get_implementation(): return 'Python' # Copy of sysconfig._getuserbase() def _getuserbase(): env_base = os.environ.get("PYTHONUSERBASE", None) if env_base: return env_base # Emscripten, iOS, tvOS, VxWorks, WASI, and watchOS have no home directories if sys.platform in {"emscripten", "ios", "tvos", "vxworks", "wasi", "watchos"}: return None def joinuser(*args): return os.path.expanduser(os.path.join(*args)) if os.name == "nt": base = os.environ.get("APPDATA") or "~" return joinuser(base, _get_implementation()) if sys.platform == "darwin" and sys._framework: return joinuser("~", "Library", sys._framework, "%d.%d" % sys.version_info[:2]) return joinuser("~", ".local") # Same to sysconfig.get_path('purelib', os.name+'_user') def _get_path(userbase): version = sys.version_info if hasattr(sys, 'abiflags') and 't' in sys.abiflags: abi_thread = 't' else: abi_thread = '' implementation = _get_implementation() implementation_lower = implementation.lower() if os.name == 'nt': ver_nodot = sys.winver.replace('.', '') return f'{userbase}\\{implementation}{ver_nodot}\\site-packages' if sys.platform == 'darwin' and sys._framework: return f'{userbase}/lib/{implementation_lower}/site-packages' return f'{userbase}/lib/python{version[0]}.{version[1]}{abi_thread}/site-packages' def getuserbase(): """Returns the `user base` directory path. The `user base` directory can be used to store data. If the global variable ``USER_BASE`` is not initialized yet, this function will also set it. """ global USER_BASE if USER_BASE is None: USER_BASE = _getuserbase() return USER_BASE def getusersitepackages(): """Returns the user-specific site-packages directory path. If the global variable ``USER_SITE`` is not initialized yet, this function will also set it. """ global USER_SITE, ENABLE_USER_SITE userbase = getuserbase() # this will also set USER_BASE if USER_SITE is None: if userbase is None: ENABLE_USER_SITE = False # disable user site and return None else: USER_SITE = _get_path(userbase) return USER_SITE def addusersitepackages(known_paths): """Add a per user site-package to sys.path Each user has its own python directory with site-packages in the home directory. """ # get the per user site-package path # this call will also make sure USER_BASE and USER_SITE are set _trace("Processing user site-packages") user_site = getusersitepackages() if ENABLE_USER_SITE and os.path.isdir(user_site): addsitedir(user_site, known_paths) return known_paths def getsitepackages(prefixes=None): """Returns a list containing all global site-packages directories. For each directory present in ``prefixes`` (or the global ``PREFIXES``), this function will find its `site-packages` subdirectory depending on the system environment, and will return a list of full paths. """ sitepackages = [] seen = set() if prefixes is None: prefixes = PREFIXES for prefix in prefixes: if not prefix or prefix in seen: continue seen.add(prefix) implementation = _get_implementation().lower() ver = sys.version_info if hasattr(sys, 'abiflags') and 't' in sys.abiflags: abi_thread = 't' else: abi_thread = '' if os.sep == '/': libdirs = [sys.platlibdir] if sys.platlibdir != "lib": libdirs.append("lib") for libdir in libdirs: path = os.path.join(prefix, libdir, f"{implementation}{ver[0]}.{ver[1]}{abi_thread}", "site-packages") sitepackages.append(path) else: sitepackages.append(prefix) sitepackages.append(os.path.join(prefix, "Lib", "site-packages")) return sitepackages def addsitepackages(known_paths, prefixes=None): """Add site-packages to sys.path""" _trace("Processing global site-packages") for sitedir in getsitepackages(prefixes): if os.path.isdir(sitedir): addsitedir(sitedir, known_paths) return known_paths def setquit(): """Define new builtins 'quit' and 'exit'. These are objects which make the interpreter exit when called. The repr of each object contains a hint at how it works. """ if os.sep == '\\': eof = 'Ctrl-Z plus Return' else: eof = 'Ctrl-D (i.e. EOF)' builtins.quit = _sitebuiltins.Quitter('quit', eof) builtins.exit = _sitebuiltins.Quitter('exit', eof) def setcopyright(): """Set 'copyright' and 'credits' in builtins""" builtins.copyright = _sitebuiltins._Printer("copyright", sys.copyright) builtins.credits = _sitebuiltins._Printer("credits", """\ Thanks to CWI, CNRI, BeOpen, Zope Corporation, the Python Software Foundation, and a cast of thousands for supporting Python development. See www.python.org for more information.""") files, dirs = [], [] # Not all modules are required to have a __file__ attribute. See # PEP 420 for more details. here = getattr(sys, '_stdlib_dir', None) if not here and hasattr(os, '__file__'): here = os.path.dirname(os.__file__) if here: files.extend(["LICENSE.txt", "LICENSE"]) dirs.extend([os.path.join(here, os.pardir), here, os.curdir]) builtins.license = _sitebuiltins._Printer( "license", "See https://www.python.org/psf/license/", files, dirs) def sethelper(): builtins.help = _sitebuiltins._Helper() def gethistoryfile(): """Check if the PYTHON_HISTORY environment variable is set and define it as the .python_history file. If PYTHON_HISTORY is not set, use the default .python_history file. """ if not sys.flags.ignore_environment: history = os.environ.get("PYTHON_HISTORY") if history: return history return os.path.join(os.path.expanduser('~'), '.python_history') def enablerlcompleter(): """Enable default readline configuration on interactive prompts, by registering a sys.__interactivehook__. """ sys.__interactivehook__ = register_readline def register_readline(): """Configure readline completion on interactive prompts. If the readline module can be imported, the hook will set the Tab key as completion key and register ~/.python_history as history file. This can be overridden in the sitecustomize or usercustomize module, or in a PYTHONSTARTUP file. """ if not sys.flags.ignore_environment: PYTHON_BASIC_REPL = os.getenv("PYTHON_BASIC_REPL") else: PYTHON_BASIC_REPL = False import atexit try: try: import readline except ImportError: readline = None else: import rlcompleter # noqa: F401 except ImportError: return try: if PYTHON_BASIC_REPL: CAN_USE_PYREPL = False else: original_path = sys.path sys.path = [p for p in original_path if p != ''] try: import _pyrepl.readline if os.name == "nt": import _pyrepl.windows_console console_errors = (_pyrepl.windows_console._error,) else: import _pyrepl.unix_console console_errors = _pyrepl.unix_console._error from _pyrepl.main import CAN_USE_PYREPL finally: sys.path = original_path except ImportError: return if readline is not None: # Reading the initialization (config) file may not be enough to set a # completion key, so we set one first and then read the file. if readline.backend == 'editline': readline.parse_and_bind('bind ^I rl_complete') else: readline.parse_and_bind('tab: complete') try: readline.read_init_file() except OSError: # An OSError here could have many causes, but the most likely one # is that there's no .inputrc file (or .editrc file in the case of # Mac OS X + libedit) in the expected location. In that case, we # want to ignore the exception. pass if readline is None or readline.get_current_history_length() == 0: # If no history was loaded, default to .python_history, # or PYTHON_HISTORY. # The guard is necessary to avoid doubling history size at # each interpreter exit when readline was already configured # through a PYTHONSTARTUP hook, see: # http://bugs.python.org/issue5845#msg198636 history = gethistoryfile() if CAN_USE_PYREPL: readline_module = _pyrepl.readline exceptions = (OSError, *console_errors) else: if readline is None: return readline_module = readline exceptions = OSError try: readline_module.read_history_file(history) except exceptions: pass def write_history(): try: readline_module.write_history_file(history) except (FileNotFoundError, PermissionError): # home directory does not exist or is not writable # https://bugs.python.org/issue19891 pass except OSError: if errno.EROFS: pass # gh-128066: read-only file system else: raise atexit.register(write_history) def venv(known_paths): global PREFIXES, ENABLE_USER_SITE env = os.environ if sys.platform == 'darwin' and '__PYVENV_LAUNCHER__' in env: executable = sys._base_executable = os.environ['__PYVENV_LAUNCHER__'] else: executable = sys.executable exe_dir = os.path.dirname(os.path.abspath(executable)) site_prefix = os.path.dirname(exe_dir) sys._home = None conf_basename = 'pyvenv.cfg' candidate_conf = next( ( conffile for conffile in ( os.path.join(exe_dir, conf_basename), os.path.join(site_prefix, conf_basename) ) if os.path.isfile(conffile) ), None ) if candidate_conf: virtual_conf = candidate_conf system_site = "true" # Issue 25185: Use UTF-8, as that's what the venv module uses when # writing the file. with open(virtual_conf, encoding='utf-8') as f: for line in f: if '=' in line: key, _, value = line.partition('=') key = key.strip().lower() value = value.strip() if key == 'include-system-site-packages': system_site = value.lower() elif key == 'home': sys._home = value sys.prefix = sys.exec_prefix = site_prefix # Doing this here ensures venv takes precedence over user-site addsitepackages(known_paths, [sys.prefix]) # addsitepackages will process site_prefix again if its in PREFIXES, # but that's ok; known_paths will prevent anything being added twice if system_site == "true": PREFIXES.insert(0, sys.prefix) else: PREFIXES = [sys.prefix] ENABLE_USER_SITE = False return known_paths def execsitecustomize(): """Run custom site specific code, if available.""" try: try: import sitecustomize except ImportError as exc: if exc.name == 'sitecustomize': pass else: raise except Exception as err: if sys.flags.verbose: sys.excepthook(*sys.exc_info()) else: sys.stderr.write( "Error in sitecustomize; set PYTHONVERBOSE for traceback:\n" "%s: %s\n" % (err.__class__.__name__, err)) def execusercustomize(): """Run custom user specific code, if available.""" try: try: import usercustomize except ImportError as exc: if exc.name == 'usercustomize': pass else: raise except Exception as err: if sys.flags.verbose: sys.excepthook(*sys.exc_info()) else: sys.stderr.write( "Error in usercustomize; set PYTHONVERBOSE for traceback:\n" "%s: %s\n" % (err.__class__.__name__, err)) def main(): """Add standard site-specific directories to the module search path. This function is called automatically when this module is imported, unless the python interpreter was started with the -S flag. """ global ENABLE_USER_SITE orig_path = sys.path[:] known_paths = removeduppaths() if orig_path != sys.path: # removeduppaths() might make sys.path absolute. # fix __file__ and __cached__ of already imported modules too. abs_paths() known_paths = venv(known_paths) if ENABLE_USER_SITE is None: ENABLE_USER_SITE = check_enableusersite() known_paths = addusersitepackages(known_paths) known_paths = addsitepackages(known_paths) setquit() setcopyright() sethelper() if not sys.flags.isolated: enablerlcompleter() execsitecustomize() if ENABLE_USER_SITE: execusercustomize() # Prevent extending of sys.path when python was started with -S and # site is imported later. if not sys.flags.no_site: main() def _script(): help = """\ %s [--user-base] [--user-site] Without arguments print some useful information With arguments print the value of USER_BASE and/or USER_SITE separated by '%s'. Exit codes with --user-base or --user-site: 0 - user site directory is enabled 1 - user site directory is disabled by user 2 - user site directory is disabled by super user or for security reasons >2 - unknown error """ args = sys.argv[1:] if not args: user_base = getuserbase() user_site = getusersitepackages() print("sys.path = [") for dir in sys.path: print(" %r," % (dir,)) print("]") def exists(path): if path is not None and os.path.isdir(path): return "exists" else: return "doesn't exist" print(f"USER_BASE: {user_base!r} ({exists(user_base)})") print(f"USER_SITE: {user_site!r} ({exists(user_site)})") print(f"ENABLE_USER_SITE: {ENABLE_USER_SITE!r}") sys.exit(0) buffer = [] if '--user-base' in args: buffer.append(USER_BASE) if '--user-site' in args: buffer.append(USER_SITE) if buffer: print(os.pathsep.join(buffer)) if ENABLE_USER_SITE: sys.exit(0) elif ENABLE_USER_SITE is False: sys.exit(1) elif ENABLE_USER_SITE is None: sys.exit(2) else: sys.exit(3) else: import textwrap print(textwrap.dedent(help % (sys.argv[0], os.pathsep))) sys.exit(10) if __name__ == '__main__': _script()
08-29
本 PPT 介绍了制药厂房中供配电系统的总体概念与设计要点,内容包括: 洁净厂房的特点及其对供配电系统的特殊要求; 供配电设计的一般原则与依据的国家/行业标准; 从上级电网到工厂变电所、终端配电的总体结构与模块化设计思路; 供配电范围:动力配电、照明、通讯、接地、防雷与消防等; 动力配电中电压等级、接地系统形式(如 TN-S)、负荷等级与可靠性、UPS 配置等; 照明的电源方式、光源选择、安装方式、应急与备用照明要求; 通讯系统、监控系统在生产管理与消防中的作用; 接地与等电位连接、防雷等级与防雷措施; 消防设施及其专用供电(消防泵、排烟风机、消防控制室、应急照明等); 常见高压柜、动力柜、照明箱等配电设备案例及部分设计图纸示意; 公司已完成的典型项目案例。 1. 工程背景与总体框架 所属领域:制药厂房工程的公用工程系统,其中本 PPT 聚焦于供配电系统。 放在整个公用工程中的位置:与给排水、纯化水/注射用水、气体与热力、暖通空调、自动化控制等系统并列。 2. Part 01 供配电概述 2.1 洁净厂房的特点 空间密闭,结构复杂、走向曲折; 单相设备、仪器种类多,工艺设备昂贵、精密; 装修材料与工艺材料种类多,对尘埃、静电等更敏感。 这些特点决定了:供配电系统要安全可靠、减少积尘、便于清洁和维护。 2.2 供配电总则 供配电设计应满足: 可靠、经济、适用; 保障人身与财产安全; 便于安装与维护; 采用技术先进的设备与方案。 2.3 设计依据与规范 引用了大量俄语标准(ГОСТ、СНиП、SanPiN 等)以及国家、行业和地方规范,作为设计的法规基础文件,包括: 电气设备、接线、接地、电气安全; 建筑物电气装置、照明标准; 卫生与安全相关规范等。 3. Part 02 供配电总览 从电源系统整体结构进行总览: 上级:地方电网; 工厂变电所(10kV 配电装置、变压
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值