| | |
+
+- builtins.object
+
-
+
+- BackgroundIOPriority
-- threading.Thread(builtins.object)
-
-
-
-- BackgroundWriteProcess
+
- threading.Thread(builtins.object)
+
-
+
+- BackgroundWriteProcess
-
-
-
-
-class BackgroundIOPriority(builtins.object) |
+
+
+
+
+class BackgroundIOPriority(builtins.object) |
-| |
-BackgroundIOPriority - Priority Profile for doing background writes.
-
- See __init__ for fields
|
-| |
-Methods defined here:
-- __getitem__(self, key)
+ | | |
+BackgroundIOPriority(chainPollTime, defaultChunkSize, bandwidthPct, numChunksRateSmoothing=5)
+
+BackgroundIOPriority - Priority Profile for doing background writes.
+
+ See __init__ for fields
|
+| |
+Methods defined here:
+- __getitem__(self, key)
-- __init__(self, chainPollTime, defaultChunkSize, priorityPct, charityRate=1.85, charityTime=0.0003)
- __init__ - Create a BackgroundIOPriority.
-
-Some terms: throughput - Bandwidth out (Megs per second)
- interactivity - CPU time available for other tasks (calculations, other I/O, etc)
-
-@param chainPollTime - float > 0, When chaining, this is the sleep time between checking if prior is finished.
- Too low and the polling takes up CPU time, too high and you'll lose a little time in between chained writes, while gaining interactivity elsewhere.
-
-@param defaultChunkSize - integer > 0, When providing a straight string/bytes to bgwrite (instead of chunking yourself, or using bgwrite_chunk) this will
- be used as the max size of each chunk. Each chunk is written and a flush is issued (if the stream supports it).
- Increasing this increases throughput while decreasing interactivity
-
-@param priorityPct - integer > 0, generally 0-100. When this number is high, throughput for the operation will be higher. When it is lower,
- interactivity is higher, e.x. if you have a calculation going and a write going, the lower this number the longer the write will take, but the more
- calculations will be performed during that period.
-
-@param charityRate - float >= 0, Every couple of blocks written, the current throughput is checked and if things have been going swiftly
- a short sleep will be incurred. Increasing this number causes that check to happen more often.
-
- This number is related to both the number of blocks and the priorityPct. The default, should be fine, but you may find it better
- as a different value in certain cases. Increasing or decreasing could either increase or decrease interactivity, depending on those other factors.
- Generally, however, increasing this increases interactivity and ability to write in parallel, at the cost of throughput.
-
-@param charityTime - float >= 0 - Used to calculate the time to sleep when the charity period hits. The equation is:
- sleepTime = charityTime * ((dataWritten / delta) / ( (dataWritten / delta) * priorityPctDec))
- Where dataWritten = number of bytes written already, delta = total time spent writing (not including charity time sleeping)
- and priorityPctDec = priorityPct / 100.
-
- Increasing this can increase interactivity and allow more parallel operations at the cost of throughput.
- The default should be fine for the majority of cases, but it is tunable.
-
+- __init__(self, chainPollTime, defaultChunkSize, bandwidthPct, numChunksRateSmoothing=5)
- __init__ - Create a BackgroundIOPriority.
+
+Some terms: throughput - Bandwidth out (Megs per second)
+ interactivity - CPU time available for other tasks (calculations, other I/O, etc)
+
+@param chainPollTime - float > 0, When chaining, this is the sleep time between checking if prior is finished.
+ Too low and the polling takes up CPU time, too high and you'll lose a little time in between chained writes, while gaining interactivity elsewhere.
+
+@param defaultChunkSize - integer > 0, When providing a straight string/bytes to bgwrite (instead of chunking yourself, or using bgwrite_chunk) this will
+ be used as the max size of each chunk. Each chunk is written and a flush is issued (if the stream supports it).
+ Increasing this increases throughput while decreasing interactivity
+
+@param bandwidthPct - integer > 0 and < 100. This is the percentage of overall bandwidth that this task will attempt to use.
+
+ A high number means higher throughput at the cost of lest interactivity for other tasks, a low number means the opposite.
+
+ So, for example, a bandwidthPct of "50" will attempt to use "50%" of the available bandwidth. Note, this does not represent theroetical
+ max bandwidth, i.e. the max rate of the I/O device, but the amount of available bandwidth available to this application. For example,
+ if this is given "100%", no throttling is performed. If this is given "80%", then it calculates the average time to write a single chunk,
+ ( see #numChunksRateSmoothing for how many chunks are used in evaluating this average ), and sleeps for then 20% of that time at the end
+ of every chunk.
+
+@param numChunksRateSmoothing - integer >= 1 , Default 5. This is the number of chunks which are used in calculating the current throughput rate.
+ See #bandwidthPct for the other half of the story. The higher this number, the more "fair" your application will be against a constant
+ rate of I/O by other applications, but the less able it may be to play fair when the external I/O is spiking.
+
+ Also, consider that this is related to the #defaultChunkSize, as it is not a constant period of time. The default of "5" should be okay,
+ but you may want to tune it if you use really large or really small chunk sizes.
+
+
An "interactivity score" is defined to be (number of calculations) / (time to write data).
-- __setitem__(self, key, value)
+- __setitem__(self, key, value)
-
-Data descriptors defined here:
-- chainPollTime
+
+Data descriptors defined here:
+- bandwidthPct
-- charityRate
+- chainPollTime
-- charityTime
+- defaultChunkSize
-- defaultChunkSize
+- numChunksRateSmoothing
-- priorityPct
-
-
|
-
+
+
+
+class BackgroundWriteProcess(threading.Thread) |
-| |
-BackgroundWriteProcess - A thread and data store representing a background write task. You should probably use one of the bgwrite* methods and not this directly.
-
-Attributes:
-
- remainingData <deque> - A queue representing the data yet to be written
-
- startedWriting <bool> - Starts False, changes to True when writing has started (thread has started and any pending prior chain has completed)
-
- finished <bool> - Starts False, changes to True after writing has completed, and if closeWhenFinished is True the handle is also closed.
|
-| |
-- Method resolution order:
-- BackgroundWriteProcess
-- threading.Thread
-- builtins.object
+| |
+BackgroundWriteProcess(fileObj, dataBlocks, closeWhenFinished=False, chainAfter=None, ioPrio=4)
+
+BackgroundWriteProcess - A thread and data store representing a background write task. You should probably use one of the bgwrite* methods and not this directly.
+
+Attributes:
+
+ remainingData <deque> - A queue representing the data yet to be written
+
+ startedWriting <bool> - Starts False, changes to True when writing has started (thread has started and any pending prior chain has completed)
+
+ finished <bool> - Starts False, changes to True after writing has completed, and if closeWhenFinished is True the handle is also closed.
|
+| |
+- Method resolution order:
+- BackgroundWriteProcess
+- threading.Thread
+- builtins.object
-
-Methods defined here:
-- __init__(self, fileObj, dataBlocks, closeWhenFinished=False, chainAfter=None, ioPrio=4)
- __init__ - Create the BackgroundWriteProcess thread. You should probably use bgwrite or bgwrite_chunk instead of calling this directly.
-
-@param fileObj <stream> - A stream, like a file, to write into. Hopefully it supports flushing, but it is not a requirement.
-
-@param dataBlocks <bytes/str/list<bytes/str>> - If a list of bytes/str, those are treated as the data blocks, written in order with heuristics for interactivity in between blocks. If bytes/str are provided not in a list form, they will be split based on the rules of the associated #ioPrio
-
-@param closeWhenFinished <bool> - Default False. If True, the fileObj will be closed after writing has completed.
-
-@param chainAfter <None/BackgroundWriteProcess> - If provided, will hold off writing until the provided BackgroundWriteProcess has completed (used for queueing multiple writes whilst retaining order)
-
-@param ioPrio <int/BackgroundIOPriority> - If an integer (1-10), a predefined BackgroundIOPriority will be used. 1 is highest throughput, 10 is most interactivity. You can also pass in your own BackgroundIOPriority object if you want to define a custom profile.
-
-
-@raises ValueError - If ioPrio is neither a BackgroundIOPriority nor integer 1-10 inclusive
- - If chainAfter is not a BackgroundWriteProcess or None
+
+Methods defined here:
+- __init__(self, fileObj, dataBlocks, closeWhenFinished=False, chainAfter=None, ioPrio=4)
- __init__ - Create the BackgroundWriteProcess thread. You should probably use bgwrite or bgwrite_chunk instead of calling this directly.
+
+@param fileObj <stream> - A stream, like a file, to write into. Hopefully it supports flushing, but it is not a requirement.
+
+@param dataBlocks <bytes/str/list<bytes/str>> - If a list of bytes/str, those are treated as the data blocks, written in order with heuristics for interactivity in between blocks. If bytes/str are provided not in a list form, they will be split based on the rules of the associated #ioPrio
+
+@param closeWhenFinished <bool> - Default False. If True, the fileObj will be closed after writing has completed.
+
+@param chainAfter <None/BackgroundWriteProcess> - If provided, will hold off writing until the provided BackgroundWriteProcess has completed (used for queueing multiple writes whilst retaining order)
+
+@param ioPrio <int/BackgroundIOPriority> - If an integer (1-10), a predefined BackgroundIOPriority will be used. 1 is highest throughput, 10 is most interactivity. You can also pass in your own BackgroundIOPriority object if you want to define a custom profile.
+
+
+@raises ValueError - If ioPrio is neither a BackgroundIOPriority nor integer 1-10 inclusive
+ - If chainAfter is not a BackgroundWriteProcess or None
+
+- run(self)
- run - Starts the thread. bgwrite and bgwrite_chunk automatically start the thread.
+
+
+Methods inherited from threading.Thread:
+- __repr__(self)
- Return repr(self).
+
+- getName(self)
+
+- isAlive(self)
- Return whether the thread is alive.
+
+This method is deprecated, use is_alive() instead.
+
+- isDaemon(self)
+
+- is_alive(self)
- Return whether the thread is alive.
+
+This method returns True just before the run() method starts until just
+after the run() method terminates. The module function enumerate()
+returns a list of all alive threads.
+
+- join(self, timeout=None)
- Wait until the thread terminates.
+
+This blocks the calling thread until the thread whose join() method is
+called terminates -- either normally or through an unhandled exception
+or until the optional timeout occurs.
+
+When the timeout argument is present and not None, it should be a
+floating point number specifying a timeout for the operation in seconds
+(or fractions thereof). As join() always returns None, you must call
+is_alive() after join() to decide whether a timeout happened -- if the
+thread is still alive, the join() call timed out.
+
+When the timeout argument is not present or None, the operation will
+block until the thread terminates.
+
+A thread can be join()ed many times.
+
+join() raises a RuntimeError if an attempt is made to join the current
+thread as that would cause a deadlock. It is also an error to join() a
+thread before it has been started and attempts to do so raises the same
+exception.
-- run(self)
- run - Starts the thread. bgwrite and bgwrite_chunk automatically start the thread.
+- setDaemon(self, daemonic)
- |
|
-
-
-
-Functions |
+- setName(self, name)
+
+- start(self)
- Start the thread's activity.
+
+It must be called at most once per thread object. It arranges for the
+object's run() method to be invoked in a separate thread of control.
+
+This method will raise a RuntimeError if called more than once on the
+same thread object.
+
+
+Data descriptors inherited from threading.Thread:
+- __dict__
+- dictionary for instance variables (if defined)
+
+- __weakref__
+- list of weak references to the object (if defined)
+
+- daemon
+- A boolean value indicating whether this thread is a daemon thread.
+
+This must be set before start() is called, otherwise RuntimeError is
+raised. Its initial value is inherited from the creating thread; the
+main thread is not a daemon thread and therefore all threads created in
+the main thread default to daemon = False.
+
+The entire Python program exits when no alive non-daemon threads are
+left.
+
+- ident
+- Thread identifier of this thread or None if it has not been started.
+
+This is a nonzero integer. See the get_ident() function. Thread
+identifiers may be recycled when a thread exits and another thread is
+created. The identifier is available even after the thread has exited.
+
+- name
+- A string used for identification purposes only.
+
+It has no semantics. Multiple threads may be given the same name. The
+initial name is set by the constructor.
+
+ |