On Thursday 30 December 2010 01:37:58 Knacktus wrote:
> Hello everyone,
>> here the background story ;-):
> ------------------------------
>> I'm writing an application which handles a lot of nested data. Imagine a
> QTreeView with 10-20 levels and up to 50000 lines.
> The standard use case would be to expand one level after the other in
> the GUI. For each level I would load the data from the database. This
> can be very slow, as I have a lot of network traffic and DB requests
> with little amounts of data.
> On the other hand, I am able to identify all the data of the tree in the
> database. Therefore I can load all items to the client with one request.
> But this is not always necessary.
> My idea is to load all the data to the client (as bulk) on certain user
> requests only, e.g. if the user clicks "expand all". In the meantime,
> the user should be able to keep on working with the app.
>> Here the question:
> ------------------
>> I've read about multithreading and multiprocessing, but have no
> experience with it yet. I've done a lot of MPI programming with Fortran,
> so multiprocessing is more familiar to me. Right now I have these
> options in my mind:
>> 1) Multiprocessing: Loading the bulk data from the DB with a seperate
> process and merge this data with the exisiting data in my main process.
> The data is in a huge dict with the item ids as keys. So merging should
> not be a problem. That sounds clean to me, but I'm concerned about the
> fact, that multiprocessing is copying the whole app (with all the data
> already loaded from the DB) in memory.
>> 2) Multithreading: I guess trying to write to the same dict with my
> items with two threads could be messy. Therefore I would create a
> temporary dict in the secondary thread that loads the bulk from the DB
> and merge with the main dict later. The question here would be the
> responsiveness of the GUI as Python can't perform "real" multithreading.
>> 3) Creating a Subprocess: I would create a helper app that loads the
> data from the DB and saves it temporarly on the local drive as pickle.
> The main app would read the data from the file and merge with the main
> dict.
>>> What are your recommendations and experiences? Are there best practices?
>I've been using threads(QThread) to run database queries and other tasks such
as image loading for years without any major problems. I am using a custom
built ORM written in c++(with python bindings), so there are a lot more
potential issues than a pure python implementation that strictly executes
queries.
Python has a Global Interpreter Lock(GIL), which will prevent conflicting
access to python data types such as dicts. This should be sufficient to make
your specific use case thread safe. The GIL can get in the way when your
threaded code is doing a lot of processing using the python interpreter, but
when the threaded code is calling long running c/c++ functions, such as a
database select, it will release the GIL until the call completes. This is
all done automatically within python and PyQt. This is "real" multithreading.
Your #1 and #3 are essentially the same, except that you don't explain how you
are going to merge the data in #1. I would imagine that you could do it via
shared memory, but I have no experience using shared memory from python. You
may be thinking of using fork for #1, which I would avoid simply because it
complicates things and provides very little advantage for your use case.
Forking would simplify passing the query or other data to the worker process,
but this should be fairly simple anyway, and you still need to worry about the
same thing when passing the data back.
When forking file handles and sockets are shared between the processes, which
can easily cause problems unless you take care to reinitialize them. Also
forking is not available on windows.
I was using fork for a linux daemon at one point, which was built using python
and my ORM. Even after dealing with the obvious stuff like reinitializing the
database connection, it still caused problems with my logging library so I
changed the code to use threads and that has been working great.
In conclusion I would try threads to solve your problem. I think they would
require the least amount of code and be the most efficient in terms of memory
and cpu requirements. If you keep the amount of code that runs in the thread
to a minimum, you can avoid most problems that people run into, and many of
those problems are already a non-issue when using python.
Matt Newell
More information about the PyQt
mailing list
‘She has never mentioned her father to me. Was he—well, the sort of man whom the County Club would not have blackballed?’ "We walked by the side of our teams or behind the wagons, we slept on the ground at night, we did our own cooking, we washed our knives by sticking them into the ground rapidly a few times, and we washed our plates with sand and wisps of grass. When we stopped, we arranged our wagons in a circle, and thus formed a 'corral,' or yard, where we drove our oxen to yoke them up. And the corral was often very useful as a fort, or camp, for defending ourselves against the Indians. Do you see that little hollow down there?" he asked, pointing to a depression in the ground a short distance to the right of the train. "Well, in that hollow our wagon-train was kept three days and nights by the Indians. Three days and nights they stayed around, and made several attacks. Two of our men were killed and three were wounded by their arrows, and others had narrow escapes. One arrow hit me on the throat, but I was saved by the knot of my neckerchief, and the point only tore the skin a little. Since that time I have always had a fondness for large neckties. I don't know how many of the Indians we killed, as they carried off their dead and wounded, to save them from being scalped. Next to getting the scalps of their enemies, the most important thing with the Indians is to save their own. We had several fights during our journey, but that one was the worst. Once a little party of us were surrounded in a small 'wallow,' and had a tough time to defend ourselves successfully. Luckily for us, the Indians had no fire-arms then, and their bows and arrows were no match for our rifles. Nowadays they are well armed, but there are[Pg 41] not so many of them, and they are not inclined to trouble the railway trains. They used to do a great deal of mischief in the old times, and many a poor fellow has been killed by them." As dusk came on nearly the whole population of Maastricht, with all their temporary guests, formed an endless procession and went to invoke God's mercy by the Virgin Mary's intercession. They went to Our Lady's Church, in which stands the miraculous statue of Sancta Maria Stella Maris. The procession filled all the principal streets and squares of the town. I took my stand at the corner of the Vrijthof, where all marched past me, men, women, and children, all praying aloud, with loud voices beseeching: "Our Lady, Star of the Sea, pray for us ... pray for us ... pray for us ...!" It had not occurred to her for some hours after Mrs. Campbell had told her of Landor's death that she was free now to give herself to Cairness. She had gasped, indeed, when she did remember it, and had put the thought away, angrily and self-reproachfully. But it returned now, and she felt that she might cling to it. She had been grateful, and she had been faithful, too.[Pg 286] She remembered only that Landor had been kind to her, and forgot that for the last two years she had borne with much harsh coldness, and with a sort of contempt which she felt in her unanalyzing mind to have been entirely unmerited. Gradually she raised herself until she sat quite erect by the side of the mound, the old exultation of her half-wild girlhood shining in her face as she planned the future, which only a few minutes before had seemed so hopeless. After he had gloated over Sergeant Ramsey, Shorty got his men into the road ready to start. Si placed himself in front of the squad and deliberately loaded his musket in their sight. Shorty took his place in the rear, and gave out: The groups about each gun thinned out, as the shrieking fragments of shell mowed down man after man, but the rapidity of the fire did not slacken in the least. One of the Lieutenants turned and motioned with his saber to the riders seated on their horses in the line of limbers under the cover of the slope. One rider sprang from each team and ran up to take the place of men who had fallen. "As long as there's men and women in the world, the men 'ull be top and the women bottom." Then, in the house, the little girls were useful. Mrs. Backfield was not so energetic as she used to be. She had never been a robust woman, and though her husband's care had kept her well and strong, her frame was not equal to Reuben's demands; after fourteen years' hard labour, she suffered from rheumatism, which though seldom acute, was inclined to make her stiff and slow. It was here that Caro and Tilly came in, and Reuben began to appreciate his girls. After all, girls were needed in a house—and as for young men and marriage, their father could easily see that such follies did not spoil their usefulness or take them from him. Caro and Tilly helped their grandmother in all sorts of ways—they dusted, they watched pots, they shelled peas and peeled potatoes, they darned house-linen, they could even make a bed between them. HoME一级毛片视频免费公开
ENTER NUMBET 0018www.b010.com.cn www.eocc.net.cn shaanxiaoti.com.cn jhly.net.cn jukewuxian.com.cn hktea.com.cn intnet.net.cn jlpic.com.cn wvedu.com.cn www.itebay.com.cn