- feeds
- popular
- recent
- reader
- about
- story
- technologies
- what is rss
- connect
- dmca
- contact
- Feed Preview PyParallel: how we removed the GIL and exploited all cores : programming
/u/trentnelson on PyParallel: how we removed the GIL and exploited all cores
Eek, bit of a slow reply, my apologies! No explicit funding; it was a personal project I embarked upon late 2012 -- I wanted something juicy to put on my resume as I was planning on relocating ...
/u/anko_painting on PyParallel: how we removed the GIL and exploited all cores
I agree with everything you said, only you said it more eloquently than i could hope to.
/u/Wagnerius on PyParallel: how we removed the GIL and exploited all cores
We agree on the limits on python (which BTW are quite connected to its strength, so trade offs and that) but a queue would give more than simply IPC : history of things done, monitoring of curren...
/u/ on PyParallel: how we removed the GIL and exploited all cores
Yes well. However it would not be required if python actually acted like a real environment. Instead it acts like it has multiple threads and a talking stick passed between them. Note: it is on...
/u/eigenlicht0 on PyParallel: how we removed the GIL and exploited all cores
As a fellow Python programmer: Interesting talk! But as a presentation techniques enthusiast: IMO you skip over too much content. Not only by clicking very fast through the slides, but also by ...
/u/Peaker on PyParallel: how we removed the GIL and exploited all cores
> Static typing is a mechanism for hinting to the compiler at compile > time It's more than that -- it is machine-checked documentation. Invariant verification. And it can also hint...
/u/shooshx on PyParallel: how we removed the GIL and exploited all cores
Great stuff! Was glued to the screen down to the last second :) One thing that made me sway uncomfortably in my chair is the no-deallocating issue. It feels like this relies heavily on "trust me,...
/u/ceol_ on PyParallel: how we removed the GIL and exploited all cores
Thanks for writing that up! It definitely clarifies a few things.
/u/trentnelson on PyParallel: how we removed the GIL and exploited all cores
There were probably 100 people in the audience. 2400 people have seen the video. I don't think anything that relies on handing out stuff to audience members is a particularly good solution. Thi...
/u/SCombinator on PyParallel: how we removed the GIL and exploited all cores
> Whitespace: I will never be able to explain the hate there. It just > doesn't make sense, especially given that IDE's often indent things > nicely for non-whitespace ...
/u/trentnelson on PyParallel: how we removed the GIL and exploited all cores
I... wait, what? I actually have taken presentation courses before. They're the type of courses that would have had me presenting 15 slides over 45 minutes. I warn people on the very first sl...
/u/trentnelson on PyParallel: how we removed the GIL and exploited all cores
Excellent question. I have no idea. If I had to guess, it's because there's no POSIX API for mirroring all of Windows' features; so if any UNIX vendor wants to provide something along the same li...
/u/trentnelson on PyParallel: how we removed the GIL and exploited all cores
It's on my something-to-look-into-later list!
/u/trentnelson on PyParallel: how we removed the GIL and exploited all cores
> but i could only watch maybe 1/3 before i had to start rewinding to > take it all in. Ah, yes, but with incredibly dense content, having a video plus slides that you can revi...
/u/thedeemon on PyParallel: how we removed the GIL and exploited all cores
As I understand, recently those marking bits for objects in Ruby's heap were moved to a separate location so you don't change anything during the mark phase in the pages where read-only objects a...
/u/rallix on PyParallel: how we removed the GIL and exploited all cores
> the amount of pain caused by tabs vs. spaces and wrong indents in a > multi developer project is ugly. I don't believe this actually happens IN PYTHON. (1) Do people not hav...
/u/ on PyParallel: how we removed the GIL and exploited all cores
> What I don't know is whether that just gets loaded into an in-memory > data structure (as Ruby does), or mmap'd It's not mmap'ed, the pyc file format is not mmap friendly a...
/u/SanityInAnarchy on PyParallel: how we removed the GIL and exploited all cores
Ruby did indeed use a "normal" garbage collector. The problem is, I believe it used a mark-and-sweep garbage collector -- variations on this are quite common, and the simplest, most naive impleme...
/u/flying-sheep on PyParallel: how we removed the GIL and exploited all cores
it is in python 3. just one more reason to use it.
/u/josefx on PyParallel: how we removed the GIL and exploited all cores
> python2 -tt That might actually help, just why isn't this the default?
/u/flying-sheep on PyParallel: how we removed the GIL and exploited all cores
use python3 (or python2 -tt, if you must) try to run your code after you wrote like 5 lines. if you’re inexperienced enough to mix tabs and spaces are mixed, python fails early and hard now. f...
/u/ on PyParallel: how we removed the GIL and exploited all cores
In my project I run processes separately (not fork() them from one master process), so COW is not much use for me anyway. If you fork() though (for example, you rely on multiprocessing module a...
/u/SanityInAnarchy on PyParallel: how we removed the GIL and exploited all cores
> The C-API is dependent on aspects of CPython being immutable, which > hamstrings attempts at revoking the GIL or changing the memory > layout... I think this is an excuse...
/u/pwang99 on PyParallel: how we removed the GIL and exploited all cores
Static typing is a mechanism for hinting to the compiler at compile time. Dynamic typing used to be just a mechanism for the runtime to dispatch at run time. Since dynamic compilation is now "A...
/u/pwang99 on PyParallel: how we removed the GIL and exploited all cores
Some people view on tablets and just want to peruse/scan the talk without downloading a large PDF, invoking another program, etc.
/u/willvarfar on PyParallel: how we removed the GIL and exploited all cores
I don't think your approach or mindset mad ;) I can imagine this approach going a lot further e.g. use the multiprocessing api but implement as threads with their own contexts and lower level l...
/u/SanityInAnarchy on PyParallel: how we removed the GIL and exploited all cores
Less RAM might be one, but I'm not entirely sure about this. When you import a Python module, does it just slurp in the compiled file into memory, or does it transform it into some in-memory str...
/u/ on PyParallel: how we removed the GIL and exploited all cores
Yeah I understand the problem exactly. However there is more complex issues involving when to lock / unlock the GIL. In a few cases in the c++ client lib we cannot unlock the GIL but also have to...
/u/ on PyParallel: how we removed the GIL and exploited all cores
No your exactly on the problem. However since the client side is written in c++ to communicate to the backend since it is also called from outside the UI by other parts of the system which is als...
/u/anko_painting on PyParallel: how we removed the GIL and exploited all cores
this was a really really interesting talk, and i'm not even a python guy. But you seriously needed about 1/3 the content - maybe lots of people are more advanced than me, but i could only watch m...
/u/ on PyParallel: how we removed the GIL and exploited all cores
PyParallel got rolled up into the PySerial project, I think. I tried using PyParallel a couple months back. Parallel port support is really bad right now, at least in Windows it was unusable. I...
/u/bms20 on PyParallel: how we removed the GIL and exploited all cores
So why are we not implementing windows like asynchronous IO in the linux kernel?
/u/thomasz on PyParallel: how we removed the GIL and exploited all cores
Could be beneficial for windows, where starting a process is very expensive.
/u/wot-teh-phuck on PyParallel: how we removed the GIL and exploited all cores
There is so much misinformation out there related to static typing that sometimes I think it warrants a course in comp sci syllabus. ;)
/u/westurner on PyParallel: how we removed the GIL and exploited all cores
> The new libevent2 is the best cross-platform IO library because it > abstracts this. http://libevent.org/ https://raw.github.com/libevent/libevent/patches-2.0/whatsnew-2.0.txt #...
/u/westurner on PyParallel: how we removed the GIL and exploited all cores
> What worked against me was that I already had the big picture in > mind: pairing this async I/O stuff with the parallel/GIL stuff such > that callbacks could actuall...
/u/ceol_ on PyParallel: how we removed the GIL and exploited all cores
So you have a web UI that controls some sort of C++ program. Obviously I don't know the specifics of your application, and I'm not a C++ developer, but wouldn't it have been better to make the we...
/u/ivosaurus on PyParallel: how we removed the GIL and exploited all cores
> This causes issue for example there is a considerable amount of code > in c++ which will work with python types (requires the GIL to be > locked). But also blocks ...
/u/Agitates on PyParallel: how we removed the GIL and exploited all cores
Static typing: done properly, it is wonderful and lightweight. Done by idiots, it becomes a nightmare.
/u/ on PyParallel: how we removed the GIL and exploited all cores
The issues are more complex that just with django / python and the GIL. There is a large amount of c++ code being called from python wrappers using boost_python. This causes issue for example the...
/u/riffito on PyParallel: how we removed the GIL and exploited all cores
I guess I've installed PDF.js on Firefox before it got bundled. Good to know it comes integrated now.
/u/username223 on PyParallel: how we removed the GIL and exploited all cores
> Firefox and Chrome have integrated PDF viewers. Safari has its own thing, too; how about IE? PDF is an awful format, but Javascript widgets showing PDFs converted to tiny JPGs (or whate...
/u/mcilrain on PyParallel: how we removed the GIL and exploited all cores
Looks like clickbaity startup spam.
/u/acdha on PyParallel: how we removed the GIL and exploited all cores
You do have profiler results demonstrating that it's really the GIL, or even Django, and not the way it's configured, right?
/u/ on PyParallel: how we removed the GIL and exploited all cores
Just because its popular does not make it good. Where I work people who are less skilled setup a web ui on a system we make. This is done inside django. The result was a massive performance issue...
/u/mattbillenstein on PyParallel: how we removed the GIL and exploited all cores
Bravo - very interesting work!
/u/trentnelson on PyParallel: how we removed the GIL and exploited all cores
The short answer is that for parallel computation, you'd follow a similar approach; implement protocol-based classes that perform computation against data, then return results. You tell us the ...
/u/josefx on PyParallel: how we removed the GIL and exploited all cores
> I have been programming in Python for 7 years and never worried > about it Everybody I work with uses tabs for c++, since they don't touch the python code very often they don't t...
/u/alecco on PyParallel: how we removed the GIL and exploited all cores
I wish I could blame autocorrect.
/u/psi- on PyParallel: how we removed the GIL and exploited all cores
Sorry, what was the answer?
/u/x-skeww on PyParallel: how we removed the GIL and exploited all cores
Firefox and Chrome have integrated PDF viewers. The one in Firefox uses PDF.js and the one in Chrome uses Foxit.
/u/ on PyParallel: how we removed the GIL and exploited all cores
Meat seams? For a moment I thought I was in the wrong subreddit...
/u/trentnelson on PyParallel: how we removed the GIL and exploited all cores
I just read the libevent2 link you posted -- looks good! We're all talking the same language, which is good :-) accept() is a funny one; it inherently serializes connections, which is one thing...
/u/trentnelson on PyParallel: how we removed the GIL and exploited all cores
Yup, benchmarks are next up on the list of things to do. Gotta' put my money where my mouth is, after all.
/u/willvarfar on PyParallel: how we removed the GIL and exploited all cores
I'm not arguing against IOCP or your talk. And if you can't use libevent2 you'd have to write - or someone would have to write - the CreateThreadpoolIo() etc for Linux/FreeBSD/etc. Once. In t...
/u/trentnelson on PyParallel: how we removed the GIL and exploited all cores
Are you saying you didn't enjoy the first 70 slides of deep IOCP internals? ;-) You're right though, the actual GIL removal stuff doesn't come 'til much later. The title is a bit misleading in ...
/u/trentnelson on PyParallel: how we removed the GIL and exploited all cores
Yeah I noticed that a couple of months ago. It's unfortunate.
/u/trentnelson on PyParallel: how we removed the GIL and exploited all cores
> FWIW, epoll can be edge-triggered and can be checked from multiple > threads/processes - you can have worker thread polls consuming epoll > - and you can even epoll ...
/u/riffito on PyParallel: how we removed the GIL and exploited all cores
PDF.js , for Firefox, Chrome and Opera. Lovely.
/u/trentnelson on PyParallel: how we removed the GIL and exploited all cores
> I explained completion-based IO to Guido when he started on the > async IO for Python; I guess it wasn't a big deal in anyone else's > eyes: > https://plus.google.com...
/u/trentnelson on PyParallel: how we removed the GIL and exploited all cores
> Honest question: if main threads are read-only for parallel threads > and parallel threads are "shared nothing", then what's the benefit > of threading at all? Excellen...
/u/trentnelson on PyParallel: how we removed the GIL and exploited all cores
> This. OP doesn't know about my SO_REUSEPORT. If I'm unaware of "your" (??) SO_REUSEPORT, that makes this commit I made to Python way back in 2008 pretty impressive :-)
/u/dataloopio on PyParallel: how we removed the GIL and exploited all cores
This looks like really cool stuff and I'd throw in a few dollars if you made a kickstarter to make it work under OSX / Linux. I love the Python language but am currently surrounded by people who ...
/u/git on PyParallel: how we removed the GIL and exploited all cores
I disagree with literally everything in this comment. Most importantly I disagree with your distinction between scripting languages and other languages. Give me a definition that makes that diffe...
/u/git on PyParallel: how we removed the GIL and exploited all cores
I have never once, in nearly 5 years of predominantly python development, encountered non-trivial problems with indentation. In multi-developer environments. In multi-organisation environments ev...
/u/Eoinoc on PyParallel: how we removed the GIL and exploited all cores
I couldn't agree more!
/u/ on PyParallel: how we removed the GIL and exploited all cores
Good slides. ignore the haters.
/u/willvarfar on PyParallel: how we removed the GIL and exploited all cores
Its a very nice talk. And its all right. There's a couple of things I want to clarify though: I explained completion-based IO to Guido when he started on the async IO for Python; I guess it was...
/u/ on PyParallel: how we removed the GIL and exploited all cores
Honest question: if main threads are read-only for parallel threads and parallel threads are "shared nothing", then what's the benefit of threading at all? Such threads are practically as limited...
/u/pingveno on PyParallel: how we removed the GIL and exploited all cores
I love it when I'm tutoring beginning computer science students. Before my university started using Python for our beginning CS class, many students seemed to be less than familiar with indentati...
/u/trentnelson on PyParallel: how we removed the GIL and exploited all cores
Rushed is an understatement :-) The sound/AV guy came up to me afterward and was like... "Dude. You talk pretty fast." ....and I was still 20 slides off finishing!
/u/trentnelson on PyParallel: how we removed the GIL and exploited all cores
Having a peak at his post history makes it a little easier to understand the rein of downvotes he seems to attract ;-)
/u/gargantuan on PyParallel: how we removed the GIL and exploited all cores
> the amount of pain caused by tabs vs. spaces and wrong indents in a > multi developer project is ugly. What about Python in 2013 do you not like? What would you suggest shou...
/u/gargantuan on PyParallel: how we removed the GIL and exploited all cores
> the amount of pain caused by tabs vs. spaces and wrong indents in a > multi developer project is ugly. I have been programming in Python for 7 years and never worried about ...
/u/ on PyParallel: how we removed the GIL and exploited all cores
Some people love to circlejerk around GIL, dynamic typing and "significant whitespace". It's a troll fest as old as Python itself. Just ignore it.
/u/NoahTheDuke on PyParallel: how we removed the GIL and exploited all cores
Yeah, I much prefer Ruby's 2 spaces.
/u/flying-sheep on PyParallel: how we removed the GIL and exploited all cores
people are stupid for downvoting that person and automatically assuming that (s)he isn’t serious. you chose to write PyParallel exclusively for windows because windows does those things right...
/u/flying-sheep on PyParallel: how we removed the GIL and exploited all cores
i don’t get why you’re downvoted. maybe people assume you’re being sarcastic, but if that way to do it is actually superiour, it would be awesome to have, and i’d love to see it in action...
/u/rlbond86 on PyParallel: how we removed the GIL and exploited all cores
WTF is with all the python hate in this thread? Python is one of the most popular languages in the world.
/u/rlbond86 on PyParallel: how we removed the GIL and exploited all cores
I love significant whitespace. In C if my indents don't match the braces it's a subtle bug. In python it's an error.
/u/flying-sheep on PyParallel: how we removed the GIL and exploited all cores
> One standard: 4 spaces i agree with the former part, but the latter is a per-project convention, not universal (e.g. i’ll choose tabs for every project, because they’re more semantic, ...
/u/ on PyParallel: how we removed the GIL and exploited all cores
I was at the presentation. He probably rushed through the slides for a beginner like me . Thanks for the link to the slides.
/u/alecco on PyParallel: how we removed the GIL and exploited all cores
The meat seams to start at slide 91.
/u/dcxi on PyParallel: how we removed the GIL and exploited all cores
I actually wrote a tool that checks stuff like this across entire repositories allowing to select applicable file extensions. Tabs vs spaces, newlines or control characters in filenames, cr/lf/cr...
/u/Jdonavan on PyParallel: how we removed the GIL and exploited all cores
I've always had a fondness for Python. I just can't stand significant whitespace.
/u/username223 on PyParallel: how we removed the GIL and exploited all cores
I'd rather download a file than deal with yet another bizarre pile of Javascript, which is probably larger than the original PDF, and not available offline.
/u/trentnelson on PyParallel: how we removed the GIL and exploited all cores
Oh, don't get me wrong, I can totally sympathize. It's only a matter of time as a software engineer before you accidentally commit a file with the wrong line endings or indentation settings due t...
/u/josefx on PyParallel: how we removed the GIL and exploited all cores
> The only way to deal with that is to rule with a ruthless iron fist. We use python to automate tasks. Out of our developers there are two who write the scripts and know which ed...
/u/trentnelson on PyParallel: how we removed the GIL and exploited all cores
Indeed. But then you'd have a much bigger problem of having to use Ruby :-)
/u/trentnelson on PyParallel: how we removed the GIL and exploited all cores
The only way to deal with that is to rule with a ruthless iron fist. One standard: 4 spaces. Anything else gets blocked at commit or instantly manually reverted. Institutionalize it at every leve...
/u/josefx on PyParallel: how we removed the GIL and exploited all cores
It is a nice and flexible scripting language with bindings for many useful libraries. My only problem with it is that "from future import braces" is an April fools joke - the amount of pain cause...
/u/trentnelson on PyParallel: how we removed the GIL and exploited all cores
Dunno', not everyone likes downloading PDFs? ;-)
/u/trentnelson on PyParallel: how we removed the GIL and exploited all cores
>> I'd rather add kernel support for thread-agnostic I/O > > Sounds great. I'm looking forward to your patches. o_O I was being serious. I'd rather implement kernel suppor...
/u/username223 on PyParallel: how we removed the GIL and exploited all cores
Why not just link to the PDF ?
/u/trentnelson on PyParallel: how we removed the GIL and exploited all cores
That's unfortunate. I spend a good twenty minutes describing how IOCP works and why it's such a good design for doing high-performance, multi-threaded I/O. The key is Windows' support for the n...
/u/amstan on PyParallel: how we removed the GIL and exploited all cores
I'm afraid i closed the presentation when i saw that.
/u/trentnelson on PyParallel: how we removed the GIL and exploited all cores
Slides for the talk are available here .