A recent experiment, which will be discussed in due course, has taken me back to thinking about parallel computing in Python using the multiprocessing library. This also reminds me of a conversation that I had with David De Roure as we walked to a seminar after some experiments using Raspberry Pi (2 and 3) boards to do some MPI programming on them. There is some mileage here I think to support different forms of computing and processing of the data without resorting to just adding more machines or memory.
It also invites a rethinking of how a job is processed and the parts that can be split apart and either gathered later or independently processed. There are computational concerns regarding the CPU and its cores rather than taking a number and merely throwing processes or threads at the issue. I suspect, but this needs more work, that there is some ground to working with Galloway’s Uncomputable here but more anon.
No Comments