首页 \ 问答 \ 使用processing.Manager时对象是否重复?(Object duplicated when using processing.Manager?)

使用processing.Manager时对象是否重复?(Object duplicated when using processing.Manager?)

我正在尝试使用processing.Manager在进程之间共享数据(processing.Manager.list,processing.Manager.dict,...)。 但是,当我使用它时,对象是重复的:

from multiprocessing import Process, Manager


def manipulate_in_process(objects):
    objs = []
    for obj in objects:
        objs.append(obj)
    print('foo objs', objs)

def do_in_process(objects):
    print('objects', objects)
    manipulate_in_process(objects)

if __name__ == '__main__':
    manager = Manager()
    objects = [object(), object()]
    print('main objects', objects)
    shared_objects = manager.list(objects)
    process = Process(target=do_in_process, args=(shared_objects,))
    process.start()
    process.join()
    print('finish')

输出:

main objects [<object object at 0xb72a64a8>, <object object at 0xb72a64b0>]
objects [<object object at 0xb72a64a8>, <object object at 0xb72a64b0>]
foo objs [<object object at 0xb72a64e8>, <object object at 0xb72a64c0>]
finish

使用时共享对象重复: 0xb72a64a8 != 0xb72a64e8

  1. 我做错了?
  2. 为什么这些对象是重复的?
  3. 如何使用processing.Manager.list / dict / etc中的对象?

I'm trying to use processing.Manager to share data between processes (processing.Manager.list, processing.Manager.dict, ...). But, when i use it, object are duplicated:

from multiprocessing import Process, Manager


def manipulate_in_process(objects):
    objs = []
    for obj in objects:
        objs.append(obj)
    print('foo objs', objs)

def do_in_process(objects):
    print('objects', objects)
    manipulate_in_process(objects)

if __name__ == '__main__':
    manager = Manager()
    objects = [object(), object()]
    print('main objects', objects)
    shared_objects = manager.list(objects)
    process = Process(target=do_in_process, args=(shared_objects,))
    process.start()
    process.join()
    print('finish')

Output:

main objects [<object object at 0xb72a64a8>, <object object at 0xb72a64b0>]
objects [<object object at 0xb72a64a8>, <object object at 0xb72a64b0>]
foo objs [<object object at 0xb72a64e8>, <object object at 0xb72a64c0>]
finish

Shareds objects are duplicated when used: 0xb72a64a8 != 0xb72a64e8.

  1. I'm doing it wrong ?
  2. Why these objects are duplicated ?
  3. How can i use objects with processing.Manager.list/dict/etc ?

原文:https://stackoverflow.com/questions/27114266
更新时间:2019-12-07 10:49

最满意答案

在进程之间复制而不是共享对象几乎是固有的。 这是单独进程的定义:它们具有单独的堆与单独的对象。

如果您想要实际的共享数据,则必须使用ValueArraysharedctypes - 但在这种情况下,您只能使用“本机”值以及可以根据C结构表达的值。 而且,即使这样,也不能保证id在整个进程中是相同的,只是值本身就是这样。

Manager的要点是,您可以在服务器进程中放置一个对象,例如list ,并在每个子进程中放置该对象的代理。 就核心列表功能而言,列表代理看起来像一个真实的列表(迭代,索引, appendinsertindex等)。 但是代理并没有试图伪造像id这样的“隐藏”内省功能。

Manager可以直接代理的确切类型列表在Python版本之间有所不同,但您可以在您的版本的文档中找到它。 从3.4开始,它包括listdictNamespaceLockRLockSemaphoreBoundedSemaphoreConditionEventBarrierQueueValueArray 。 无论如何,它永远不会包含object ,因为object没有超出隐藏内容的功能。

当然,您可以使用自定义代理向Manager注册自己的类型。 但我不确定你想与object分享什么。

有关完整详细信息,请参阅概述的流程管理器 之间的共享状态


It's pretty much inherent that objects are copied, not shared, between processes. That's the definition of separate processes: that they have separate heaps with separate objects.

If you want actual shared data, you have to use Value, Array, or sharedctypes—but of course in that case, you're limited to "native" values and those you can express in terms of C structs. And, even then, there's no guarantee that the id will be the same across processes, only that the value itself will be.

The point of Manager is that you can put an object like, say, a list in a server process, and a proxy to that object in each child process. The list proxy looks like a real list as far as core list functionality goes (iteration, indexing, append, insert, index, etc.). But the proxy doesn't try to fake "under-the-covers" introspection functionality like id.

The exact list of types that can be proxied by Manager out of the box differs between Python versions, but you can find it in the docs for your version. As of 3.4, it includes list, dict, Namespace, Lock, RLock, Semaphore, BoundedSemaphore, Condition, Event, Barrier, Queue, Value and Array. At any rate, it's never going to include object, because object has no functionality beyond the under-the-covers stuff.

Of course you can register your own types with the Manager with custom proxies. But I'm not sure what you'd want to share with object.

See Sharing state between processes for an overview and Managers for full details.

2014-11-24

相关文章

更多

最新问答

更多
  • 这个listArray是如何填充的?(How is this listArray populated?)
  • iOS 7上的CTSubscriber(以及如何使用它)是什么?(What's CTSubscriber (and how to use it) on iOS 7?)
  • 手动创建VisualStudio 2012项目文件(Manually creating VisualStudio 2012 project file)
  • 删除不适用于JSP中使用for循环的每个id(Deletion not working for every id using for loop in JSP)
  • 如何从std :: filesystem :: path中删除引号(How to remove quotation marks from std::filesystem::path)
  • 验证多个控制器方法的URL路径(Validate URL path for several controller methods)
  • 如何在datarow []中的列中找到最大值?(How to find max value in a column in a datarow[] ?)
  • 如何使用预定义文本替换来自数据库的部分结果(How do I replace part of result coming from Database with predefined text)
  • Selenium Java注入了新的Javascript函数(Selenium Java inject new Javascript function)
  • 使用.on的多个下拉菜单选择文本仅适用于第一个下拉列表(Multiple Dropdowns Menu Selection text using .on works only on first dropdown)
  • 快速将黄土曲线添加到大型数据集图中的方法(Quick way to add loess curve to large data set graph)
  • FilteringSelect in mvc(FilteringSelect in mvc)
  • 在Delphi XE2中开发Mac或iOS应用程序需要哪些硬件/软件?(What hardware/software is necessary to develop Mac or iOS apps in Delphi XE2?)
  • 在原型的构造函数中初始化属性时获取“未定义”(Getting 'undefined' when a property is initialized in the constructor of a prototype)
  • 通过越狱加载的应用程序的Documents文件夹位置(Location of Documents folder for an app loaded via jailbreak)
  • 在OpenGL中使用可编程和固定管道功能(Using both programmable and fixed pipeline functionality in OpenGL)
  • 将任何用户输入重定向到单独的底层程序(redirect any user input to a separate underlying program)
  • 编辑文本不能正常工作android(Edit texts not working properly android)
  • “user_denied”Facebook应用页面上的Facebook用户区域设置(Facebook user locale on “user_denied” facebook app page)
  • 在大图像中找到小的部分透明图像的坐标(find coordinates of small partially-transparent image within a large image)
  • 我如何在cakephp 3.1中获得完整的相对路径?(How i can get full relative path of image in cakephp 3.1?)
  • 如何保存拖动标记的新本地化?(How to save new localization of dragged marker?)
  • MySQL UPDATE vs INSERT和DELETE(MySQL UPDATE vs INSERT and DELETE)
  • 在执行查询之前,在SQLAlchemy模型中将datetime转换为unix时间戳?(Convert datetime to unix timestamp in SQLAlchemy model before executing query?)
  • OpenCL与OpenGL互操作的优势(Advantage of OpenCL interoperability with OpenGL)
  • 如何解析用点和等分隔的数据然后添加到listview(How to parsing data from delimited with dot and equal then add to listview)
  • 带调试输出的X3解析器段错误(BOOST_SPIRIT_X3_DEBUG)(X3 parser segfaults with debug output (BOOST_SPIRIT_X3_DEBUG))
  • 将文件夹名称添加到fgrep结果(Add folder name to fgrep result)
  • 在MySQL中加载一个表是非常慢的(Loading one table in MySQL is ridiculously slow)
  • 如何将JSON放入PHP变量?(How do I put JSON into a PHP Variable?)