easy_mp_share/README.md

Easy MP Share

A robust and efficient way to share data between processes using Python's multiprocessing.Manager. This module implements a singleton pattern to ensure consistent data access across different modules and processes.

Features

• Singleton Pattern: Ensures consistent data access across different modules and processes
• Automatic Serialization: Supports both standard pickle and cloudpickle for complex objects
• Dictionary-like Interface: Intuitive API that mimics Python dictionaries
• Thread-Safe: Safe to use in multi-threaded environments
• Process-Safe: Designed for multi-process applications
• Error Handling: Comprehensive error handling and logging
• Resource Management: Automatic cleanup of resources

Installation

The easiest way to use Easy MP Share is:

1. Copy easy_mp_share/core.py to your project as emps.py.
2. Run from emps import MultiProcessingSharedPool, and be happy :-).
3. Any subdirectory will work as long as importing prefix adjusted.
4. (Optional) Add cloudpickle>=3 to your project dependencies if you need sharing local functions/lambdas.

pip install easy-mp-share

Quick Start

Basic Usage

from easy_mp_share import MultiProcessingSharedPool

# Create or get the singleton instance
pool = MultiProcessingSharedPool()

# Store data
pool.put("user_name", "Alice")
pool.put("user_age", 30)
pool.put("config", {"theme": "dark", "language": "en"})

# Retrieve data
name = pool.get("user_name")  # "Alice"
age = pool.get("user_age")    # 30
config = pool.get("config")   # {"theme": "dark", "language": "en"}

# Dictionary-like access
pool["session_id"] = "abc123"
session = pool["session_id"]  # "abc123"

# Check existence
if "user_name" in pool:
    print("User name exists!")

Using the Convenience Instance

from easy_mp_share import shared_pool

# Direct use of the singleton instance
shared_pool.put("api_key", "secret_key")
key = shared_pool.get("api_key")

Advanced Usage with Complex Objects

import multiprocessing as mp
from easy_mp_share import shared_pool

# Store functions and lambdas (requires cloudpickle)
shared_pool.put("processor", lambda x: x * 2)
shared_pool.put("custom_func", my_custom_function)

# Use in different processes
def worker():
    processor = shared_pool.get("processor")
    result = processor(21)  # 42
    print(f"Worker result: {result}")

if __name__ == "__main__":
    p1 = mp.Process(target=worker)
    p2 = mp.Process(target=worker)
    
    p1.start()
    p2.start()
    
    p1.join()
    p2.join()

Context Manager Support

from easy_mp_share import MultiProcessingSharedPool

with MultiProcessingSharedPool() as pool:
    pool.put("temp_data", "value")
    # Auto-cleanup when exiting context

API Reference

MultiProcessingSharedPool

The main class that provides a dictionary-like interface for shared data storage.

Methods

• put(label: str, data: Any) -> bool: Store data with a label
• get(label: str, default: Any = None) -> Any: Retrieve data by label
• exists(label: str) -> bool: Check if label exists
• remove(label: str) -> bool: Remove a label and its data
• pop(label: str, default: Any = None) -> Any: Remove and return data
• size() -> int: Get the number of items stored
• clear(): Clear all shared data
• keys() -> List[str]: Get all labels

Dictionary-like Operations

• pool[key] = value: Store data
• value = pool[key]: Retrieve data
• del pool[key]: Remove data
• key in pool: Check existence

Requirements

• Python 3.8+
• cloudpickle (optional, for complex object serialization)

Development

Installation for Development

git clone https://github.com/your-username/easy-mp-share.git
cd easy-mp-share
pip install -e ".[dev]"

Running Tests

pytest

Code Formatting

black easy_mp_share tests
ruff check easy_mp_share tests

License

This project is licensed under the MIT License - see the LICENSE file for details.

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

Changelog

0.1.0

• Initial release
• Basic multi-process data sharing functionality
• Singleton pattern implementation
• Cloudpickle support for complex objects
• Dictionary-like interface