Add README.
This commit is contained in:
176
README.md
Normal file
176
README.md
Normal file
@ -0,0 +1,176 @@
|
||||
# Easy MP Share
|
||||
|
||||
[](https://python.org)
|
||||
[](LICENSE)
|
||||
[](https://github.com/psf/black)
|
||||
|
||||
A robust and efficient way to share data between processes using Python's `multiprocessing.Manager`. This module implements a singleton pattern to ensure consistent data access across different modules and processes.
|
||||
|
||||
## Features
|
||||
|
||||
- **Singleton Pattern**: Ensures consistent data access across different modules and processes
|
||||
- **Automatic Serialization**: Supports both standard pickle and cloudpickle for complex objects
|
||||
- **Dictionary-like Interface**: Intuitive API that mimics Python dictionaries
|
||||
- **Thread-Safe**: Safe to use in multi-threaded environments
|
||||
- **Process-Safe**: Designed for multi-process applications
|
||||
- **Error Handling**: Comprehensive error handling and logging
|
||||
- **Resource Management**: Automatic cleanup of resources
|
||||
|
||||
## Installation
|
||||
|
||||
The easiest way to use *Easy MP Share* is:
|
||||
|
||||
1. Copy `easy_mp_share/core.py` to your project as `emps.py`.
|
||||
2. Run `from emps import MultiProcessingSharedPool`, and be happy :-).
|
||||
3. Any subdirectory will work as long as importing prefix adjusted.
|
||||
4. (Optional) Add `cloudpickle>=3` to your project dependencies if you need sharing local functions/lambdas.
|
||||
|
||||
|
||||
```bash
|
||||
pip install easy-mp-share
|
||||
```
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Basic Usage
|
||||
|
||||
```python
|
||||
from easy_mp_share import MultiProcessingSharedPool
|
||||
|
||||
# Create or get the singleton instance
|
||||
pool = MultiProcessingSharedPool()
|
||||
|
||||
# Store data
|
||||
pool.put("user_name", "Alice")
|
||||
pool.put("user_age", 30)
|
||||
pool.put("config", {"theme": "dark", "language": "en"})
|
||||
|
||||
# Retrieve data
|
||||
name = pool.get("user_name") # "Alice"
|
||||
age = pool.get("user_age") # 30
|
||||
config = pool.get("config") # {"theme": "dark", "language": "en"}
|
||||
|
||||
# Dictionary-like access
|
||||
pool["session_id"] = "abc123"
|
||||
session = pool["session_id"] # "abc123"
|
||||
|
||||
# Check existence
|
||||
if "user_name" in pool:
|
||||
print("User name exists!")
|
||||
```
|
||||
|
||||
### Using the Convenience Instance
|
||||
|
||||
```python
|
||||
from easy_mp_share import shared_pool
|
||||
|
||||
# Direct use of the singleton instance
|
||||
shared_pool.put("api_key", "secret_key")
|
||||
key = shared_pool.get("api_key")
|
||||
```
|
||||
|
||||
### Advanced Usage with Complex Objects
|
||||
|
||||
```python
|
||||
import multiprocessing as mp
|
||||
from easy_mp_share import shared_pool
|
||||
|
||||
# Store functions and lambdas (requires cloudpickle)
|
||||
shared_pool.put("processor", lambda x: x * 2)
|
||||
shared_pool.put("custom_func", my_custom_function)
|
||||
|
||||
# Use in different processes
|
||||
def worker():
|
||||
processor = shared_pool.get("processor")
|
||||
result = processor(21) # 42
|
||||
print(f"Worker result: {result}")
|
||||
|
||||
if __name__ == "__main__":
|
||||
p1 = mp.Process(target=worker)
|
||||
p2 = mp.Process(target=worker)
|
||||
|
||||
p1.start()
|
||||
p2.start()
|
||||
|
||||
p1.join()
|
||||
p2.join()
|
||||
```
|
||||
|
||||
### Context Manager Support
|
||||
|
||||
```python
|
||||
from easy_mp_share import MultiProcessingSharedPool
|
||||
|
||||
with MultiProcessingSharedPool() as pool:
|
||||
pool.put("temp_data", "value")
|
||||
# Auto-cleanup when exiting context
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### MultiProcessingSharedPool
|
||||
|
||||
The main class that provides a dictionary-like interface for shared data storage.
|
||||
|
||||
#### Methods
|
||||
|
||||
- `put(label: str, data: Any) -> bool`: Store data with a label
|
||||
- `get(label: str, default: Any = None) -> Any`: Retrieve data by label
|
||||
- `exists(label: str) -> bool`: Check if label exists
|
||||
- `remove(label: str) -> bool`: Remove a label and its data
|
||||
- `pop(label: str, default: Any = None) -> Any`: Remove and return data
|
||||
- `size() -> int`: Get the number of items stored
|
||||
- `clear()`: Clear all shared data
|
||||
- `keys() -> List[str]`: Get all labels
|
||||
|
||||
#### Dictionary-like Operations
|
||||
|
||||
- `pool[key] = value`: Store data
|
||||
- `value = pool[key]`: Retrieve data
|
||||
- `del pool[key]`: Remove data
|
||||
- `key in pool`: Check existence
|
||||
|
||||
## Requirements
|
||||
|
||||
- Python 3.8+
|
||||
- cloudpickle (optional, for complex object serialization)
|
||||
|
||||
## Development
|
||||
|
||||
### Installation for Development
|
||||
|
||||
```bash
|
||||
git clone https://github.com/your-username/easy-mp-share.git
|
||||
cd easy-mp-share
|
||||
pip install -e ".[dev]"
|
||||
```
|
||||
|
||||
### Running Tests
|
||||
|
||||
```bash
|
||||
pytest
|
||||
```
|
||||
|
||||
### Code Formatting
|
||||
|
||||
```bash
|
||||
black easy_mp_share tests
|
||||
ruff check easy_mp_share tests
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
|
||||
|
||||
## Contributing
|
||||
|
||||
Contributions are welcome! Please feel free to submit a Pull Request.
|
||||
|
||||
## Changelog
|
||||
|
||||
### 0.1.0
|
||||
- Initial release
|
||||
- Basic multi-process data sharing functionality
|
||||
- Singleton pattern implementation
|
||||
- Cloudpickle support for complex objects
|
||||
- Dictionary-like interface
|
||||
Reference in New Issue
Block a user