-
Notifications
You must be signed in to change notification settings - Fork 45
Python3 AsyncIO implementation #55
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
This package already provides asynchronicity ( While those don't provide How would async/await syntax save on IO waiting time if this client is already asynchronous? Something would still have to wait on I/O as is already being done. |
I think a larger point to possibly be made here is the difficulty of using pulsar with other libraries without these features. For example, consider a user trying to write a simple FastAPI server that simply exposes a REST endpoint for a client that gives that latest value in a topic. If they were using Kafka, they could use I'm not even sure on how to nicely implement the FastAPI example with Pulsar client (apart from implementing blocking threads), or if the increased complexity means that for many Python developers they might simply see the provided kafka solutions and be swayed. |
@zbentley Integrating this client with asyncio is frustrating, because it's not entirely clear which thread things are running in and much of asyncio is not threadsafe. In order to get these callbacks to work with asyncio you have to resort to low level threadsafe facilities of asyncio, making it far less trivial to safely integrate this client with asyncio than you portray. For example if you want to use I think the argument that this is strictly syntactical is simply not true. These concurrency models are fundamentally different. Pulsar is using a threaded blocking network model, where threads handle blocking operation as to not block the main thread, these threads are put to sleep by the OS when they make a blocking syscall. Asyncio is driven by an event loop and non-blocking operations running on a single thread. I understand if an asyncio wrapper is outside the scope of this project, but these concurrency models diverge on more than just syntax, and integrating with asyncio is not trivial or straightforward especially for most python developers who may not understand the low level details of how asyncio works. |
Master issue: apache#55 ### Modifications Introduce a `pulsar.asyncio` module that includes the asynchronous APIs to work with Python asyncio module. Example: ```python async def main(): client = Client('pulsar://localhost:6650') try: producer = await client.create_producer('topic') msg_id = await producer.send('msg'.encode()) await producer.close() except PulsarException as e: error = e.error() await client.close() ``` The creation of `Client` reuses the same keyword arguments from `pulsar.Client` but the options for creating producer and sending messages are not added yet.
Master issue: #55 ### Modifications Introduce a `pulsar.asyncio` module that includes the asynchronous APIs to work with Python asyncio module. Example: ```python async def main(): client = Client('pulsar://localhost:6650') try: producer = await client.create_producer('topic') msg_id = await producer.send('msg'.encode()) await producer.close() except PulsarException as e: error = e.error() await client.close() ``` The creation of `Client` reuses the same keyword arguments from `pulsar.Client` but the options for creating producer and sending messages are not added yet.
I see the asyncio module in the reference docs (https://pulsar.apache.org/api/python/3.5.x/pulsar.asyncio.html). However, I get Can someone suggest the proper way to access these classes using the Python Pulsar Client?. Info @BewareMyPower . Does it require a custom build from src? |
It might be something wrong with the release process, let me check it next week. |
I can find this module when I installed it:
Could you check if you installed an old version that is not removed? @daveqs |
Is there any plan to support asyncio for consumer? |
No plan for now |
sounds a bad news. |
Hi guys, i am currently implementing the consumer using asyncio in my fork and would like to open a pull request as soon as its finished. I used the c++ clients Async methods in c++ and futures in python (in the same way as the producer) to archive nonblocking methods. As soon as the code is ready help and feedback would be greatly appreciated. |
Copy from: apache/pulsar#6308
Is your feature request related to a problem? Please describe.
Implement python3 await/ async and using the event loop in order to save IO waiting time
Describe the solution you'd like
Using async, await and execute via the event loop
Describe alternatives you've considered
Using threads although the GIL preventing pure multi threading
The text was updated successfully, but these errors were encountered: