how to use skywalking python agent by kafka

[xiaozaiyuji]

What happened
I do a test by skywalking python kafka 0.8.0,but it can not find the service in skywalking ui and elasticsearch.I use the oap server with version 9.2.0,and many java agent use kafka reporter had already manage trace by it.But the python application can not find, I check the config many times,and I did many test.I had to get help by the officical ,maybe this is a config error by me,but ,I think it is a rarely mistake by me.

What you expected to happen
I want to see the python application in the skywalking ui,

How to reproduce
Dockerfile is this.

FROM python:3.9

#setting the environment
ENV SW_AGENT_NAMESPACE v92
ENV SW_AGENT_COLLECTOR_BACKEND_SERVICES http://oap-server.address/
ENV SW_AGENT_PROTOCOL kafka
ENV SW_KAFKA_REPORTER_BOOTSTRAP_SERVERS "kafka1:port"
ENV SW_AGENT_INSTANCE skywalking-python-test
ENV SW_AGENT_NAME skywalking-python-test

install telnet
RUN apt-get update &&
apt-get install -y --no-install-recommends telnet &&
apt-get clean &&
rm -rf /var/lib/apt/lists/*;

install dependency
COPY requirements.txt requirements.txt
RUN pip install -r /requirements.txt

print the infomation
RUN pip --version
RUN pip list

RUN mkdir /release
COPY / /release/
WORKDIR /release
EXPOSE 8080
CMD ["sw-python","run","python3","app.py"]`'''

app.py

from aiohttp import web

routes = web.RouteTableDef()

@routes.get('/healthCheck')
async def health_check(request):
return web.Response(text="OK")

if name == "main":
app = web.Application()
app.add_routes(routes)
web.run_app(app, host="0.0.0.0", port=8080)

requirements.txt

apache-skywalking[kafka]
aiohttp==3.7.4

Anything else
the log is as follows

skywalking [MainThread] [WARNING] plugin sw_bottle failed to install, please disregard this warning if the corresponding package was not used in your project
2023-01-29T08:57:38.080541949Z skywalking [MainThread] [WARNING] plugin sw_celery failed to install, please disregard this warning if the corresponding package was not used in your project
2023-01-29T08:57:38.081060151Z skywalking [MainThread] [WARNING] plugin sw_django failed to install, please disregard this warning if the corresponding package was not used in your project
2023-01-29T08:57:38.081177214Z skywalking [MainThread] [WARNING] plugin sw_elasticsearch failed to install, please disregard this warning if the corresponding package was not used in your project
2023-01-29T08:57:38.081522769Z skywalking [MainThread] [WARNING] plugin sw_falcon failed to install, please disregard this warning if the corresponding package was not used in your project
2023-01-29T08:57:38.081794040Z skywalking [MainThread] [WARNING] plugin sw_fastapi failed to install, please disregard this warning if the corresponding package was not used in your project
2023-01-29T08:57:38.081898196Z skywalking [MainThread] [WARNING] plugin sw_flask failed to install, please disregard this warning if the corresponding package was not used in your project
2023-01-29T08:57:38.084788975Z skywalking [MainThread] [WARNING] plugin sw_mysqlclient failed to install, please disregard this warning if the corresponding package was not used in your project
2023-01-29T08:57:38.085171451Z skywalking [MainThread] [WARNING] plugin sw_psycopg failed to install, please disregard this warning if the corresponding package was not used in your project
2023-01-29T08:57:38.085723379Z skywalking [MainThread] [WARNING] plugin sw_psycopg2 failed to install, please disregard this warning if the corresponding package was not used in your project
2023-01-29T08:57:38.085757244Z skywalking [MainThread] [WARNING] plugin sw_pymongo failed to install, please disregard this warning if the corresponding package was not used in your project
2023-01-29T08:57:38.086524734Z skywalking [MainThread] [WARNING] plugin sw_pymysql failed to install, please disregard this warning if the corresponding package was not used in your project
2023-01-29T08:57:38.086535096Z skywalking [MainThread] [WARNING] plugin sw_pyramid failed to install, please disregard this warning if the corresponding package was not used in your project
2023-01-29T08:57:38.086554099Z skywalking [MainThread] [WARNING] plugin sw_rabbitmq failed to install, please disregard this warning if the corresponding package was not used in your project
2023-01-29T08:57:38.086694331Z skywalking [MainThread] [WARNING] plugin sw_redis failed to install, please disregard this warning if the corresponding package was not used in your project
2023-01-29T08:57:38.087051310Z skywalking [MainThread] [WARNING] plugin sw_requests failed to install, please disregard this warning if the corresponding package was not used in your project
2023-01-29T08:57:38.087367258Z skywalking [MainThread] [WARNING] plugin sw_sanic failed to install, please disregard this warning if the corresponding package was not used in your project
2023-01-29T08:57:38.088014390Z skywalking [MainThread] [WARNING] plugin sw_tornado failed to install, please disregard this warning if the corresponding package was not used in your project
2023-01-29T08:57:38.088024525Z skywalking [MainThread] [WARNING] plugin sw_urllib3 failed to install, please disregard this warning if the corresponding package was not used in your project

As a result,the skywalking ui and es can not find the application, I had already confirm that the network about oap server and the kafka server is right.So ,can any body help me to find what's wront

[Superskyyy]

Python Agent on Kafka is covered by E2E on SkyWalking OAP v9.3.0. Anyway I'm gonna try to reproduce the issue since you provided all codes. Please stay tuned.

[xiaozaiyuji]

HI,I use the skywalking oap server 9.2.0,this is different with you.

[Superskyyy]

@xiaozaiyuji I have tested on released python agent [kafka] and everything is working fine with bitnami/confluent kafka. The service is visible in dashboard immediately and traces are there. I used a recent oap revision at v9.3.0 93d021a

Since there's no errors in your log, I assume that kafka reporter didn't throw any exception, that we can 99% sure the data is already sent out.

Other than your code is missing __ for
if name == "main":
should be
if __name__ == "__main__":
I don't see other problems.

I'd like to remind you to double check the kafka broker log AND use provectuslabs/kafka-ui to verify skywalking-consumer and all topics are created, check within management topic to see if messages have been successfully reported to kafka.

It is most likely a kafka setup issue (things like incorrect replication factor exception https://github.com/robcowart/docker_compose_cookbook/blob/dcf7799b96947d20e0f4a185195a5abe96715561/confluent_kafka_oss/docker-compose.yml#L505) outside of SkyWalking agent scope, although I admit other possibilities. I request you to investigate in your environment the above bolded comment and feed the finding to me next.

[xiaozaiyuji]

@Superskyyy I was considering to be a contributer to skywalking,but I now is a low programer to python.I just can read the code by guess,I will try my best to learning.But I will contributing to skywalking to give some useful advice.For example , are we have a consideration about to support java GraalVM native image.
@wu-sheng Dear wu,I now is participate in this test,but I did not know the skywalking is support GraalVM or not.So can I get some advice from you?

[wu-sheng]

Dear wu,I now is participate in this test,but I did not know the skywalking is support GraalVM or not.So can I get some advice from you?

First of all, we never tested it.

https://www.graalvm.org/latest/docs/introduction/

From its architecture, it is still a Java HotSpot VM, which means we have supported.
Have you run into some particular issues?

[wu-sheng]

A tip, we recommend one context per discussion. If you want to discuss GraalVM, please open a new discussion.

[xiaozaiyuji]

Dear wu,I now is participate in this test,but I did not know the skywalking is support GraalVM or not.So can I get some advice from you?

First of all, we never tested it.

https://www.graalvm.org/latest/docs/introduction/

From its architecture, it is still a Java HotSpot VM, which means we have supported. Have you run into some particular issues?

I just begin to test,I will follow your advice and I will open a new discussion

[deep-iimjobs]

@Superskyyy @wallezhang @wu-sheng #12086 (comment) please help

[Superskyyy]

Tracking created #10317