How to write Kafka connector to integrate with Facebook API

Question

I am trying to write a Kafka connector to fetch data from the facebook. My problems are, 

1. How to fetch data from facebook through their API without exceeding the limit of API hit provided by Facebook? The connector should call facebook API for data after a specific time interval so that the number of hits won't exceed.

2. Each user can hit the API with their Access Token so users can't share the same topic partition. So how to handle this scenario. Do we have to create one partition for each user?

I read a few guides and blogs to understand Kafka connect and write a connector.

Kafka Documentation- https://kafka.apache.org/documentation/#connect

Conceptually It gave me an idea about what is Kafka connect, how it works and what are the important classes to write a Kafka connector. But still, I am confused that practically how to write and run a connector. I tried to find step by step development guide but didn't get. 

Any tutorial or pdf If you could suggest which have detailed step by step development guide to write and run Kafka connector.

Answer 1

If you know the limit of API hit provided by Facebook, then you can limit the number of objects returned. To do this, you will have to add the .limit() argument to any field or an edge. For example:

curl -i -X GET \
  "https://graph.facebook.com/{your-user-id}
    ?fields=feed.limit(3)
    &access_token={your-access-token}"

The limit(3) will limit the number of objects in each page to three. Replace this with the limit that Facebook has set and I think it should work. 

Hope this will help.

To know more about Kafka, get your Apache Kafka certification course today.

Thank You