Weaviate OpenAi import data rate limit error

Question

I have a book list in json file and i need to import them to weaviate db with opeai - ada module. I will ask questions to OpenAI through a web page about these, and I want it to find relevant answers from the description section of the books I have previously imported, and then return them to me in a meaningful and logical way.

schema = {
    "classes": [
            {
                "class": "PersonalLibrary",
                "description": "Information from Library Books",
                "vectorizer": "text2vec-openai",
                "moduleConfig": {"text2vec-openai": {"model": "ada", "type": "text"}},
                "properties": [
                    {
                        "dataType": ["text"],
                        "description": "The information",
                        "moduleConfig": {"text2vec-openai": {"skip": "false", "vectorizePropertyName": False}},
                        "name": "description",
                    },
                    {
                        "dataType": ["text"],
                        "description": "The title",
                        "moduleConfig": {"text2vec-openai": {"skip": "false", "vectorizePropertyName": False}},
                        "name": "title",
                    },
                    {
                        "dataType": ["text"],
                        "description": "The author",
                        "moduleConfig": {"text2vec-openai": {"skip": "false", "vectorizePropertyName": False}},
                        "name": "author",
                    },
                    {
                        "dataType": ["text"],
                        "description": "The url",
                        "moduleConfig": {"text2vec-openai": {"skip": "false", "vectorizePropertyName": False}},
                        "name": "url",
                    },
                ],
            }
        ]
    }

I have 5000+ book in my json and need to import them to weaveiate

df = pandas.read_json("dummy4.json")

with client.batch as batch:
    for index, row in df.iterrows():
        properties = {
            'title': row['title'],
            'description': row['description'],
            'author': row['author'],
            'url': row['url'],
            'path': "https://books.mypage.com/explore/"+ str(row['id'])
        }
        batch.add_data_object(
                properties,
                "PersonalLibrary",
            )

Then i always take that error, {'error': [{'message': 'update vector: connection to: OpenAI API failed with status: 429 error: Rate limit reached for text-embedding-ada-002 in organization org-WWwo45yj2shuV173thd8l5S5 on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method.'}]} Just 3 books entered weaviate DB.

Can you give me advice, how can i import that datas for search with openai. I need to search some text in description then AI should give a meaningful answer

Answer 1

Duda from Weaviate here :)

You will need to throttle your data ingestion, as described here: https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai#import-throttling

Let me know if that helps.

And thanks for using Weaviate ;)