Logo-amall

**Describe the bug** { "uid": 0, "indexUid": "post_history", "status": "failed", "type": "documentAdditionOrUpdate", "details": { "receivedDocuments": 360461, "indexedDocuments": 0 }, "error": { "message": "Too many open files (os error 24)", "code": "internal", "type": "internal", "link": "https://docs.meilisearch.com/errors#internal" }, "duration": "PT99.484276S", "enqueuedAt": "2022-07-31T03:20:03.724226Z", "startedAt": "2022-07-31T03:20:03.724793Z", "finishedAt": "2022-07-31T03:21:43.209069Z" } **To Reproduce** Steps to reproduce the behavior: 360461 documents of forum small posts I had created an index on my M1 macbook 32GB Ram, Unable to inject via Rust api. **Expected behavior** A clear and concise description of what you expected to happen. **Screenshots** If applicable, add screenshots to help explain your problem. **Meilisearch version:** [e.g. v0.20.0] 0.28.0 **Additional context** Mac OS, 32 GB, M1Pro

Last active 5 months ago

5 replies

27 views

  • CS

    I split the data set into 100K chunks of documents, then the injection is okay. Any way to have Meilisearch inject 1 million documents?

  • OL

    Hi @cseblog

    Regarding the error, the default setting for the maximum number of open files might be too low, as is usually the case on most machines. Here is an article to help you better understand the problem and implement the solution to fix it. To index 1M records, I'd advise updating the default searchableAttributeslist to only contain fields you want to search. You should send your documents in batches, but I strongly advise you to try the auto-batching feature to speed up the indexing process. About disk space, we recommend reserving at least 10 times the size of your raw data, meaning if your document size is 10GB, you should at least reserve 100GB of SSD disk space for Meilisearch. See more performance optimization suggestions in the dedicated documentation.

    Hope this helps. Thanks.

  • CS

    Thanks for your nice reply. I will check it out.

  • CS

    Describe the bug
    {
    "uid": 0,
    "indexUid": "post_history",
    "status": "failed",
    "type": "documentAdditionOrUpdate",
    "details": {
    "receivedDocuments": 360461,
    "indexedDocuments": 0
    },
    "error": {
    "message": "Too many open files (os error 24)",
    "code": "internal",
    "type": "internal",
    "link": "https://docs.meilisearch.com/errors#internal"
    },
    "duration": "PT99.484276S",
    "enqueuedAt": "2022-07-31T03:20:03.724226Z",
    "startedAt": "2022-07-31T03:20:03.724793Z",
    "finishedAt": "2022-07-31T03:21:43.209069Z"
    }
    To Reproduce

    Steps to reproduce the behavior:
    360461 documents of forum small posts
    I had created an index on my M1 macbook 32GB Ram,
    Unable to inject via Rust api.

    Expected behavior
    A clear and concise description of what you expected to happen.

    Screenshots
    If applicable, add screenshots to help explain your problem.

    Meilisearch version: [e.g. v0.20.0]
    0.28.0
    Additional context
    Mac OS, 32 GB, M1Pro

  • CU

    I close this issue due to a lack of answers. Nothing is definitive of course, feel free to ask to re-open if needed 😊

Last active 5 months ago

5 replies

27 views