Opensearch Error Unable to upload document:

[midoo45]

Is it wrong to upload different types of logs to the same opensearch index?
When I try to upload plaso file and for example I try to upload JSONL file with MFT analysis results I get the following error:
timesketch.opensearch/ERROR Unable to upload document: RlqiqZIBZRVrk1Gg-1So to index 25172ac2568e4a90ab5917230b6a099c - [201] <Unknown Type> No reason given [Unknown Type/Unknown Detailed Reason]
When I try to upload the two files separately I don't get such error.
So I am asking here in general if this is not a good practice to mix different logs on the same index?
And the reason I try to do so is to make each machine represent by one timeline instead of multiple timelines for one system.

[jkppr]

In theory there should be no issue with uploading different data_types to the same searchindex.
For example: A disk image parsed via Plaso results in a .plaso file that has many different data_types and is imported in the same searchindex for Timesketch.

So the problem is most likely on a import level between Timesketch and Opensearch. I did some quick tests and it looks like the issue is with mixing Plaso imports with jsonl/csv imports. This is most likely due to both types using different importer modules that are not compatible with each other.

So a solution for your use-case to have each timeline represent one machine is to have all logs in one format. So either all Plaso or all json & csv. The easiest is probably to export the Plaso file to jsonl or csv using psort output and formatting. During upload and import in Timesketch just ensure they all use the same Timeline name.

[midoo45]

Please try with the following sample machinet.zip
I adjusted the mappings on my setup to make it work, there is an issue though the total number of events should be 13 (12 from MFT and 1 from Timeline) but the total number of events I have is 26 ( so each event ingested twice may be) I don't know why.
If you have an explanation please let me know.

[midoo45]

@jkppr did you have the time to check the sample files I sent?

[jkppr]

Not yet, sorry. I'll give it a try next week if time permits.

[jkppr]

Alright, I had some time to look into this issue. Thanks for providing the test data, that helped a lot. (And sorry if I state the obvious for you here, but had to test it myself)

So it looks like the order of how you upload stuff matters in this case. Uploading machine_mft.jsonl before machine_tl.jsonl works just fine. The other way round results in the [400] <mapper_parsing_exception> failed to parse field [flags] of type [long] in document with id 'Ml956JIBo-xfjZC0YsDQ'. Preview of field's value: 'ALLOCATED' [illegal_argument_exception/For input string: "ALLOCATED"] error.

Taking a look at the mapping in both cases:

Case 1 - machine_mft.jsonl before machine_tl.jsonl :

        "flags" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },

Case 2 - machine_tl.jsonl before machine_mft.jsonl

        "flags" : {
          "type" : "long"
        },

While parsing long to text works (case 1) it fails the other way round. So we run into a mapping conflict that Timesketch can't solve (but maybe better report back to the user).

Possible solutions I see here:

1. Manual mapping of the field as you already are doing it. And if you can predict your fields well this is for sure the best solution in my opinion.
2. Ensure the pipeline that creates machine_tl.jsonl stores integers as strings and enable the "numeric_detection": true in the index mapping to let Opensearch handle what it will parse and map long and what not. (untested, might have other sideeffects)
3. Check other methods to handle mapping conflicts, e.g. https://opster.com/guides/elasticsearch/glossary/elasticsearch-conflicting-field/

[jkppr]

[...] there is an issue though the total number of events should be 13 (12 from MFT and 1 from Timeline) but the total number of events I have is 26 ( so each event ingested twice may be) I don't know why. If you have an explanation please let me know.

I was not able to reproduce this issue with the default dynamic mapping. Do you mind sharing your custom mapping, maybe it is related?

[jkppr]

Also there are some other observations:
After converting .plaso to .jsonl , I wasn't able to upload the file using the Web GUI because of missing datetime but I can upload it using CLI, I ended up with 11 datasources instead of 1.
The WEB GUI shows multiple of the following:
Original filename: File on disk: File size: NaN Uploaded by: user Provider: CLI importer tool Context: /home/user/.local/bin/timesketch_importer --sketch_id 8 --timeline_name Machine Machine_TL.jsonl Data label: jsonl Status: queueing Total File Events: 0
If I upload the .plaso directly the datasources is "1"
Original filename: Machine_TL.plaso File on disk: /file/path File size: 281.68MB Uploaded by: user Provider: WebUpload Context: Machine_TL.plaso Status: ready Total File Events: 461.7K

This sounds like a bug. Could you create an issue for this and provide some detailed steps to reproduce? Thanks