Parsing JSONL with utils/read_and_validate_jsonl

[midoo45]

Hi everyone,

I want to understand more why during the handling of JSONL file, the behavior is handled by raising an error and stopping the processing of the file.
Sometimes the file is too large and the issue could be in a couple of lines, why don't the code only logging the lines with issues and the rest of the file get processed and the timeline can be created normally?

So instead of handling the exception like this (lib/utils.py line 523)
except ValueError as e: raise errors.DataIngestionError( "Error parsing JSON at line {0:n}: {1:s}".format(lineno, str(e)) )
It could be handled like this for example
except ValueError as e: logger.error("Error parsing JSON at line {0:n}: {1:s}".format(lineno, str(e))) continue

Still worker.log will have the lines that weren't processed.