util/log: skip malformed log entries while reading from file

Previously, readAllEntriesFromFile was returning an error if the log file
contains the malformed log entry. The malformed log entry primarily could be
the unexpected panic log. This would result in failure in fetching the logs
from file. This was inadequate because single log line can cause failure in
reading rest of the logs which are with valid format. To address this, this
patch skips the malformed log entries during decode and only valid log lines
are returned.

Fixes: #149866
Informs: #151493
Release note: None

Author: aa-joshi

pkg/util/log/file_api.go

@@ -429,6 +429,13 @@ func readAllEntriesFromFile(
 			if err == io.EOF {
 				break
 			}
+
+			// skip the malformed log entry. This is primarily observed in the
+			// panic which is unstructured.
+			// See: https://github.com/cockroachdb/cockroach/issues/151493
+			if errors.Is(err, ErrMalformedLogEntry) {
+				continue
+			}
 			return nil, false, err
 		}
 		var match bool