JSON parse error

[micahpotts]

I'm having trouble with a high number of records not being indexed and I'm not sure what's causing the error. Can you help determine if it's a config on my end or an issue?

Here is my config:

# Config for logger_logstash_backend
config :logger,
  backends: [{LoggerLogstashBackend, :error_log}, :console],
  format: "$time $metadata[$level] $message\n",
  metadata: [:request_id]

config :logger, :error_log,
  host: "logstash.hostname.com",
  port: 10_001,
  level: :info,
  type: "elixir_log",
  metadata: [
    application_name: "og_v2",
    env: "prod"
  ]

Here's the input filter:

udp {
    codec => json
    port => 10001
    type => elixir_log
    queue_size => 10000
    workers => 10
  }

Here's the output filter:

}else if [type] == "elixir_log" {
      elasticsearch {
                hosts => ["elasticsearch.hostname.com:9200"]
                index => "%{[fields][application_name]}-%{+YYYY.MM.dd}"
                document_type => "doc"
    }

Here's an example of the log:

[2018-10-10T14:31:37,223][ERROR][logstash.codecs.json     ] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unexpected character ('\' (code 92)): was expecting double-quote to start field name
 at [Source: (String)"{\"@timestamp\":\"2018-10-10T14:31:37+0000\",\"fields\":{\"application\":\"plug\",\"application_name\":\"og_v2\",\"env\":\"prod\",\"file\":\"lib/plug/logger.ex\",\"function\":\"call/2\",\"level\":\"info\",\"line\":34,\"module\":\"Elixir.Plug.Logger\",\"pid\":\"#PID<0.9798.1>\",\"request_id\":\"ha8cd41ahpu56t1905jev0a17s8k8dd3\"},\"message\":\"Sent 200 in 261\xB5s\",\"type\":\"elixir_log\"}"; line: 1, column: 3]>, :data=>"{\\\"@timestamp\\\":\\\"2018-10-10T14:31:37+0000\\\",\\\"fields\\\":{\\\"application\\\":\\\"plug\\\",\\\"application_name\\\":\\\"og_v2\\\",\\\"env\\\":\\\"prod\\\",\\\"file\\\":\\\"lib/plug/logger.ex\\\",\\\"function\\\":\\\"call/2\\\",\\\"level\\\":\\\"info\\\",\\\"line\\\":34,\\\"module\\\":\\\"Elixir.Plug.Logger\\\",\\\"pid\\\":\\\"#PID<0.9798.1>\\\",\\\"request_id\\\":\\\"ha8cd41ahpu56t1905jev0a17s8k8dd3\\\"},\\\"message\\\":\\\"Sent 200 in 261\\xB5s\\\",\\\"type\\\":\\\"elixir_log\\\"}"}

[otherchris]

We're seeing this issue because of how a unicode character µ is being escaped in the message. I can make it work by changing the options on JSX.encode. If there's interest I can make a pull request to allow passing JSX options through in configuration.

[otherchris]

#26

[kenny-evitt]

@otherchris

We're seeing this issue because of how a unicode character µ is being escaped in the message.

I'm a little confused by what you wrote. I used netcat (nc) as a 'dummy' Logstash server and the µ character came thru fine. The problem definitely seems to be with Logstash – it's not accepting that character as being valid UTF-8 (which it totally is). Maybe you meant that you were seeing this because of how that character is not being escaped?

I noticed that other Unicode characters in log messages would result in the error covered in issue #28. The changes in your pull request (#26), along with the option (:uescape) you used in the unit test you added, resolved that error – thanks!

I'm willing to accept that these characters (i.e. any non-ASCII characters) will need to be escaped for Logstash itself – I'm definitely not 'happy' that, despite their claims to the contrary, they don't in fact accept (all possible valid) UTF-8 – but I wonder if another option would be more appropriate for, e.g. a better behaved Logstash.

[kenny-evitt]

@otherchris Thanks again for the pull request – that definitely resolves this error.

I'm still frustrated that Logstash won't just accept UTF-8. From all of the things I read, I vaguely remember someone speculating that it's likely due to Ruby conventions about handling 'printable characters' (and assuming that at least some output devices won't be able to handle anything beyond ASCII characters). [The Logstash JSON codec plugin is written in Ruby.]