Would you recommend converting message value using the schema before storing in S3 or PostgreSQL sinks?

It really depends on your storage pattern and partitioning scheme. Messages on Kafka topics are serialized and stored in Avro format, directly from Kafka without deserializing. If you need to partition data by some type (messagetype, symbol, etc), then you’ll need to deserialize the message. Nasdaq Data Link converts the Avro messages to Parquet in a consumer before writing to S3. 

Did this answer your question? Thanks for the feedback There was a problem submitting your feedback. Please try again later.