Both are language-neutral and platform-neutral data exchange libraries. I wonder what are the difference of them and which library is good for which situations.
Related Questions in PROTOCOL-BUFFERS
- Custom rewriter for json
- Cannot resolve method 'merge(String, Builder)' on JsonFormat after upgrading protobuf-java-util from 3.25.3 -> 4.26.0
- "No map fields found in com.google.cloud.compute.v1.Instance" error when getting instances in Google Cloud API Java
- How can I decode a Protocol Buffer that uses NanoPB?
- Is it possible to add a "this is what you should use" message when deprecating a Protocol buffer field?
- File passthrough in Meson project
- Least Connection Load balancing using Grpc
- Fatal Exception: java.lang.VerifyError when using Datastore with my own data class on some devices
- using gdscript procotcol buffers
- Use google.api.field_behavior with protoc-gen-ts_proto
- Unmarshalling json into protobuf having oneof type
- How can I export Pub/Sub messages using a Protobuf schema to a GCS bucket?
- Whats the proper way to fill a recyclerView with data from a proto datastore
- Having shared swagger (openapi) type definitions alongside with protobuf ones
- Errors with reading GTFS tripupdates.pb real time data using get() function
Related Questions in APACHE-ARROW
- How do I locally host an Apache Arrow Flight server using Go and retrieve in Javascript?
- Alternatives for distinct(.keep_all = TRUE) in arrow?
- R arrow query extremely slow first time, fast thereafter?
- Is there any way to stream to a parquet file in Ruby?
- parquet StreamReader giving blank values for few columns, and correct for another?
- How can I order an arrow2 Chunk by a given column in rust?
- How can I read a reqwest::Response object's bytes_stream() with an implementer of arrow_array::RecordBatchReader?
- how to create a dataframe in Rust so it can be used in DataFusion?
- how to create a polars-arrow `Array` from raw values (`&[u8]`)
- How to group arrow table by column value in C++?
- arrow::open_dataset, hive partitioning, and number-like strings
- One-hot-encoding while loading data with arrow-rs
- SQL query on arrow duckdb workflow in R
- Arrow RecordBatch as Polars DataFrame
- apache arrow - array of variant type
Related Questions in DATA-EXCHANGE
- How to integrate Power Automate with BIM360
- Accesses files from outside of the project folder in Angular
- Can't fix `NotFoundException` error when calling SendApiAsset in AWS CLI
- Is Two Way data providing possible in one Snowflake Account between multiple teams from different companies?
- Control anylogic through external programs
- Recovering structs sent over a network
- How to keep track of dirty controls in a dialog
- Comparison of protobuf and arrow
- Lighting diagrams language and notation
- How to create batch process to upload Oracle DB data to AWS Data Exchange?
- Is there a way to return data from the called azure function back to logic app who called it?
- Temporary s3 buckets or different storage methodologies to support AWS batch execution
- What is the equivalent Google Cloud Platform product related to AWS Data Exchange
- How to exchange data between bamboo task and a microservice
- How to send a file from Bamboo task to a our python Microservice
Popular Questions
- How do I undo the most recent local commits in Git?
- How can I remove a specific item from an array in JavaScript?
- How do I delete a Git branch locally and remotely?
- Find all files containing a specific text (string) on Linux?
- How do I revert a Git repository to a previous commit?
- How do I create an HTML button that acts like a link?
- How do I check out a remote Git branch?
- How do I force "git pull" to overwrite local files?
- How do I list all files of a directory?
- How to check whether a string contains a substring in JavaScript?
- How do I redirect to another webpage?
- How can I iterate over rows in a Pandas DataFrame?
- How do I convert a String to an int in Java?
- Does Python have a string 'contains' substring method?
- How do I check if a string contains a specific word?
Popular Tags
Trending Questions
- UIImageView Frame Doesn't Reflect Constraints
- Is it possible to use adb commands to click on a view by finding its ID?
- How to create a new web character symbol recognizable by html/javascript?
- Why isn't my CSS3 animation smooth in Google Chrome (but very smooth on other browsers)?
- Heap Gives Page Fault
- Connect ffmpeg to Visual Studio 2008
- Both Object- and ValueAnimator jumps when Duration is set above API LvL 24
- How to avoid default initialization of objects in std::vector?
- second argument of the command line arguments in a format other than char** argv or char* argv[]
- How to improve efficiency of algorithm which generates next lexicographic permutation?
- Navigating to the another actvity app getting crash in android
- How to read the particular message format in android and store in sqlite database?
- Resetting inventory status after order is cancelled
- Efficiently compute powers of X in SSE/AVX
- Insert into an external database using ajax and php : POST 500 (Internal Server Error)
They are intended for two different problems. Protobuf is designed to create a common "on the wire" or "disk" format for data.
Arrow is designed to create a common "in memory" format for the data.
Of course, the next question, is what does this mean?
In Protobuf, if an application wants to work with the data, they first deserialize the data into some kind of "in memory" representation. This must be done because the Protobuf format is not easily compatible with CPU instructions. For example, protobuf packs unsigned integers into varints. These have a variable # of bytes and the wire-type of the field is crammed into the 3 least significant bits. You cannot take two unsigned integers and just add them without first converting them to some kind of "in memory" representation.
Now,
protocdoes have libraries for every language to convert to an "in memory" representation for those languages. However, this "in memory" representation is not common. You cannot take a Protobuf message, deserialize it into C# (usingprotocgenerated code) and then process on these in-memory bytes in Java without doing some kind of C#->Java marshalling of the data.Arrow, on the other hand, solves this problem. If you have an Arrow table in C# you can map that memory to a different language and start processing on it without doing any kind of "language-to-language" marshaling of data. This zero-copy allows for efficient hand-off between languages. Python has been employing tricks like this (e.g. the array protocol) for a while now and it works great for data analysis.
However, Arrow is not always the greatest format for over-the-wire transmission because it can be inefficient. Those varints I mentioned before help Protobuf cut down on message size. Also, Protobuf tags each field so it can save space when there are many optional fields. In fact, Arrow uses Protobuf & gRPC for over-the-wire transmission of metadata in Arrow Flight (an RPC framework).