I am trying to merge key-value pairs stored in multiple files. The files themselves are large so I want to do this in memory. The files are sorted, and contain 1 key-value pair per line, so normal k-way merge would work but I don't know how to iterate through the files together. Also in case I encounter multiple data points with the same keys, I want to combine them (add their values together) so the simple solution in Implementing an external merge sort does not work.I have tried using readlines to iterate over the files and storing indices, but readlines loads the entire file into memory so it does not acheive my need. As clarified by comments, the question boils down to reading consecutive lines from multiple files without using a for line in file loop.
Merging files on disk in Python
61 views Asked by Aman Sharma At
1
There are 1 answers
Related Questions in PYTHON
- How to store a date/time in sqlite (or something similar to a date)
- Instagrapi recently showing HTTPError and UnknownError
- How to Retrieve Data from an MySQL Database and Display it in a GUI?
- How to create a regular expression to partition a string that terminates in either ": 45" or ",", without the ": "
- Python Geopandas unable to convert latitude longitude to points
- Influence of Unused FFN on Model Accuracy in PyTorch
- Seeking Python Libraries for Removing Extraneous Characters and Spaces in Text
- Writes to child subprocess.Popen.stdin don't work from within process group?
- Conda has two different python binarys (python and python3) with the same version for a single environment. Why?
- Problem with add new attribute in table with BOTO3 on python
- Can't install packages in python conda environment
- Setting diagonal of a matrix to zero
- List of numbers converted to list of strings to iterate over it. But receiving TypeError messages
- Basic Python Question: Shortening If Statements
- Python and regex, can't understand why some words are left out of the match
Related Questions in MERGE
- Purpose of last 2 while loops in the merge algorithm of merge sort sorting technique
- Having trouble merging these two datasets for a Spatial Analysis
- Merge Azure mp4 blobs via API (Preferred Azure)
- Git merge strategies vs. merge drivers vs. mergetools
- Merge Request in Bitbucket: Possible to exempt a specific branch to ask for Merge Request?
- How to properly extend the generic interface with a new generic parametr using decration merging in Typescript?
- Merge effective dated records of an attribute with the main effective dated table (SQL)
- How do I merge multiple tables into a new table in BigQuery?
- Exclude a file from merging to the main branch
- Usage of merge in linux sort utility
- How can I collapse repeated missing observations into a single nonmissing observation for the same ID in SAS?
- Best way to automate auto-merging git branches
- git: merging a branch that's already been merged by mistake
- Dynamically create, merge & save dataframes in a for loop
- VBA find matching Excel files with a subtext - and merge them into single new file
Related Questions in FILE-HANDLING
- how to edit or update the values of records text file in python using seek and tell function
- How to handle a file handling code that shows no output after first time?
- Unexpected output due to file.readline() when using 'a+'
- how can I detect if a file is open by windows app in WSL and either overwrite or rename?
- Python Seek Function with an offset exceeding the file size
- FileOutputStream not closing properly?
- I'm trying to create an ai chatbot using electron and python. code is working properly but in js part their is problem
- PermissionError 13? Permission denied?
- Merging files on disk in Python
- Receiving multiples files from server
- My tkinter app wont change the file it is loading
- How does the serialisation using child class of ObjectOutputStream does not give error when reading the objects using ObjectInputStream?
- How can I test whether I can read or write a directory?
- Ruby Script Recursively Creates Deeply Nested Directories
- Changing the start address of the IntelHex file using Python
Popular Questions
- How do I undo the most recent local commits in Git?
- How can I remove a specific item from an array in JavaScript?
- How do I delete a Git branch locally and remotely?
- Find all files containing a specific text (string) on Linux?
- How do I revert a Git repository to a previous commit?
- How do I create an HTML button that acts like a link?
- How do I check out a remote Git branch?
- How do I force "git pull" to overwrite local files?
- How do I list all files of a directory?
- How to check whether a string contains a substring in JavaScript?
- How do I redirect to another webpage?
- How can I iterate over rows in a Pandas DataFrame?
- How do I convert a String to an int in Java?
- Does Python have a string 'contains' substring method?
- How do I check if a string contains a specific word?
Popular Tags
Trending Questions
- UIImageView Frame Doesn't Reflect Constraints
- Is it possible to use adb commands to click on a view by finding its ID?
- How to create a new web character symbol recognizable by html/javascript?
- Why isn't my CSS3 animation smooth in Google Chrome (but very smooth on other browsers)?
- Heap Gives Page Fault
- Connect ffmpeg to Visual Studio 2008
- Both Object- and ValueAnimator jumps when Duration is set above API LvL 24
- How to avoid default initialization of objects in std::vector?
- second argument of the command line arguments in a format other than char** argv or char* argv[]
- How to improve efficiency of algorithm which generates next lexicographic permutation?
- Navigating to the another actvity app getting crash in android
- How to read the particular message format in android and store in sqlite database?
- Resetting inventory status after order is cancelled
- Efficiently compute powers of X in SSE/AVX
- Insert into an external database using ajax and php : POST 500 (Internal Server Error)
It seems your real question was "how do I arbitrarily read the next line from a file, without using a
for line in fileloop?"The answer to that is
file.readline().