Let me give an example: I exported 1TB of data yesterday. Today, the database got another 1GB of data. If I try to import the data again today, Sqoop will import 1TB+1GB of data, then I am merging it. So it's a headache. I want to import only new data and append it to the old data. In this way, on a daily basis, I'll pull the RDBMS data into HDFS.
How to import only new data by using Sqoop?
2.2k views Asked by Venu A Positive At
1
There are 1 answers
Related Questions in HADOOP
- Can i move items from a custom list to another list after a specific retention?
- weblogic Ws-security policy vs oasis policy
- Your implementation of PreferenceActivity is vulnerable to fragment injection
- Granting Lync Polcies Via AD Group Member using PowerShell
- Amazon AWS S3 IAM Policy based on namespace or tag
- Reset quota is not working as expected in apigee
- Pundit Policy Scope for Has Many Through Relationship
- invalid according to policy policy condition failed starts-with $content-type ""
- What does S3 Policy Version mean?
- Facebook Log Out required for Unity apps?
Related Questions in IMPORT
- Can i move items from a custom list to another list after a specific retention?
- weblogic Ws-security policy vs oasis policy
- Your implementation of PreferenceActivity is vulnerable to fragment injection
- Granting Lync Polcies Via AD Group Member using PowerShell
- Amazon AWS S3 IAM Policy based on namespace or tag
- Reset quota is not working as expected in apigee
- Pundit Policy Scope for Has Many Through Relationship
- invalid according to policy policy condition failed starts-with $content-type ""
- What does S3 Policy Version mean?
- Facebook Log Out required for Unity apps?
Related Questions in HDFS
- Can i move items from a custom list to another list after a specific retention?
- weblogic Ws-security policy vs oasis policy
- Your implementation of PreferenceActivity is vulnerable to fragment injection
- Granting Lync Polcies Via AD Group Member using PowerShell
- Amazon AWS S3 IAM Policy based on namespace or tag
- Reset quota is not working as expected in apigee
- Pundit Policy Scope for Has Many Through Relationship
- invalid according to policy policy condition failed starts-with $content-type ""
- What does S3 Policy Version mean?
- Facebook Log Out required for Unity apps?
Related Questions in RDBMS
- Can i move items from a custom list to another list after a specific retention?
- weblogic Ws-security policy vs oasis policy
- Your implementation of PreferenceActivity is vulnerable to fragment injection
- Granting Lync Polcies Via AD Group Member using PowerShell
- Amazon AWS S3 IAM Policy based on namespace or tag
- Reset quota is not working as expected in apigee
- Pundit Policy Scope for Has Many Through Relationship
- invalid according to policy policy condition failed starts-with $content-type ""
- What does S3 Policy Version mean?
- Facebook Log Out required for Unity apps?
Related Questions in SQOOP
- Can i move items from a custom list to another list after a specific retention?
- weblogic Ws-security policy vs oasis policy
- Your implementation of PreferenceActivity is vulnerable to fragment injection
- Granting Lync Polcies Via AD Group Member using PowerShell
- Amazon AWS S3 IAM Policy based on namespace or tag
- Reset quota is not working as expected in apigee
- Pundit Policy Scope for Has Many Through Relationship
- invalid according to policy policy condition failed starts-with $content-type ""
- What does S3 Policy Version mean?
- Facebook Log Out required for Unity apps?
Popular Questions
- How do I undo the most recent local commits in Git?
- How can I remove a specific item from an array in JavaScript?
- How do I delete a Git branch locally and remotely?
- Find all files containing a specific text (string) on Linux?
- How do I revert a Git repository to a previous commit?
- How do I create an HTML button that acts like a link?
- How do I check out a remote Git branch?
- How do I force "git pull" to overwrite local files?
- How do I list all files of a directory?
- How to check whether a string contains a substring in JavaScript?
- How do I redirect to another webpage?
- How can I iterate over rows in a Pandas DataFrame?
- How do I convert a String to an int in Java?
- Does Python have a string 'contains' substring method?
- How do I check if a string contains a specific word?
Popular Tags
Trending Questions
- UIImageView Frame Doesn't Reflect Constraints
- Is it possible to use adb commands to click on a view by finding its ID?
- How to create a new web character symbol recognizable by html/javascript?
- Why isn't my CSS3 animation smooth in Google Chrome (but very smooth on other browsers)?
- Heap Gives Page Fault
- Connect ffmpeg to Visual Studio 2008
- Both Object- and ValueAnimator jumps when Duration is set above API LvL 24
- How to avoid default initialization of objects in std::vector?
- second argument of the command line arguments in a format other than char** argv or char* argv[]
- How to improve efficiency of algorithm which generates next lexicographic permutation?
- Navigating to the another actvity app getting crash in android
- How to read the particular message format in android and store in sqlite database?
- Resetting inventory status after order is cancelled
- Efficiently compute powers of X in SSE/AVX
- Insert into an external database using ajax and php : POST 500 (Internal Server Error)
You can use sqoop Incremental Imports:
Sqoop provides an
incremental importmode which can be used to retrieve only rows newer than some previously-imported set of rows.Incremental import arguments:
--check-column (col)Specifies the column to be examined when determining which rows to import.--incremental (mode)Specifies how Sqoop determines which rows are new. Legal values for mode include append and last modified.--last-value (value)Specifies the maximum value of the check column from the previous import.Reference: https://sqoop.apache.org/docs/1.4.2/SqoopUserGuide.html#_incremental_imports
For Incremental Import: You would need to specify a value in a check column against a reference value for the most recent import. For example, if the
–incrementalappend argument was specified, along with–check-column id and –last-value 100, all rows with id > 100 will be imported. If an incremental import is run from the command line, the value which should be specified as–last-valuein a subsequent incremental import will be printed to the screen for your reference. If an incremental import is run from a saved job, this value will be retained in the saved job. Subsequent runs ofsqoop job –execsome Incremental Job will continue to import only newer rows than those previously imported.For importing all the tables at one go, you would need to use sqoop-import-all-tables command, but this command must satisfy the below criteria to work
Each table must have a single-column primary key. You must intend to import all columns of each table. You must not intend to use non-default splitting column, nor impose any conditions via a WHERE clause.
Reference: https://hortonworks.com/community/forums/topic/sqoop-incremental-import/