How can I append data to an existing file in HDFS

Question

I am facing issues in appending data to an existing file in HDFS. I want to append data to an existing file, but instead, it is creating a new file.

I have embedded my code in this query, kindly go through that.

if (!file.exists(path)){
   file.createNewFile(path);
}
FSDataOutputStream fileOutputStream = file.append(path); 
BufferedWriter br = new BufferedWriter(new OutputStreamWriter(fileOutputStream));
br.append("Content: " + content + "\n");
br.close();

The code I used to run it is as follows.

RunTimeCalculationHdfsWrite.hdfsWriteFile("RunTimeParserLoaderMapperTest2", "Error message test 2.2", context, null);

1st parameter is the file name, and the second parameter is the message and you can ignore the others.

Answer 1

You have to do some configurations as shown below :

Set dfs.support.append as true in hdfs-site.xml :

<property>
   <name>dfs.support.append</name>
   <value>true</value>
</property>

Stop all your daemon services using stop-all.sh and restart it again using start-all.sh

If you have a single-node cluster, you have to set the replication factor to 1.

You can use the following command line.

./hdfs dfs -setrep -R 1 filepath/directory

Or you can do the same at run time through java code:

fShell.setrepr((short) 1, filePath);  

Follow this code to append data into your File

public void createAppendHDFS() throws IOException {
    Configuration hadoopConfig = new Configuration();
    hadoopConfig.set("fs.defaultFS", hdfsuri);
    FileSystem fileSystem = FileSystem.get(hadoopConfig);
    String filePath = "/test/doc.txt";
    Path hdfsPath = new Path(filePath);
    fShell.setrepr((short) 1, filePath); 
    FSDataOutputStream fileOutputStream = null;
    try {
        if (fileSystem.exists(hdfsPath)) {
            fileOutputStream = fileSystem.append(hdfsPath);
            fileOutputStream.writeBytes("appending into file. \n");
        } else {
            fileOutputStream = fileSystem.create(hdfsPath);
            fileOutputStream.writeBytes("creating and writing into file\n");
        }
    } finally {
        if (fileSystem != null) {
            fileSystem.close();
        }
        if (fileOutputStream != null) {
            fileOutputStream.close();
        }
    }
}

I hope this helps.