Why does my yarn application not have logs even with logging enabled

Question

I have enabled logs using yarn-site.xml. After this I restarted yarn as well.

After running a sample MR application, I was able to see the applicationID using command: 

yarn application -list. 

Now, for checking logs when I issue the command - yarn logs -applicationId <application ID>, and I am get the following prompt instead of logs:

hdfs://<ip address>/var/log/hadoop-yarn/path/to/application/  does not have any log files

Please help me to figure out what went wrong and what is the correct way to check logs.

Thank you.

Answer 1

yarn application -list command shows you all the application having the following state: SUBMITED ACCEPTED RUNNING Logs are always collected once the application has been finished into dir set to yarn.nodemanager.remote-app-log-dir.Basically, the Log Aggregation feature aggregates logs across all containers on a worker node and stores them as one aggregated log file per worker node after the application has been finished.

 Hence, the application for which you are trying to get the logs has not been completed yet. So, wait for the application to be finished and then you use the same command i.e.: yarn logs -applicationId <your_application_id>

 To check the state of all application use command:

 yarn application -list -appStates FINISHED ALL