spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jae Yoon Yoo <j...@fortinet.com>
Subject redirectToLog: new-bee question
Date Thu, 10 Oct 2019 23:48:29 GMT
Hello,

My application is launched by SparkLauncher and I am trying to collect all logs (from launcher,
application, and Spark core(?)) into one log file.
I can see all those from the console (using Eclipse IDE) but can’t make them to be written
in the log file. With the (simplified) code under, only log (in the log file) I see is from
the launcher.

For testing purpose, I used redirectOutput(java.io.File) and with it I can see the application’s
log also in the log file but not Spark core’s.
Spark core’s (if I am not using the name properly, please correct) output - I mean, like
under and the exception messages if happens.

INFO: 19/10/10 15:50:08 INFO Utils: Successfully started service 'sparkDriver' on port 62573.

Can you shed some light for me? Thanks in advance,
/ Jae

Reference: https://stackoverflow.com/questions/45781842/save-spark-launcher-output-to-file
Spark version: 2.4.3
Code example: the launcher and the application (TestLog),

public class LauncherTest {

      final static Logger log = Logger.getLogger(LauncherTest.class);

      public static void main(String[] args) throws Exception {

            String log4jConfPath = "log4j.properties";
            PropertyConfigurator.configure(log4jConfPath);

            String appName = LauncherTest.class.getName();

            log.info("appName=" + appName);  // this I see in the log file and the console

            SparkLauncher launcher = new SparkLauncher()
                        .setAppResource(myJar)
                        .setMainClass(myClass)
                        .setSparkHome(sparkHome)
                        .setMaster("local[*]")
                        .setAppName(appName);

            launcher.redirectToLog(LauncherTest.class.getName());

            SparkAppHandle handle = launcher.startApplication();
            State state = handle.getState();

            while (state == null || !state.isFinal()) {
                  Thread.sleep(1000);
                  state = handle.getState();
                  if (state != null) {
                        log.info("state: " + state.toString());
                  }
            }
      }
}

public class TestLog {

      public static void main(String[] args) throws Exception {

            System.out.println("TestLog: start"); // this I see in the console but not in
the log file

            SparkSession spark = SparkSession.builder().appName("Test Log").getOrCreate();
            Dataset<String> logData = spark.read().textFile(filePath).cache();

            System.out.println("TestLog: end,count=" + logData.count(););

            spark.stop();
      }
}


***  Please note that this message and any attachments may contain confidential and proprietary
material and information and are intended only for the use of the intended recipient(s). If
you are not the intended recipient, you are hereby notified that any review, use, disclosure,
dissemination, distribution or copying of this message and any attachments is strictly prohibited.
If you have received this email in error, please immediately notify the sender and destroy
this e-mail and any attachments and all copies, whether electronic or printed. Please also
note that any views, opinions, conclusions or commitments expressed in this message are those
of the individual sender and do not necessarily reflect the views of Fortinet, Inc., its affiliates,
and emails are not binding on Fortinet and only a writing manually signed by Fortinet's General
Counsel can be a binding commitment of Fortinet to Fortinet's customers or partners. Thank
you. *** 

Mime
View raw message