Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sending empty parameter hashmap in case of retried execution #627

Open
wants to merge 5 commits into
base: tuning_20190221
Choose a base branch
from
Open

Sending empty parameter hashmap in case of retried execution #627

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
3b1a7cb
Sending empty parameter hashmap in case of retried execution
dushyantk1509 Aug 30, 2019
c543f64
Bug fix while calculating best parameter for retired jobs (#629)
dushyantk1509 Sep 12, 2019
7c39663
Integrating Spark and MR exception fingerprinting (#623)
ShubhamGupta29 Sep 16, 2019
a5d0d78
Merge branch 'fixRetry2' of github.com:dushyantk1509/dr-elephant into…
dushyantk1509 Sep 17, 2019
e624a83
Addressing comments
dushyantk1509 Sep 17, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 10 additions & 7 deletions app/com/linkedin/drelephant/tuning/AutoTuningAPIHelper.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -441,19 +441,22 @@ private List<JobSuggestedParamValue> processParameterTuningEnabled(TuningInput t
JobExecution jobExecution) {
JobSuggestedParamSet jobSuggestedParamSet;
logger.debug("Finding parameter suggestion for job: " + jobExecution.job.jobName);
List<JobSuggestedParamValue> jobSuggestedParamValues = null;
if (tuningInput.getRetry()) {
logger.info(" Retry ");
logger.info(" Retried job execution " + tuningInput.getJobExecId());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this method called when execution is re-tried or is about to retry?

Copy link
Author

@dushyantk1509 dushyantk1509 Sep 5, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

About to retry. Will correct in next commit.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks.

applyPenalty(tuningInput.getJobExecId());
jobSuggestedParamSet = getBestParamSet(tuningInput.getJobDefId());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will there be no parameters suggested in case of retry?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No. It sends empty hashmap so that default parameters can be applied for retried execution.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So was getting bestParameters and sending them was a BUG which you fixed in this PR?

Copy link
Author

@dushyantk1509 dushyantk1509 Sep 9, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes because after getting these parameters dr. elephant is adding the retried job execution to TuningJobExecutionParamSet against these parameters which is not correct as this is a dummy call. Retried execution always run with default parameters. It ignores the parameters sent by dr. elephant.

Therefore this step was unnecessary step and next step was not correct.

} else {
logger.debug("Finding parameter suggestion for job: " + jobExecution.job.jobName);
if (debugEnabled) {
logger.debug("Finding parameter suggestion for job: " + jobExecution.job.jobName);
}
jobSuggestedParamSet = getNewSuggestedParamSet(jobExecution.job, tuningInput.getTuningAlgorithm());
markParameterSetSent(jobSuggestedParamSet);
addNewTuningJobExecutionParamSet(jobSuggestedParamSet, jobExecution);
jobSuggestedParamValues = getParamSetValues(jobSuggestedParamSet.id);
if(debugEnabled) {
logger.debug("Number of output parameters for execution " + tuningInput.getJobExecId() + " = " + jobSuggestedParamValues.size());
}
}
addNewTuningJobExecutionParamSet(jobSuggestedParamSet, jobExecution);
List<JobSuggestedParamValue> jobSuggestedParamValues = getParamSetValues(jobSuggestedParamSet.id);
logger.debug("Number of output parameters for execution " + tuningInput.getJobExecId() + " = "
+ jobSuggestedParamValues.size());
logger.info("Finishing getCurrentRunParameters");
return jobSuggestedParamValues;
}
Expand Down
9 changes: 5 additions & 4 deletions test/com/linkedin/drelephant/tuning/AlertingTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
import static common.DBTestUtil.*;

import models.TuningJobDefinition;
import models.JobDefinition;
import org.apache.hadoop.conf.Configuration;
import static com.linkedin.drelephant.tuning.alerting.Constant.*;

Expand Down Expand Up @@ -47,7 +48,7 @@ private void testDeveloperAlerting() {



JobSuggestedParamSet jobSuggestedParamSet = JobSuggestedParamSet.find.select("*").where().findUnique();
JobSuggestedParamSet jobSuggestedParamSet = JobSuggestedParamSet.find.select("*").where().eq(JobSuggestedParamSet.TABLE.id, "1137").findUnique();
jobSuggestedParamSet.updatedTs = new Timestamp(startTime + 100);
jobSuggestedParamSet.createdTs = new Timestamp(endTime+1-259200000);
jobSuggestedParamSet.update();
Expand Down Expand Up @@ -77,15 +78,15 @@ private void testDeveloperAlerting() {
/**
* If user want to test email functionality
*/
// assertTrue(" Email send successfully ", manager.sendNotification(notificationDataAfterUpdate));
// assertTrue(" Email send successfully ", manager.sendNotification(notificationDataAfterUpdate));
}

private void testSKAlerting(){
Configuration configuration = ElephantContext.instance().getAutoTuningConf();

long startTime = System.currentTimeMillis();
long endTime = System.currentTimeMillis() + 1000;
TuningJobDefinition tuningJobDefinition = TuningJobDefinition.find.select("*").where().findUnique();
TuningJobDefinition tuningJobDefinition = TuningJobDefinition.find.select("*").where().eq(TuningJobDefinition.TABLE.job + "." + JobDefinition.TABLE.id, 100003).findUnique();
tuningJobDefinition.updatedTs = new Timestamp(startTime + 100);
tuningJobDefinition.tuningEnabled=false;
tuningJobDefinition.autoApply=true;
Expand All @@ -108,4 +109,4 @@ private void testSKAlerting(){
//assertTrue(" Email send successfully ", manager.sendNotification(notificationData));
}

}
}
43 changes: 36 additions & 7 deletions test/com/linkedin/drelephant/tuning/AutoTunerApiTestRunner.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@
import models.FlowExecution;
import models.JobSuggestedParamSet;
import models.JobSuggestedParamValue;
import models.TuningAlgorithm;
import org.junit.Test;

import static org.junit.Assert.*;
Expand Down Expand Up @@ -69,16 +70,44 @@ public void run() {
List<FlowExecution> flowExecution = FlowExecution.find.where().eq(FlowExecution.TABLE.flowExecId, 1).findList();
assertTrue(" Flow Execution ", flowExecution.size()==1);
populateTestData();
testJobSuggestedParamValueListToMap();
testGetcurrentRunParameter();
}

private void testJobSuggestedParamValueListToMap(){
List<JobSuggestedParamValue> jobSuggestedParamValues = JobSuggestedParamValue.find.where()
.eq(JobSuggestedParamValue.TABLE.jobSuggestedParamSet + '.' + JobSuggestedParamSet.TABLE.id, 1137)
.findList();
private void testGetcurrentRunParameter(){
TuningInput tuningInput = new TuningInput();
tuningInput.setFlowDefId("https://elephant.linkedin.com:8443/manager?project=AzkabanHelloPigTest&flow=countByCountryFlow");
tuningInput.setJobDefId("https://elephant.linkedin.com:8443/manager?project=AzkabanHelloPigTest&flow=countByCountryFlow&job=countByCountryFlow_countByCountry");
tuningInput.setFlowDefUrl("https://elephant.linkedin.com:8443/manager?project=AzkabanHelloPigTest&flow=countByCountryFlow");
tuningInput.setJobDefUrl("https://elephant.linkedin.com:8443/manager?project=AzkabanHelloPigTest&flow=countByCountryFlow&job=countByCountryFlow_countByCountry");
tuningInput.setFlowExecId("https://elephant.linkedin.com:8443/executor?execid=5416294");
tuningInput.setJobExecId("https://elephant.linkedin.com:8443/executor?execid=5416294&job=countByCountryFlow_countByCountry&attempt=0");
tuningInput.setFlowExecUrl("https://elephant.linkedin.com:8443/executor?execid=5416294");
tuningInput.setJobExecUrl("https://elephant.linkedin.com:8443/executor?execid=5416294&job=countByCountryFlow_countByCountry&attempt=0");
tuningInput.setJobName("countByCountryFlow_countByCountry");
// tuningInput.setUserName("dukumar");
tuningInput.setClient("azkaban");
tuningInput.setScheduler("azkaban");
// tuningInput.setDefaultParams(defaultParams);
tuningInput.setVersion(1);
tuningInput.setRetry(true);
// tuningInput.setSkipExecutionForOptimization(skipExecutionForOptimization);
tuningInput.setJobType("PIG");
tuningInput.setOptimizationAlgo("HBT");
tuningInput.setOptimizationAlgoVersion("4");
tuningInput.setOptimizationMetric("RESOURCE");
tuningInput.setAllowedMaxExecutionTimePercent(null);
tuningInput.setAllowedMaxResourceUsagePercent(null);

AutoTuningAPIHelper autoTuningAPIHelper = new AutoTuningAPIHelper();
Map<String, Double> paramValues = autoTuningAPIHelper.jobSuggestedParamValueListToMap(jobSuggestedParamValues);
/**
* Testing for empty parameter hashMap if execution is retried.
*/
Map<String, Double> paramValues = null;
try {
paramValues = autoTuningAPIHelper.getCurrentRunParameters(tuningInput);
} catch (Exception e) {
e.printStackTrace();
}
assertTrue("Param values : " + paramValues, paramValues.isEmpty());
}
}
}
35 changes: 19 additions & 16 deletions test/resources/test-param-generate-data.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -142,37 +142,40 @@ insert into yarn_app_heuristic_result_details (yarn_app_heuristic_result_id,name
(137594640,'Number of tasks','20','NULL');

INSERT INTO flow_definition(id, flow_def_id, flow_def_url) VALUES
(10003,'https://ltx1-holdemaz01.grid.linkedin.com:8443/manager?project=AzkabanHelloPigTest&flow=countByCountryFlow','https://ltx1-holdemaz01.grid.linkedin.com:8443/manager?project=AzkabanHelloPigTest&flow=countByCountryFlow');
(10003,'https://ltx1-holdemaz01.grid.linkedin.com:8443/manager?project=AzkabanHelloPigTest&flow=countByCountryFlow','https://ltx1-holdemaz01.grid.linkedin.com:8443/manager?project=AzkabanHelloPigTest&flow=countByCountryFlow'),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Kindly remove internal Url links.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These lines are already there. I have not modified them except for ';' changed to ',' as I wanted to another line.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, but you can help with correcting this and if need can ask for the help of the author of the respective part.

(10004,'https://elephant.linkedin.com:8443/manager?project=AzkabanHelloPigTest&flow=countByCountryFlow','https://elephant.linkedin.com:8443/manager?project=AzkabanHelloPigTest&flow=countByCountryFlow');

INSERT INTO job_definition(id, job_def_id, flow_definition_id, job_name, job_def_url, scheduler, username, created_ts, updated_ts) VALUES
(100003,'https://ltx1-holdemaz01.grid.linkedin.com:8443/manager?project=AzkabanHelloPigTest&flow=countByCountryFlow&job=countByCountryFlow_countByCountry',10003,'countByCountryFlow_countByCountry','https://ltx1-holdemaz01.grid.linkedin.com:8443/manager?project=AzkabanHelloPigTest&flow=countByCountryFlow&job=countByCountryFlow_countByCountry','azkaban','pkumar2','2018-02-12 08:40:42','2018-02-12 08:40:43');

(100003,'https://ltx1-holdemaz01.grid.linkedin.com:8443/manager?project=AzkabanHelloPigTest&flow=countByCountryFlow&job=countByCountryFlow_countByCountry',10003,'countByCountryFlow_countByCountry','https://ltx1-holdemaz01.grid.linkedin.com:8443/manager?project=AzkabanHelloPigTest&flow=countByCountryFlow&job=countByCountryFlow_countByCountry','azkaban','pkumar2','2018-02-12 08:40:42','2018-02-12 08:40:43'),
(100004,'https://elephant.linkedin.com:8443/manager?project=AzkabanHelloPigTest&flow=countByCountryFlow&job=countByCountryFlow_countByCountry',10004,'countByCountryFlow_countByCountry','https://elephant.linkedin.com:8443/manager?project=AzkabanHelloPigTest&flow=countByCountryFlow&job=countByCountryFlow_countByCountry','azkaban','dukumar','2018-02-12 08:40:42','2018-02-12 08:40:43');

INSERT INTO tuning_job_definition(job_definition_id, client, tuning_algorithm_id, tuning_enabled, average_resource_usage, average_execution_time, average_input_size_in_bytes, allowed_max_resource_usage_percent, allowed_max_execution_time_percent, created_ts, updated_ts, tuning_disabled_reason, number_of_iterations, auto_apply)
VALUES
(100003,'azkaban',4,1,null,5.178423333333334,324168876088,150,150,'2018-02-12 08:40:42','2018-02-12 08:40:43', NULL, 5, true);



(100003,'azkaban',4,1,null,5.178423333333334,324168876088,150,150,'2018-02-12 08:40:42','2018-02-12 08:40:43', NULL, 5, true),
(100004,'azkaban',4,1,null,5.178423333333334,324168876088,150,150,'2018-02-12 08:40:42','2018-02-12 08:40:43', NULL, 5, true);

INSERT INTO flow_execution(id, flow_exec_id, flow_exec_url, flow_definition_id) VALUES
(1541,'https://ltx1-holdemaz01.grid.linkedin.com:8443/executor?execid=5416293','https://ltx1-holdemaz01.grid.linkedin.com:8443/executor?execid=5416293',10003);
(1541,'https://ltx1-holdemaz01.grid.linkedin.com:8443/executor?execid=5416293','https://ltx1-holdemaz01.grid.linkedin.com:8443/executor?execid=5416293',10003),
(1543,'https://elephant.linkedin.com:8443/executor?execid=5416294','https://elephant.linkedin.com:8443/executor?execid=5416294',10004);

INSERT INTO job_execution(id, job_exec_id, job_exec_url, job_definition_id, flow_execution_id, execution_state, resource_usage, execution_time, input_size_in_bytes, created_ts, updated_ts) VALUES
(1541,'https://ltx1-holdemaz01.grid.linkedin.com:8443/executor?execid=5416293&job=countByCountryFlow_countByCountry&attempt=0','https://ltx1-holdemaz01.grid.linkedin.com:8443/executor?execid=5416293&job=countByCountryFlow_countByCountry&attempt=0',100003,1541,'SUCCEEDED',21.132545572916666,3.2694833333333335,324713861757,'2018-02-14 05:30:42','2018-02-14 05:30:42');
(1541,'https://ltx1-holdemaz01.grid.linkedin.com:8443/executor?execid=5416293&job=countByCountryFlow_countByCountry&attempt=0','https://ltx1-holdemaz01.grid.linkedin.com:8443/executor?execid=5416293&job=countByCountryFlow_countByCountry&attempt=0',100004,1541,'SUCCEEDED',21.132545572916666,3.2694833333333335,324713861757,'2018-02-14 05:30:42','2018-02-14 05:30:42'),
(1543,'https://elephant.linkedin.com:8443/executor?execid=5416294&job=countByCountryFlow_countByCountry&attempt=0','https://elephant.linkedin.com:8443/executor?execid=5416294&job=countByCountryFlow_countByCountry&attempt=0',100004,1543,'SUCCEEDED',21.132545572916666,3.2694833333333335,324713861757,'2018-02-14 05:30:42','2018-02-14 05:30:42');

INSERT INTO job_suggested_param_set VALUES
(1137,100003,4,'FITNESS_COMPUTED', 0 ,1 , 0 , 1 , 0 ,10000 , 1086,'2018-09-17 23:22:31' ,'2018-09-17 11:09:31');
(1137,100003,4,'FITNESS_COMPUTED', 0 ,1 , 0 , 1 , 0 ,10000 , 1086,'2018-09-17 23:22:31' ,'2018-09-17 11:09:31'),
(1139,100004,4,'FITNESS_COMPUTED', 0 ,1 , 0 , 1 , 0 ,10000 , 1543,'2018-09-17 23:22:31' ,'2018-09-17 11:09:31');

INSERT INTO tuning_job_execution_param_set (job_suggested_param_set_id,job_execution_id,tuning_enabled,created_ts,updated_ts) VALUES
(1137,1541,1,'2018-09-17 23:22:31','2018-09-17 10:52:31');
(1137,1541,1,'2018-09-17 23:22:31','2018-09-17 10:52:31'),
(1139,1543,1,'2018-09-17 23:22:31','2018-09-17 10:52:31');

INSERT INTO job_suggested_param_value VALUES
(366044, 1137, 21, 6112.888799667358, '2018-12-21 14:55:36', '2018-12-21 14:55:37'),
(366045, 1137, 22, 1854.7078742980957, '2018-12-21 14:55:36', '2018-12-21 14:55:37'),
(366046, 1137, 23, 3, '2018-12-21 14:55:36', '2018-12-21 14:55:37'),
(366047, 1137, 24, 0.27054230043558763, '2018-12-21 14:55:36', '2018-12-21 14:55:37');




(366047, 1137, 24, 0.27054230043558763, '2018-12-21 14:55:36', '2018-12-21 14:55:37'),
(366052, 1139, 21, 6112.888799667358, '2018-12-21 14:55:36', '2018-12-21 14:55:37'),
(366053, 1139, 22, 1854.7078742980957, '2018-12-21 14:55:36', '2018-12-21 14:55:37'),
(366054, 1139, 23, 3, '2018-12-21 14:55:36', '2018-12-21 14:55:37'),
(366055, 1139, 24, 0.27054230043558763, '2018-12-21 14:55:36', '2018-12-21 14:55:37');