| 大数据处理框架中基于MDP的任务调度算法 |
| |
| 引用本文： | 冯延蓬,仵 博,孟宪军,何国坤,江建举.大数据处理框架中基于MDP的任务调度算法[J].深圳职业技术学院学报,2014(1):7-10. |
| |
| 作者姓名： | 冯延蓬 仵 博 孟宪军 何国坤 江建举 |
| |
| 作者单位： | 深圳职业技术学院教育技术与信息中心,广东深圳518055 |
| |
| 基金项目： | 广东省自然科学基金项目（$2011040004769）和深圳市科技研发资金项目（JcYJ2012061713481736） |
| |
| 摘 要： | 针对大数据处理框架MapReduce中的任务调度问题，提出一种基于Markov决策过程（Markov Decision Process，MDP）的任务调度算法，通过状态集来描述集群中节点的负载和作业的数据本地性需求，使用状态转移函数表示调度策略对状态的影响，采用值迭代求解算法求取最优策略，实现集群中节点的最优调度．实验结果表明，该算法能够保证数据本地性的同时，减少作业响应时间，提高系统综合性能．
|
| 关 键 词： | 大数据 MapReduce Markov决策过程 任务调度 |
Task Scheduling Algorithm Based on MDP in Big Data Processing Framework |
| |
| FENG Yanpeng,WU Bo,MENG Xianjun,HE Guokun,JIANG Jianju.Task Scheduling Algorithm Based on MDP in Big Data Processing Framework[J].Journal of Shenzhen Polytechnic,2014(1):7-10. |
| |
| Authors: | FENG Yanpeng WU Bo MENG Xianjun HE Guokun JIANG Jianju |
| |
| Institution: | (Education Technology and Information Center, Shenzhen Polytechnic, Shenzhen, Guangdong 518055, China ) |
| |
| Abstract: | A task scheduling algorithm based on Markov decision process is proposed to address the problem of task scheduling in MapReduce framework. The algorithm describes the load of node in cluster and data localization using state space. The state transfer function represents the influence scheduling strategy of the state. The optimal scheduling policy is obtained by solving the MDP using value iteration. The experimental results show that this algorithm can guarantee the data locality, reduce job response time and improve the overall performance of the system. |
| |
| Keywords: | big data MapReduce Markov decision process task scheduling |
| 本文献已被 CNKI 维普 等数据库收录！ |
