{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"inputHidden": false
},
"source": [
"# 依托大数据的行政区划调整决策预测——以县（市）改区为例"
]
},
{
"cell_type": "markdown",
"metadata": {
"inputHidden": false
},
"source": [
"<img src=\"./image/ZJU.png\" width=30> 汇报人：陈思瑾"
]
},
{
"cell_type": "markdown",
"metadata": {
"inputHidden": false
},
"source": [
"## 选题背景\n",
"---\n",
"### 大数据时代对政府科学决策提出了更高的要求。\n",
"### 行政区划调整是政府决策的一个重要的表现形式。\n",
"\n",
"### 但是，行政区划调整的依据是什么？\n",
"\n",
"+ 哪些地方会采取行政区划调整？\n",
"+ 哪些地方适合行政区划调整？\n",
"+ 相关的理论探讨还很欠缺。"
]
},
{
"cell_type": "markdown",
"metadata": {
"inputHidden": false
},
"source": [
"## 文献综述\n",
"---\n",
"### 大数据与政府决策\n",
"+ 数据来源于客观事实、反映着实际问题\n",
"+ 在科学完整的数据搜集、分析的基础\n",
"+ 领导者可以利用大数据技术的预测性分析功能\n",
"+ 有效提高决策的精准化、科学化水平\n",
"\n",
"### 行政区划调整的经验主义弊端\n",
"+ 行政区划调整已经成为重要促进城市区域一体化和区域经济协调发展的重要手段。 \n",
"+ 然而，从实践效果来看，在促进经济发展和城市化的效果却不尽相同。\n",
"+ 主要是由于决策上的路径依赖，主观经验主义\n",
"\n",
"### 县（市）改区是当前行政区划调整的重要方式和关注热点"
]
},
{
"cell_type": "markdown",
"metadata": {
"inputHidden": false
},
"source": [
"## 解决思路\n",
"---\n",
"### 搭建政府智能决策支持系统\n",
"\n",
"<img src=\"./image/pipeline.png\" width=800>"
]
},
{
"cell_type": "markdown",
"metadata": {
"inputHidden": false
},
"source": [
"## 数据收集与探索性数据分析\n",
"---\n",
"+ 本文选取面板数据（1999-2018年）\n",
"\n",
"****《城市统计年鉴》《县域统计年鉴》《国民经济和社会发展统计公报》****\n",
"\n",
"+ 区划信息来源于中华人民共和国民政部和中国行政区划网\n",
"\n",
"<img src=\"./image/division.png\" width=400>\n",
"\n",
"+ 本研究数据集中可选择的部分变量指标\n",
"\n",
"<img src=\"./image/stats.png\" width=400>\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"inputHidden": false
},
"source": [
"## 特征选择\n",
"---\n",
"为了简化数据搜集，主要搜集6项指标：\n",
"\n",
"`行政区域面积` 、\n",
"`居民储蓄存款余额`、\n",
"`第一产业增加值`、\n",
"`第二产业增加值`、\n",
"`规模以上工业企业单位数`、\n",
"`年末金融机构各项贷款余额`\n",
"\n",
"其中，`行政区域面积`用于体现县(市)可供开发面积大小，`居民储蓄存款余额`用于体现县(市)居民实际经济条件，`第一、二产业增加值`用于体现县(市)农业、工业发展速度，`规模以上工业企业单位数`用于体现县(市)企业整体规模，`年末金融机构各项贷款余额`用于体现县(市)企业总体负债情况。\n",
"\n",
"\n",
"\n",
"每个县(市)搜集其该区前10年的数据，保存在单个csv文件中，以下为\"**长乐区.csv**\"文件中数据，\n",
"\n",
"<img src=\"./image/sample.png\" width=800>\n",
"\n",
"对于未改区的县(市)，搜集其所在省份改区县(市)同期的数据"
]
},
{
"cell_type": "markdown",
"metadata": {
"inputHidden": false
},
"source": [
"### 读取文件名\n",
"县(市)改区数据路径为`./mydata/Yes/`，\n",
"未县(市)改区数据路径为`./mydata/No/`"
]
},
{
"cell_type": "code",
"execution_count": 35,
"metadata": {},
"outputs": [],
"source": [
"# pandas 即 Python Data Analysis Library\n",
"import pandas as pd\n",
"import os\n",
"\n",
"# 县市改区 数据路径\n",
"yesPath = './mydata/Yes/'\n",
"# 县市改区 数据路径\n",
"noPath = './mydata/No/'\n",
"\n",
"# 读取给定路径的下所有csv文件，存于list csvFiles中，并作为返回值返回\n",
"def get_file_name(dataPath):\n",
" csvFiles = []\n",
" # root:当前目录路径 dirs：当前目录下所有子目录 files：当前路径下所有非目录文件\n",
" for root,dirs,files in os.walk(dataPath):\n",
" #(files)\n",
" for file in files:\n",
" #for i,file in enumerate(files):\n",
" if os.path.splitext(file)[1] == '.csv':\n",
" csvFiles.append(file)\n",
" break # 避免读取check_point文件\n",
" return csvFiles\n",
"\n",
"\n",
"fileNameYes = get_file_name(yesPath)\n",
"fileNameNo = get_file_name(noPath)\n",
"# print(fileNameYes)\n",
"# print(fileNameNo)\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"inputHidden": false
},
"source": [
"### 读取csv表格数据\n",
"根据csv文件名称列表`fileNameYes`读取每个csv中的表格数据，存储到列表`csv_data_yes`中，元素格式为`dataframe`\n",
"\n",
"`dataframe`为表格型数据结构,二维的数据模型\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 36,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>指 标</th>\n",
" <th>2017</th>\n",
" <th>2016</th>\n",
" <th>2015</th>\n",
" <th>2014</th>\n",
" <th>2013</th>\n",
" <th>2012</th>\n",
" <th>2011</th>\n",
" <th>2010</th>\n",
" <th>2009</th>\n",
" <th>2008</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>行政区域面积</td>\n",
" <td>664</td>\n",
" <td>664</td>\n",
" <td>664</td>\n",
" <td>664</td>\n",
" <td>664</td>\n",
" <td>672</td>\n",
" <td>658</td>\n",
" <td>658</td>\n",
" <td>658</td>\n",
" <td>724</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>第一产业增加值</td>\n",
" <td>491522</td>\n",
" <td>513711</td>\n",
" <td>432828</td>\n",
" <td>427178</td>\n",
" <td>390770</td>\n",
" <td>376367</td>\n",
" <td>335246</td>\n",
" <td>287757</td>\n",
" <td>243915</td>\n",
" <td>239740</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>第二产业增加值</td>\n",
" <td>4706738</td>\n",
" <td>3953424</td>\n",
" <td>3731900</td>\n",
" <td>3592249</td>\n",
" <td>3257469</td>\n",
" <td>2876000</td>\n",
" <td>2553100</td>\n",
" <td>1931600</td>\n",
" <td>1588804</td>\n",
" <td>1616983</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>居民储蓄存款余额</td>\n",
" <td>3151662</td>\n",
" <td>2816356</td>\n",
" <td>2618514</td>\n",
" <td>2597699</td>\n",
" <td>2509358</td>\n",
" <td>2306090</td>\n",
" <td>1982502</td>\n",
" <td>1894371</td>\n",
" <td>1542389</td>\n",
" <td>1376119</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>年末金融机构各项贷款余额</td>\n",
" <td>8689156</td>\n",
" <td>7832658</td>\n",
" <td>6030171</td>\n",
" <td>6020311</td>\n",
" <td>5414426</td>\n",
" <td>4321378</td>\n",
" <td>3151546</td>\n",
" <td>2405598</td>\n",
" <td>1753597</td>\n",
" <td>1051800</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 指 标 2017 2016 2015 2014 2013 2012 \\\n",
"0 行政区域面积 664 664 664 664 664 672 \n",
"1 第一产业增加值 491522 513711 432828 427178 390770 376367 \n",
"2 第二产业增加值 4706738 3953424 3731900 3592249 3257469 2876000 \n",
"3 居民储蓄存款余额 3151662 2816356 2618514 2597699 2509358 2306090 \n",
"4 年末金融机构各项贷款余额 8689156 7832658 6030171 6020311 5414426 4321378 \n",
"\n",
" 2011 2010 2009 2008 \n",
"0 658 658 658 724 \n",
"1 335246 287757 243915 239740 \n",
"2 2553100 1931600 1588804 1616983 \n",
"3 1982502 1894371 1542389 1376119 \n",
"4 3151546 2405598 1753597 1051800 "
]
},
"execution_count": 36,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# 读取县(市)改区信息\n",
"# csv_data_yes列表 用于保存 县(市)改区 所有数据， 每个元素为一个dataframe(表格型数据结构)\n",
"csv_data_yes = []\n",
"for i in range(len(fileNameYes)):\n",
" csv_data_yes.append(pd.read_csv(yesPath+fileNameYes[i], error_bad_lines=False))\n",
"csv_data_yes[0].head()\n",
"# print(csv_data_yes)"
]
},
{
"cell_type": "code",
"execution_count": 37,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>指 标</th>\n",
" <th>2017</th>\n",
" <th>2016</th>\n",
" <th>2015</th>\n",
" <th>2014</th>\n",
" <th>2013</th>\n",
" <th>2012</th>\n",
" <th>2011</th>\n",
" <th>2010</th>\n",
" <th>2009</th>\n",
" <th>2008</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>行政区域面积</td>\n",
" <td>1467</td>\n",
" <td>1467</td>\n",
" <td>1467</td>\n",
" <td>1467</td>\n",
" <td>1467</td>\n",
" <td>1467</td>\n",
" <td>1467</td>\n",
" <td>1467</td>\n",
" <td>1467</td>\n",
" <td>1467</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>第一产业增加值</td>\n",
" <td>282590</td>\n",
" <td>276808</td>\n",
" <td>254933</td>\n",
" <td>230741</td>\n",
" <td>203721</td>\n",
" <td>196012</td>\n",
" <td>177808</td>\n",
" <td>157023</td>\n",
" <td>130424</td>\n",
" <td>123354</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>第二产业增加值</td>\n",
" <td>756832</td>\n",
" <td>785238</td>\n",
" <td>779507</td>\n",
" <td>741551</td>\n",
" <td>673699</td>\n",
" <td>607700</td>\n",
" <td>537500</td>\n",
" <td>479300</td>\n",
" <td>411956</td>\n",
" <td>417242</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>居民储蓄存款余额</td>\n",
" <td>993018</td>\n",
" <td>870944</td>\n",
" <td>840324</td>\n",
" <td>738858</td>\n",
" <td>645667</td>\n",
" <td>571882</td>\n",
" <td>526651</td>\n",
" <td>445659</td>\n",
" <td>383279</td>\n",
" <td>317078</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>年末金融机构各项贷款余额</td>\n",
" <td>767371</td>\n",
" <td>644153</td>\n",
" <td>582445</td>\n",
" <td>502392</td>\n",
" <td>427285</td>\n",
" <td>345973</td>\n",
" <td>281859</td>\n",
" <td>237260</td>\n",
" <td>199415</td>\n",
" <td>143451</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 指 标 2017 2016 2015 2014 2013 2012 2011 \\\n",
"0 行政区域面积 1467 1467 1467 1467 1467 1467 1467 \n",
"1 第一产业增加值 282590 276808 254933 230741 203721 196012 177808 \n",
"2 第二产业增加值 756832 785238 779507 741551 673699 607700 537500 \n",
"3 居民储蓄存款余额 993018 870944 840324 738858 645667 571882 526651 \n",
"4 年末金融机构各项贷款余额 767371 644153 582445 502392 427285 345973 281859 \n",
"\n",
" 2010 2009 2008 \n",
"0 1467 1467 1467 \n",
"1 157023 130424 123354 \n",
"2 479300 411956 417242 \n",
"3 445659 383279 317078 \n",
"4 237260 199415 143451 "
]
},
"execution_count": 37,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# 读取非县(市)改区信息\n",
"csv_data_no = []\n",
"for i in range(len(fileNameNo)):\n",
" csv_data_no.append(pd.read_csv(noPath+fileNameNo[i], error_bad_lines=False))\n",
"csv_data_no[0].head()"
]
},
{
"cell_type": "markdown",
"metadata": {
"inputHidden": false
},
"source": [
"### 格式转换\n",
"将6行10列表格数据转换为1行60列数据，保存在列表`yes_no_data`中\n",
"\n",
"同时将tag(y值)保存在列表`yes_no`中，县(市)改区 = '`T`', 非县(市)改区 = '`F`' \n"
]
},
{
"cell_type": "code",
"execution_count": 38,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"\n",
"# 列表yes_no_data用于保存\n",
"# 列表yes_no用于保存tag(y值) 是否 县(市)改区，县(市)改区 = 'T', 非县(市)改区 = 'F' \n",
"yes_no_data = []\n",
"yes_no = []\n",
"for csv in csv_data_yes:\n",
" # 将dataframe 转换为 array\n",
" arr = csv.iloc[:,1:].values.copy()\n",
" # 将6x10转换为1x60\n",
" # arr.shape[0] = 6 行数目\n",
" # arr.shape[1] = 10 列数目\n",
" arr.resize(1,(arr.shape[0] * arr.shape[1]))\n",
" yes_no_data.append(pd.DataFrame(arr))\n",
" # 添加tag\n",
" yes_no.append('T')\n",
" \n",
"for csv in csv_data_no:\n",
" arr = csv.iloc[:,1:].values.copy()\n",
" arr.resize(1,(arr.shape[0] * arr.shape[1]))\n",
" yes_no_data.append(pd.DataFrame(arr))\n",
" yes_no.append('F')"
]
},
{
"cell_type": "markdown",
"metadata": {
"inputHidden": false
},
"source": [
"用`concat`函数将`yes_no_data`中所有dataframe数据整合到一个表中"
]
},
{
"cell_type": "code",
"execution_count": 39,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" <th>3</th>\n",
" <th>4</th>\n",
" <th>5</th>\n",
" <th>6</th>\n",
" <th>7</th>\n",
" <th>8</th>\n",
" <th>9</th>\n",
" <th>...</th>\n",
" <th>50</th>\n",
" <th>51</th>\n",
" <th>52</th>\n",
" <th>53</th>\n",
" <th>54</th>\n",
" <th>55</th>\n",
" <th>56</th>\n",
" <th>57</th>\n",
" <th>58</th>\n",
" <th>59</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>1997</th>\n",
" <td>3027.143873</td>\n",
" <td>316.112545</td>\n",
" <td>537.402911</td>\n",
" <td>1116.635355</td>\n",
" <td>2217.234179</td>\n",
" <td>667.382718</td>\n",
" <td>848.448337</td>\n",
" <td>2593.9035</td>\n",
" <td>1028.914618</td>\n",
" <td>240.020221</td>\n",
" <td>...</td>\n",
" <td>88.881129</td>\n",
" <td>124.731911</td>\n",
" <td>58.896904</td>\n",
" <td>179.816589</td>\n",
" <td>19.048878</td>\n",
" <td>110.69095</td>\n",
" <td>52.696025</td>\n",
" <td>57.573446</td>\n",
" <td>49.756759</td>\n",
" <td>31.022207</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1998</th>\n",
" <td>3027.143873</td>\n",
" <td>316.112545</td>\n",
" <td>537.402911</td>\n",
" <td>1116.635355</td>\n",
" <td>2217.234179</td>\n",
" <td>667.382718</td>\n",
" <td>930.747826</td>\n",
" <td>2593.9035</td>\n",
" <td>1028.914618</td>\n",
" <td>240.020221</td>\n",
" <td>...</td>\n",
" <td>88.881129</td>\n",
" <td>124.731911</td>\n",
" <td>58.896904</td>\n",
" <td>179.816589</td>\n",
" <td>19.048878</td>\n",
" <td>110.69095</td>\n",
" <td>52.696025</td>\n",
" <td>61.315720</td>\n",
" <td>49.756759</td>\n",
" <td>31.022207</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1999</th>\n",
" <td>3211.799649</td>\n",
" <td>316.112545</td>\n",
" <td>537.402911</td>\n",
" <td>1116.635355</td>\n",
" <td>2217.234179</td>\n",
" <td>667.382718</td>\n",
" <td>930.747826</td>\n",
" <td>2593.9035</td>\n",
" <td>1028.914618</td>\n",
" <td>240.020221</td>\n",
" <td>...</td>\n",
" <td>88.881129</td>\n",
" <td>125.979230</td>\n",
" <td>58.896904</td>\n",
" <td>179.816589</td>\n",
" <td>19.048878</td>\n",
" <td>110.69095</td>\n",
" <td>52.696025</td>\n",
" <td>61.315720</td>\n",
" <td>49.756759</td>\n",
" <td>31.022207</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2000</th>\n",
" <td>3211.799649</td>\n",
" <td>316.112545</td>\n",
" <td>537.402911</td>\n",
" <td>1116.635355</td>\n",
" <td>2217.234179</td>\n",
" <td>667.382718</td>\n",
" <td>930.747826</td>\n",
" <td>2593.9035</td>\n",
" <td>1125.632592</td>\n",
" <td>240.020221</td>\n",
" <td>...</td>\n",
" <td>88.881129</td>\n",
" <td>125.979230</td>\n",
" <td>58.896904</td>\n",
" <td>179.816589</td>\n",
" <td>19.048878</td>\n",
" <td>110.69095</td>\n",
" <td>52.696025</td>\n",
" <td>61.315720</td>\n",
" <td>53.339246</td>\n",
" <td>31.022207</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2001</th>\n",
" <td>3211.799649</td>\n",
" <td>316.112545</td>\n",
" <td>537.402911</td>\n",
" <td>1116.635355</td>\n",
" <td>2217.234179</td>\n",
" <td>667.382718</td>\n",
" <td>930.747826</td>\n",
" <td>2593.9035</td>\n",
" <td>1125.632592</td>\n",
" <td>258.501778</td>\n",
" <td>...</td>\n",
" <td>89.858822</td>\n",
" <td>125.979230</td>\n",
" <td>58.896904</td>\n",
" <td>179.816589</td>\n",
" <td>19.048878</td>\n",
" <td>110.69095</td>\n",
" <td>52.696025</td>\n",
" <td>61.315720</td>\n",
" <td>53.339246</td>\n",
" <td>31.022207</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>5 rows × 60 columns</p>\n",
"</div>"
],
"text/plain": [
" 0 1 2 3 4 \\\n",
"1997 3027.143873 316.112545 537.402911 1116.635355 2217.234179 \n",
"1998 3027.143873 316.112545 537.402911 1116.635355 2217.234179 \n",
"1999 3211.799649 316.112545 537.402911 1116.635355 2217.234179 \n",
"2000 3211.799649 316.112545 537.402911 1116.635355 2217.234179 \n",
"2001 3211.799649 316.112545 537.402911 1116.635355 2217.234179 \n",
"\n",
" 5 6 7 8 9 ... \\\n",
"1997 667.382718 848.448337 2593.9035 1028.914618 240.020221 ... \n",
"1998 667.382718 930.747826 2593.9035 1028.914618 240.020221 ... \n",
"1999 667.382718 930.747826 2593.9035 1028.914618 240.020221 ... \n",
"2000 667.382718 930.747826 2593.9035 1125.632592 240.020221 ... \n",
"2001 667.382718 930.747826 2593.9035 1125.632592 258.501778 ... \n",
"\n",
" 50 51 52 53 54 55 \\\n",
"1997 88.881129 124.731911 58.896904 179.816589 19.048878 110.69095 \n",
"1998 88.881129 124.731911 58.896904 179.816589 19.048878 110.69095 \n",
"1999 88.881129 125.979230 58.896904 179.816589 19.048878 110.69095 \n",
"2000 88.881129 125.979230 58.896904 179.816589 19.048878 110.69095 \n",
"2001 89.858822 125.979230 58.896904 179.816589 19.048878 110.69095 \n",
"\n",
" 56 57 58 59 \n",
"1997 52.696025 57.573446 49.756759 31.022207 \n",
"1998 52.696025 61.315720 49.756759 31.022207 \n",
"1999 52.696025 61.315720 49.756759 31.022207 \n",
"2000 52.696025 61.315720 53.339246 31.022207 \n",
"2001 52.696025 61.315720 53.339246 31.022207 \n",
"\n",
"[5 rows x 60 columns]"
]
},
"execution_count": 39,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# 设置ignore_index = true以去除index\n",
"total_data = pd.concat(yes_no_data, ignore_index=True)\n",
"total_data.tail()"
]
},
{
"cell_type": "markdown",
"metadata": {
"inputHidden": false
},
"source": [
"将列表`yes_no`转换为pandas一维数组"
]
},
{
"cell_type": "code",
"execution_count": 40,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"1997 F\n",
"1998 F\n",
"1999 F\n",
"2000 F\n",
"2001 F\n",
"dtype: object"
]
},
"execution_count": 40,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"yes_no = pd.Series(yes_no)\n",
"yes_no.tail()"
]
},
{
"cell_type": "markdown",
"metadata": {
"inputHidden": false
},
"source": [
"### 样本划分\n",
"用`train_test_split`函数将样本数据随机划分为训练集(`X_train`、`y_train`)和测试集(`X_test`、`y_test`)\n",
"\n",
"`train_test_split`随机客观的划分数据，减少人为因素\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 42,
"metadata": {},
"outputs": [],
"source": [
"from sklearn.model_selection import train_test_split\n",
"\n",
"# 把数据分为训练集和测试集\n",
"X_train, X_test, y_train, y_test = train_test_split(total_data, yes_no,\n",
" test_size = 50, # 50\n",
" random_state = 2,\n",
" stratify = yes_no)\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"inputHidden": true
},
"source": [
"### 定义函数\n",
"定义训练模型函数`train_classifier`、模型预测函数`predict_labels`、训练评估函数`train_predict`\n",
"\n",
"此处函数复制自[英超足球比赛结果预测](https://momodel.cn/explore/5c8efdda1afd9477ea6b0102?type=app)"
]
},
{
"cell_type": "code",
"execution_count": 41,
"metadata": {},
"outputs": [],
"source": [
"from time import time\n",
"from sklearn.metrics import f1_score\n",
"\n",
"def train_classifier(clf, X_train, y_train):\n",
" ''' 训练模型 '''\n",
"\n",
" # 记录训练时长\n",
" start = time()\n",
" clf.fit(X_train, y_train)\n",
" end = time()\n",
"\n",
" print(\"训练时间 {:.4f} 秒\".format(end - start))\n",
"\n",
"\n",
"def predict_labels(clf, features, target):\n",
" ''' 使用模型进行预测 '''\n",
"\n",
" # 记录预测时长\n",
" start = time()\n",
" y_pred = clf.predict(features)\n",
"\n",
" end = time()\n",
"\n",
" print(\"预测时间 in {:.4f} 秒\".format(end - start))\n",
"\n",
" return f1_score(target, y_pred, pos_label='T'), sum(target == y_pred) / float(len(y_pred))\n",
"\n",
"\n",
"def train_predict(clf, X_train, y_train, X_test, y_test):\n",
" ''' 训练并评估模型 '''\n",
"\n",
" # Indicate the classifier and the training set size\n",
" print(\"训练 {} 模型，样本数量 {}. . .\".format(clf.__class__.__name__, len(X_train)))\n",
"\n",
" # 训练模型\n",
" train_classifier(clf, X_train, y_train)\n",
"\n",
" # 在测试集上评估模型\n",
" f1, acc = predict_labels(clf, X_train, y_train)\n",
" print(\"训练集上的 F1 分数和准确率为: {:.4f} , {:.4f}.\".format(f1 , acc))\n",
"\n",
" f1, acc = predict_labels(clf, X_test, y_test)\n",
" print(\"测试集上的 F1 分数和准确率为: {:.4f} , {:.4f}.\".format(f1 , acc))"
]
},
{
"cell_type": "markdown",
"metadata": {
"inputHidden": false
},
"source": [
"### 安装`xgboost`包"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"/home/jovyan/.virtualenvs/basenv/lib/python3.5/site-packages/OpenSSL/crypto.py:12: CryptographyDeprecationWarning: Python 3.5 support will be dropped in the next release ofcryptography. Please upgrade your Python.\n",
" from cryptography import x509\n",
"Looking in indexes: https://mirrors.aliyun.com/pypi/simple\n",
"Collecting xgboost\n",
"\u001b[?25l Downloading https://mirrors.aliyun.com/pypi/packages/7c/32/a11befbb003e0e6b7e062a77f010dfcec0ec3589be537b02d2eb2ff93b9a/xgboost-1.1.1-py3-none-manylinux2010_x86_64.whl (127.6MB)\n",
"\u001b[K |████████████████████████████████| 127.6MB 706kB/s eta 0:00:01 | 1.4MB 1.5MB/s eta 0:01:25\n",
"\u001b[?25hRequirement already satisfied: scipy in /home/jovyan/.virtualenvs/basenv/lib/python3.5/site-packages (from xgboost) (1.4.1)\n",
"Requirement already satisfied: numpy in /home/jovyan/.virtualenvs/basenv/lib/python3.5/site-packages (from xgboost) (1.16.0)\n",
"Installing collected packages: xgboost\n",
"Successfully installed xgboost-1.1.1\n",
"\u001b[33mWARNING: You are using pip version 19.1.1, however version 20.2.4 is available.\n",
"You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\n"
]
}
],
"source": [
"!pip install xgboost"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 模型建立与评估\n",
"\n",
"`Logistic Regression` 逻辑回归 主要用于二分类\n",
"\n",
"`SVM` 支持向量积\n",
"\n",
"`xgboost` "
]
},
{
"cell_type": "code",
"execution_count": 43,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/home/jovyan/work/.localenv/lib/python3.5/site-packages/xgboost/__init__.py:29: FutureWarning: Python 3.5 support is deprecated; XGBoost will require Python 3.6+ in the near future. Consider upgrading to Python 3.6+.\n",
" FutureWarning)\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"训练 LogisticRegression 模型，样本数量 1952. . .\n",
"训练时间 0.0267 秒\n",
"预测时间 in 0.0010 秒\n",
"训练集上的 F1 分数和准确率为: 1.0000 , 1.0000.\n",
"预测时间 in 0.0008 秒\n",
"测试集上的 F1 分数和准确率为: 1.0000 , 1.0000.\n",
"\n",
"训练 SVC 模型，样本数量 1952. . .\n",
"训练时间 0.3767 秒\n",
"预测时间 in 0.3408 秒\n",
"训练集上的 F1 分数和准确率为: 1.0000 , 1.0000.\n",
"预测时间 in 0.0096 秒\n",
"测试集上的 F1 分数和准确率为: 0.6667 , 0.5000.\n",
"\n",
"训练 XGBClassifier 模型，样本数量 1952. . .\n",
"训练时间 0.1625 秒\n",
"预测时间 in 0.1064 秒\n",
"训练集上的 F1 分数和准确率为: 1.0000 , 1.0000.\n",
"预测时间 in 0.0016 秒\n",
"测试集上的 F1 分数和准确率为: 1.0000 , 1.0000.\n",
"\n"
]
}
],
"source": [
"import xgboost as xgb\n",
"from sklearn.linear_model import LogisticRegression\n",
"from sklearn.svm import SVC\n",
"\n",
"# 分别建立三个模型\n",
"clf_A = LogisticRegression(random_state = 42)\n",
"clf_B = SVC(random_state = 42, kernel='rbf',gamma='auto')\n",
"clf_C = xgb.XGBClassifier(seed = 42)\n",
"\n",
"train_predict(clf_A, X_train, y_train, X_test, y_test)\n",
"print('')\n",
"train_predict(clf_B, X_train, y_train, X_test, y_test)\n",
"print('')\n",
"train_predict(clf_C, X_train, y_train, X_test, y_test)\n",
"print('')\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 为xgboost调参"
]
},
{
"cell_type": "code",
"execution_count": 44,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,\n",
" colsample_bynode=1, colsample_bytree=1, gamma=0, gpu_id=-1,\n",
" importance_type='gain', interaction_constraints='',\n",
" learning_rate=0.300000012, max_delta_step=0, max_depth=5,\n",
" min_child_weight=1, missing=nan, monotone_constraints='()',\n",
" n_estimators=90, n_jobs=0, num_parallel_tree=1,\n",
" objective='binary:logistic', random_state=42, reg_alpha=0,\n",
" reg_lambda=1, scale_pos_weight=1, seed=42, subsample=1,\n",
" tree_method='exact', validate_parameters=1, verbosity=None)\n",
"预测时间 in 0.0045 秒\n",
"F1 score and accuracy score for training set: 1.0000 , 1.0000.\n",
"预测时间 in 0.0015 秒\n",
"F1 score and accuracy score for test set: 1.0000 , 1.0000.\n"
]
}
],
"source": [
"from sklearn.model_selection import GridSearchCV\n",
"from sklearn.metrics import make_scorer\n",
"import xgboost as xgb\n",
"\n",
"# 设置想要自动调参的参数\n",
"parameters = { 'n_estimators':[90,100,110],\n",
" 'max_depth': [5,6,7],\n",
" }\n",
"\n",
"# 初始化模型\n",
"clf = xgb.XGBClassifier(seed=42)\n",
"\n",
"f1_scorer = make_scorer(f1_score,pos_label='T')\n",
"\n",
"# 使用 grdi search 自动调参\n",
"grid_obj = GridSearchCV(clf,\n",
" scoring=f1_scorer,\n",
" param_grid=parameters,\n",
" cv=5) # 5\n",
"\n",
"grid_obj = grid_obj.fit(X_train,y_train)\n",
"\n",
"# 得到最佳的模型\n",
"clf = grid_obj.best_estimator_\n",
"print(clf)\n",
"\n",
"# 查看最终的模型效果\n",
"f1, acc = predict_labels(clf, X_train, y_train)\n",
"print(\"F1 score and accuracy score for training set: {:.4f} , {:.4f}.\".format(f1 , acc))\n",
"\n",
"f1, acc = predict_labels(clf, X_test, y_test)\n",
"print(\"F1 score and accuracy score for test set: {:.4f} , {:.4f}.\".format(f1 , acc))\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 安装joblib\n",
"`joblib`用于保存模型和加载模型"
]
},
{
"cell_type": "code",
"execution_count": 39,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"/home/jovyan/.virtualenvs/basenv/lib/python3.5/site-packages/OpenSSL/crypto.py:12: CryptographyDeprecationWarning: Python 3.5 support will be dropped in the next release ofcryptography. Please upgrade your Python.\n",
" from cryptography import x509\n",
"Looking in indexes: https://mirrors.aliyun.com/pypi/simple\n",
"Requirement already satisfied: joblib in /home/jovyan/.virtualenvs/basenv/lib/python3.5/site-packages (0.14.1)\n",
"\u001b[33mWARNING: You are using pip version 19.1.1, however version 20.2.4 is available.\n",
"You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\n"
]
}
],
"source": [
"!pip install joblib"
]
},
{
"cell_type": "code",
"execution_count": 45,
"metadata": {},
"outputs": [],
"source": [
"import joblib\n",
"#保存模型\n",
"joblib.dump(clf, 'xgboost_model.model')\n",
"\n",
"#读取模型\n",
"xgb = joblib.load('xgboost_model.model')"
]
},
{
"cell_type": "code",
"execution_count": 46,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>0</th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" <th>3</th>\n",
" <th>4</th>\n",
" <th>5</th>\n",
" <th>6</th>\n",
" <th>7</th>\n",
" <th>8</th>\n",
" <th>9</th>\n",
" <th>...</th>\n",
" <th>50</th>\n",
" <th>51</th>\n",
" <th>52</th>\n",
" <th>53</th>\n",
" <th>54</th>\n",
" <th>55</th>\n",
" <th>56</th>\n",
" <th>57</th>\n",
" <th>58</th>\n",
" <th>59</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>1467</th>\n",
" <td>2480.332109</td>\n",
" <td>1005.537137</td>\n",
" <td>771.843217</td>\n",
" <td>1416.428538</td>\n",
" <td>1506.696157</td>\n",
" <td>1167.427993</td>\n",
" <td>1133.619332</td>\n",
" <td>1903.042117</td>\n",
" <td>2478.064433</td>\n",
" <td>581.543427</td>\n",
" <td>...</td>\n",
" <td>74.402223</td>\n",
" <td>87.078194</td>\n",
" <td>52.6178</td>\n",
" <td>122.864835</td>\n",
" <td>48.829782</td>\n",
" <td>103.537119</td>\n",
" <td>59.784423</td>\n",
" <td>102.581711</td>\n",
" <td>79.436192</td>\n",
" <td>92.347643</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>1 rows × 60 columns</p>\n",
"</div>"
],
"text/plain": [
" 0 1 2 3 4 \\\n",
"1467 2480.332109 1005.537137 771.843217 1416.428538 1506.696157 \n",
"\n",
" 5 6 7 8 9 ... \\\n",
"1467 1167.427993 1133.619332 1903.042117 2478.064433 581.543427 ... \n",
"\n",
" 50 51 52 53 54 55 \\\n",
"1467 74.402223 87.078194 52.6178 122.864835 48.829782 103.537119 \n",
"\n",
" 56 57 58 59 \n",
"1467 59.784423 102.581711 79.436192 92.347643 \n",
"\n",
"[1 rows x 60 columns]"
]
},
"execution_count": 46,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# 随机挑选模型用于测试\n",
"sample1 = X_test.sample(n=1, random_state=8000)\n",
"sample1\n"
]
},
{
"cell_type": "code",
"execution_count": 47,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"RangeIndex(start=0, stop=60, step=1)"
]
},
"execution_count": 47,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sample1.keys()"
]
},
{
"cell_type": "code",
"execution_count": 48,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array(['F'], dtype=object)"
]
},
"execution_count": 48,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# 进行模型预测\n",
"y_pred = xgb.predict(sample1)\n",
"y_pred\n"
]
},
{
"cell_type": "code",
"execution_count": 76,
"metadata": {},
"outputs": [],
"source": [
"# 用于生成handle函数的参数定义\n",
"total_year = 10\n",
"count = 0\n",
"\n",
"def printParas(str1, str2, count):\n",
" for i in range(total_year):\n",
" count += 1\n",
" # 打印app_spec.yml参数\n",
" #print(' %s%d:'%(str1,i+1))\n",
" #print(' name: %s%d'%(str1,i+1))\n",
" #print(' value_type: float')\n",
" #print(' description: 县(市)改区前%d年%s(%s)'%(i+1,str1,str2))\n",
" \n",
" #打印handle函数 参数\n",
" #print('\\'%s%d\\','%(str1,i+1), end='')\n",
" return count\n",
"\n",
"count = printParas('行政区域面积', '平方公里', count)\n",
"count = printParas('第一产业增加值', '万元', count)\n",
"count = printParas('第二产业增加值', '万元', count)\n",
"count = printParas('居民储蓄存款余额', '万元', count)\n",
"count = printParas('年末金融机构各项贷款余额', '万元', count)\n",
"count = printParas('规模以上工业企业单位数', '个', count)"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"import joblib\n",
"def handle(conf):\n",
" \"\"\"\n",
" 该方法是部署之后，其他人调用你的服务时候的处理方法。\n",
" 请按规范填写参数结构，这样我们就能替你自动生成配置文件，方便其他人的调用。\n",
" 范例：\n",
" params['key'] = value # value_type: str # description: some description\n",
" value_type 可以选择：img, video, audio, str, int, float, [int], [str], [float]\n",
" 参数请放到params字典中，我们会自动解析该变量。\n",
" \"\"\"\n",
" df = pd.DataFrame(conf, index=[0])\n",
" conf_str = ['行政区域面积1','行政区域面积2','行政区域面积3','行政区域面积4','行政区域面积5',\\\n",
" '行政区域面积6','行政区域面积7','行政区域面积8','行政区域面积9','行政区域面积10',\\\n",
" '第一产业增加值1','第一产业增加值2','第一产业增加值3','第一产业增加值4','第一产业增加值5',\\\n",
" '第一产业增加值6','第一产业增加值7','第一产业增加值8','第一产业增加值9','第一产业增加值10',\\\n",
" '第二产业增加值1','第二产业增加值2','第二产业增加值3','第二产业增加值4','第二产业增加值5',\\\n",
" '第二产业增加值6','第二产业增加值7','第二产业增加值8','第二产业增加值9','第二产业增加值10',\\\n",
" '居民储蓄存款余额1','居民储蓄存款余额2','居民储蓄存款余额3','居民储蓄存款余额4','居民储蓄存款余额5',\\\n",
" '居民储蓄存款余额6','居民储蓄存款余额7','居民储蓄存款余额8','居民储蓄存款余额9','居民储蓄存款余额10',\\\n",
" '年末金融机构各项贷款余额1','年末金融机构各项贷款余额2','年末金融机构各项贷款余额3','年末金融机构各项贷款余额4','年末金融机构各项贷款余额5',\\\n",
" '年末金融机构各项贷款余额6','年末金融机构各项贷款余额7','年末金融机构各项贷款余额8','年末金融机构各项贷款余额9','年末金融机构各项贷款余额10',\\\n",
" '规模以上工业企业单位数1','规模以上工业企业单位数2','规模以上工业企业单位数3','规模以上工业企业单位数4','规模以上工业企业单位数5',\\\n",
" '规模以上工业企业单位数6','规模以上工业企业单位数7','规模以上工业企业单位数8','规模以上工业企业单位数9','规模以上工业企业单位数10']\n",
"\n",
" df = df[conf_str]\n",
" model = joblib.load('xgboost_model.model')\n",
" result = model.predict(df)\n",
" return {'res': result.tolist()[0]}\n",
" "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Reference\n",
"+ pandas https://pandas.pydata.org/pandas-docs/stable/index.html\n",
"+ 采用 Python 机器学习预测足球比赛结果 https://blog.csdn.net/weixin_44015907/article/details/89947334"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.5.2"
}
},
"nbformat": 4,
"nbformat_minor": 4
}