cpef_test.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# 私募基金信息数据库\n",
        "\n",
        "## 总结\n",
        "\n\n",
        "## 思路\n",
        "Selenium和Requests/BeautifulSoup\n",
        "\n",
        "## 应用\n",
        "1. 私募基金求职\n",
        "2. 未来客户预测：结合客户画像分析\n",
        "\n",
        "## 步骤\n",
        "1. 读取页数：从'共 23569 条记录，共236页'截取pages (236)\n",
        "2. 读取每页的所有链接数\n",
        "    1. 等候并点击登录确认\n",
        "    1. 爬取第一页,append到`df`\n",
        "    2. 点击（Click）下一页，休眠1秒\n",
        "    3. 直到'paginate_button paginate_number current' == pages (236)\n",
        "    4. 登录每一个链接，使用KYC中的`result_m`，merge其信息到`df`中\n",
        "3. 打开每个链接爬取信息并append到`df`\n",
        "4. 记录新增/删除等变动信息\n",
        "5. 数据可视化分析\n",
        "6. 性能优化：Big O\n",
        "\n",
        "## 当前总数\n",
        "共 23569 条记录，共236页 2018-5-11\n",
        "\n",
        "## 问题\n",
        "1. 问什么不用定义Global？\n",
        "\n",
        "## Append df的性能优化\n",
        "V1.0 基础功能 2018-5-11\n",
        "\n",
        "V1.1 使用Numpy Array代替Pandas DataFrame \n",
        "\n",
        "2018-6-6 11:28:57 Wall time: 5min 42s\n",
        "\n",
        "V2.0 使用PhantomJS\n",
        "\n",
        "V3.0 MultiProcessing\n",
        "\n",
        "## Append df_m的性能优化\n",
        "\n",
        "V1.0 原始功能实现，性能优化 2018-5-11\n",
        "[04:57<88:37:39, 13.54s/it]\n",
        "\n",
        "V1.1 删除函数get_result_m中的df1冗余项 **Debugging Now**\n",
        "\n",
        "V1.2 函数get_result_m中添加并加快`存续产品数量`查询的速度\n",
        "\n",
        "v1.3 函数get_result_m中使用numpy代替pandas\n",
        "\n",
        "V2.0 完善Result_m中复杂项目信息的可读性（人员信息等）\n",
        "\n",
        "V3.0 使用Scrapy的多线程爬虫\n"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "from cpef import *"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "collapsed": false,
        "outputHidden": false,
        "inputHidden": false
      }
    },
    {
      "cell_type": "code",
      "source": [
        "df = pd.read_excel('funds_list.xlsx', encoding='gb18030')\n",
        "df.head()"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "collapsed": false,
        "outputHidden": false,
        "inputHidden": false
      }
    },
    {
      "cell_type": "code",
      "source": [
        "try:\n",
        "    df = pd.read_excel('funds_list.xlsx', encoding='gb18030')\n",
        "except:\n",
        "    df = get_cpef_list()\n",
        "\n",
        "df_info = pd.DataFrame()\n",
        "\n",
        "for j in tqdm(range(len(df))):\n",
        "    result_m = get_cpef_manager(df.网址[j],tqdm_use=False)\n",
        "    list_manager = []\n",
        "    for i in range(len(df.columns)):\n",
        "        list_manager.append([df.columns[i],df.iloc[j].values[i]])\n",
        "    df_manager = pd.DataFrame(np.concatenate((list_manager, result_m.values), axis=0))\n",
        "    df_manager.index = df_manager.iloc[:,0]\n",
        "    df_manager = df_manager.T.drop([0])\n",
        "    df_info = df_info.append(df_manager,ignore_index=True)\n",
        "    \n",
        "df_info"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "collapsed": false,
        "outputHidden": false,
        "inputHidden": false
      }
    },
    {
      "cell_type": "code",
      "source": [
        "get_cpef_info()"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "collapsed": false,
        "outputHidden": false,
        "inputHidden": false
      }
    }
  ],
  "metadata": {
    "kernel_info": {
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": "3.6.5",
      "mimetype": "text/x-python",
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "pygments_lexer": "ipython3",
      "nbconvert_exporter": "python",
      "file_extension": ".py"
    },
    "kernelspec": {
      "name": "python3",
      "language": "python",
      "display_name": "Python 3"
    },
    "nteract": {
      "version": "0.9.1"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 4
}